World Bank Document

Public Disclosure Authorized EVALUATION CAPACITY DEVELOPMENT IN ASIA Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized EVALUATION CAPACITY DEVELOPMENT IN ASIA

Selected Proceedings from the International Conference B e i j i n g, October 1999

Edited by Khalid Malik and Christine Roth

United Nations Development Programme Evaluation Office (UNDP/EO) Chinese National Center for Science and Te c h n o l o gy Evaluation (NCSTE) The World Bank Operations Evaluation Department and World Bank Institute (WBI) The views expressed in this report are those of the authors and do not necessarily represent those of the United Nations or the United Nations Developement Programme.

Evaluation Office United Nations Developement Programme One United Nations Plaza New York, NY 10017, USA

Design: Julia Ptasznik, Colonial Communications Corp. Cover Photograph: David Kinley/UNDP CONTENTS

Foreword ...... v

Acronyms ...... vii

Executive Summary ...... ix

P a rt I: The Evolving Context for ECD

1. Opening Speech ...... 3 Ms. Den Nan, Vice Minister of Science and Technology of PRC

2. Opening Statement: The New Dimensions of ECD in International Development ...... 5 Mr. Khalid Malik, Director, UNDP Evaluation Office

3. The Asia Perspective in Evaluation Capacity Development: The Challenges of the New Millennium ...... 9 Mr.Adil Khan

4. Aid Evaluation:A Donor Perspective ...... 19 Mr.Niels Dabelstein, Head of Evaluation Secretariat, DANIDA, OECD/DAC WP

P a rt II: Evaluation of Performance Management in the Public Sector

5. Results Based Management in the Development Cooperation Agencies: A review of Experience ...... 23 Ms. Annette Binnendijk, Consultant UNDP

6. Evaluation Capacity Development in the People’s Republic of China: Trends and Prospects ...... 39 Mr. Ray Rist, Evaluation Advisor,World Bank

7. Evolution of Evaluation in the People’s Republic of China ...... 45 Mr.Peter C. Darjes, Operations Manager, Asian Development Bank

8. Linking Evaluation to Policy Formulation and Budget Processes – Lessons Learned from Australia ...... 49 Mr.Leonard J. Early, Depatment of Finance – Australian Public Sector

9. Public Performance Evaluation and Improvement ...... 55 Mr. Marc Holzer, Executive Director, National Center for Public Productivity, NJ - USA

ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium i i i P a rt III: A) Monitoring Evaluation in a Results-Based Management A p p r o a c h B) Evaluation Initiatives in Science and Te c h n o l o gy

10.Evaluation Parternships ...... 65 Mr.Niels Dabelstein , Head of Evaluation Secretariat DANIDA, OECD/DAC WP

11.Evaluation Tools and Methodologies:Case Studies ...... 69 Mr. Jacques Toulemonde, Director, Centre for European Evaluation Expertise

12.Evaluation Initiatives in Science and Technology Programs ...... 77 Mr.Rustam Lalkaka, President, Business & Technology Development Strategies

13.Review of Draft of Science and Technology Evaluation Standard of China ...... 91 Ms. Chen Zhaoying,Vice President NCSTE

P a rt IV: ECD Opportunities for National Initiatives and International Cooperation

14.Guide to Diagnosing ECD Needs: Video Speech ...... 97 Mr. Keith Mackay, Senior Evaluation Officer,World Bank Institute

P a rt V: Meeting W r a p - U p

15.Meeting Wrap-Up ...... 103 Ms. Christine Roth, Evaluation Advisor, UNDP

P a rt VI: A n n e x e s

Annex 1.List of Participants ...... 107

Annex 2.Conference Agenda ...... 111

i v E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing FOREWORD

There is growing awareness in both developed and developing countries of the potential role that evaluation can play in improving the quality of public sector decision-making. Evaluation is becoming recognized as particularly important in promoting transparency and accountability; in some instances, it is even viewed as an integral part of responsible civic action and civil society development.

Concurrent with the increasing interest in evaluation as a component of public sector development and civil service reform, evaluation capacity development (ECD) is being placed on the agenda of cooperat i o n between developing countrie s and development assistance agencies. The development of national evaluation capacities can be a critical component of efforts to strengthen the responsiveness of public management systems to both internal and external demands for transparency, effectiveness and a results orientation. Therefore, the World Bank, for example, organized a high-level seminar in Abidjan, Côte d’Ivoire, in 1998, during which senior officials from various African countries and international development assistance agencies looked at the status – and helped to plan the future – of monitoring and evaluation (M&E) capacity development in Africa. In the past few yea r s , UNDP has also been parti c u l a r ly active in the field of ECD. A survey of existing national M&E systems was undertaken in more than fifteen programme countries. In collaboration with national entities, recommendations were made to strengthen the effectiveness of these systems.

In this context, the United Nations Development Programme and the Chinese National Center for Science and Technology Evaluation co-sponsored the Beijing Conference on Evaluation Capacity Development (ECD) with strong support from the World Bank. The conference, which was held 27-28 October 1999, focused particularly on issues of performance management and public sector reform.

Par ticipants debated intensely on such issues as tran s p a re n c y and the independence of the eva l u a t i o n function and, as a coroll a r y, the cha l lenges of introducing a culture of eva l u a t i o n at the national level . Some of these cha l lenges stem from the political environment and the need to develop an appreciation at decision-making level of the importance of and the need for evaluation. Other challenges are linked to the systemic problems of institutionalizing and integrating evaluation into the government machinery and the weakness of internal capacities, human and financial, necessary to support a professional cadre of evaluators.

The development of national capacities in evaluation requires strong alliances and partnership-building. Conference participants, aware of the benefits of such relationships, showed strong interest in launching an international development eva l u a t i o n association to help forging links with evaluators engaged in devel o pm e n t related activities in both donor and programme countries and support individual efforts of countries interested in creating such an association at the national level. UNDP and the World Bank emerged as the main promoters of this international association.

Another topic of considerable interest for the Conference was the work carried out by aid agencies on the interface between results-based management (RBM) and monitoring and evaluation in connection with public sector reform and good

F O R E W O R D V govern a n c e . Despite the enormous methodological di v i d e n d s . In parti c u l a r ,the Confe r nce highligted the and techn i c al cha l lenges linked to this management strong potential for maximizing synergies between approach, results orientation in the public sector is the work of devel o p ment agencies,de vel o p ment banks clearly the way of the future. and national entities to promote the role of eva l u a t i o n as part of good gover nance and public sector ref o rm . Gi v en the high level of parti c i p a t i o n and interest in the topics debated during the Confe re n c e , it is cle a r It is hoped that the insight contained in the formal that the rel a t i ve l y modest investment made to finance presentations of this volume will stimulate further the Conference has the potential for yielding big discussions and developments in the field of ECD.

A C K N O W L E D G M E N T S

The sponsors of this conference were: ■ The United Nations Development Programme Evaluation Office (UNDP/EO)

■ The National Center for Science and Technology Evaluation (NCSTE)

■ The World Bank Operations Evaluation Department and World Bank Institute (WBI)

Special thanks go to Professor Fang Yan and Christine Roth for their dedicated work in organizing the Conference. Thanks also go to others who provided substantive inputs in preparing the Agenda: Ray Rist, Keith McKay and Niels Dabelstein. This event would have not been possible without the logistical support provided by Feng Bo, Bousso N’Diaye, Victoria Vieitez,John James and the whole administrative team of the NCSTE. The report was produced by the UNDP’s Evaluatin Office staff including Christine Roth (task manager) Habila Maiga, Barbara Brewka (editor) and Anish Pradhan.

v i E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing ACRONYMS

ADB: Asian Development Bank CBC: Construction Bank of China CIECC: China International Engineering Consulting Company CMED: Central Monitoring and Evaluation Division C N AO: China National Audit Office DAC: Development Assistance Committee DANIDA: Danish International Development Agency ECB: Evaluation Capacity Building ECD: Evaluation Capacity Development EPU: Economic Planning Unit EXSL: Exemplary State and Local Programs G A S B : Governmental Accounting Standard Board IFI: International Finance Institutions IMF: International Monetary Fund ITD: Industrial Technology Development KPIO: Key Project Inspectorate Office MEANS: Methods for Evaluating Actions of a Structural Nature MOF: Ministry of Finance ODA: Overseas Development Administration OECD: Organization for Economic Co-operation Development PEO: Programme Evaluation Organizations PPMS: Project Performance Management System PRC: People’s Republic of China R&D: Research and Development RBM: Results-Based Management RTO: Research and Technology Organizations S & T: Science and Technology SAA: State Audit Administration SDB: State Development Bank SDPC: State Development Planning Commission SERU: Socio-Economic Research Unit SLEA: Sri Lanka Evaluation Association SME: Small and Medium Enterprise SPC: State Planning Commission S P O : State Planning Office SRF: Strategic Result Framework SURF: Sub-regional resource facilities TBI: Technology Business Incubators TEC: Training and Enterprise Council UNDP: United Nations Development Programme UNFSTD: United Nations Fund for Science and Technology Development UNICEF: United Nations Children’s Fund USAID: United States Agency for International Development U S PAP: Uniform Standards for Professional appraisal Practice WAITRO: Wor ld Association of Industrial and Tech n o l o g y Research Organizat i o n

A C R O N Y M S v i i EXECUTIVE SUMMARY

In both developed and developing countries, there is a growing awareness of the potential role that the evaluation function can play in improving the quality of public sector decision-making. Evaluation is being recognized as a mechanism for providing accountability for the use of public res o u r ces as well as an instrum e n t for organizational learning.

Against this backdrop, the United Nations Development Programme (UNDP) and the Chinese National Center for Science and Technology Evaluation (NCSTE) co-sponsored the Beijing Conference on Evaluation and Capacity Development, which took place from 27-28 October 1999. This initiative also re c e i ved strong support from the Asian Deve l o pment Bank (ADB), t h e Development Assistance Committee of the Organisation for Economic Co- operation and Development (OECD/DAC) and the World Bank.

The objectives of the Conference were fourfold: (a) to stimulate reflection on the role of eva l u a t i o n in good gover nance and public sector ref o rm , in c luding sharin g of experiences; (b) to explore the interface between results-based management (RBM) and monitoring and evaluation in connection with good governance; (c) to identify strategies and resources for building monitoring and evaluation (M&E) supply and demand in Asian countries;and (d) to encourage and support the creation of country and regional networks to facilitate follow-up actions.

To achieve these objectives, the Conference agenda was organized around two broad categories of evaluation issues:

(a) The policy and institutional context of evaluation: ■ Policy frameworks for public sector results-orientation ■ Organizational learning vs. management accountability ■ Linkage between evaluation and policy formulation, resource planning, programme monitoring and management ;

(b) The practice of evaluation: ■ Evaluation criteria and conceptual approaches ■ Options for organization of the evaluation function ■ Skills, resources and methodologies ■ Ch a l lenges of eva l u a t i o n in specific sectors, e. g. , science and techn o l o g y.

The Conference brought together high-level officials from 13 Asia and Pacific co u n t r ies (China, De m o c r atic Republic of Korea , Fi j i ,I n d i a ,I n d on e s i a , Kaza k h s t a n , Kyrgyzstan,Malaysia,Mongolia, Nepal, Pakistan, Sri Lanka, Viet Nam), staff of the UNDP country offices in the region, members of EVALNET (the UNDP eva l u a t i o n netwo rk ) , a large number of rep re s e n t a t i v es of the interna t i o nal public sector and development agencies such as the World Bank,The National Center of Public Productivity (USA), The Centre for European Evaluation Expertise,

EXECUTIVE SUMMARY i x O E C D / DAC , the Asian Deve l o pment Bank In view of the short period allocated for group and resource persons from the community of wo rk , the participants were not requested to devel o p international experts. action plans for ECD in their countries but rather to identify a few conc re t e , realistic actions / s t ra t e g i e s The agenda comprised formal presentations by to promote and strengthen ECD in their res p e c t i v e experts in M&E and RBM that explored general countries in the short term. themes as well as specific experiences of donor agencies and national institutions. Participants The resulting action points isolated some importa n t s h a red their experiences and discussed issues elements that were seen as critical to any plan to surrounding the building of a national M&E strengthen ECD: system at the country level. Part of the agenda was also designed to start the process of reflection by ■ Institutional support. Institutional support participants on the design of their own ECD for ECD is cruc i a l . Th e r e should be an awaren e s s action plans. and acceptance of M&E activities in key decision-making centres. Evaluation should Discussions during the Conference highlighted be promoted as an independent function and some important issues faced by national entities. standards and criteria should be developed. First and fore m o s t ,t h e re was unanimous agree m e n t that in a world of globalization and shrinking ■ Trai n i n g . Training still rep r esents an importa n t res o u rc e s , gover nments and public organizat i o ns are element of any ECD programme: training of faced with growing scrut i n y from their cons t i t u e n c y officials and technicians involved in M&E; to ensure increased responsiveness, transparency participation of officials and civil society in and accountability in the delivery of services. In joint evaluations with external aid agencies. this context, the role of both evaluation as part of good gover nance and ECD prog r ammes that could ■ Networks. The concept of networks emerged help to support planned public sector ref o r ms become as a powerful instrument for facilitating the extremely important. Second, the independence ex change of experie n c e s , access to best prac t i c e s of the evaluation function was hotly debated as and sharing of databases. were the potential constraints to the establishment of a culture of eva l u a t i on . These cons t r aints inclu d e The establishment of evaluation networks at both the political environment and the weak demand the national and regional levels is the first concrete for evaluation in many countries, the systemic follow-up action to which participating countries problems that still exist in various countries that made a commitment, with the support of the hinder the applicat i o n of sound eva l u a t i o n syst e m s , international community. China in particular and and limited internal capacities in terms of financial the National Center for Science and Technology and qualified human res o u rc e s . Th i rd , with the sprea d Evaluation volunteered to establish and maintain of results-based management and the realization such a network for the region. that development outcomes require the forging of strategic alliances, discussions centered on the At the international level, both UNDP and the im p o r tance of partnerships in joint eva l u a t i o ns and World Bank, in agreement with several other the need to involve stakeholders and civil society. development agencies, are now working together to promote the establishment of an international Based on the experiences shared by the various development evaluation association. The idea of participants, it became quite obvious that despite the association was widely endorsed at the Beijing the commonality of issues,there is no standardized Conference. Such an association will help in a p p ro a ch to strengthening ev a l u a t i on ca p a c i ty forming links and support the individual efforts of development and developing effective monitoring the several countries that expressed interest in and evaluation systems. creating such an association at the national level.

x E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PART I The Evolving Context for ECD

E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing OPENING SPEECH 1 Deng Nan, Vice Minister of Ministry of Science and Technology of PRC

Distinguish guests, Ladies and Gentlemen:

Good morning! First of all,please allow me on behalf of the Ministry of Science and Technology, as well as myself, to warmly welcome all the officials from UNDP and the Wor ld Bank, as well as the distinguish guests and expert s ,s ch o l a r s on the evaluation from all the other countries participating at the International Conference on Evaluation Capacity Development.

We Chinese people have just celebrated the fiftieth anniversary of the founding of the great People’s Republic of China, and together with other people in the world we are looking for the coming of the new century. Today, initiated jointly by UNDP and National Center for Science & Technology Evaluation, delegates from all over the world gathered here in Beijing, with the topic of improvement of cap a c i t y of eva l u a t i o n to discuss and try to make improvement on the scientific decision making as well as the improvement of responsibility mechanism of government, and the utilization of public resources. This meeting has not only been a very important strategic impact on the course of the Chinese Open-door policy, but also a great impact on the development of all the other countries, so as to the progress of human being.

As everybody knows, humanity is facing a brand new century, with marks of information technology, and a focus on the competition of hi-tech. Technology innovation has greatly motivated the technological progress of the world. In or der to grasp the future of our devel o pm e n t , to manage the society more efficiently, for the best utilization of the resources, which are common to us, for steadily making a better living environment, the improvement of the quality of decision making and efficiency of management are vital.

Along with the development of Chinese reform and its open-door policy, the Chinese government, as early as 1986, has started to focus on the importance of scientific decision making according to the public opinions . In 1993, the Ministry of Science and Technology of China initated technological evaluation methods into the process of macro science and technological management, which is also regarded as an important breaking through of the reform of management on the Chinese Nat i o nal Key Science and Tech n o l o g i c al Pla n . It has resulted in the foundation of the Center of the National Center of Science and Technological Evaluation. Stu d y ing interna t i o nal experie n c e , coping with the Chinese rea l i t i e s , the Nat i on a l Center for Science ant Technology Evaluation (NCSTE) has made relevant and proper eva l u a t i o n rules and indicators syst e m s . The national Center for Science and Tech n o l o g y Evaluation has made a series of effec t i v e eva l u a t i o ns on the effec t i ve n e s s of national key science and technological projects; the capacity of innovation, as well as the operational mechanism of the R&D insititutions and industrial bases.

Although, those activities are still explorative, people have seen the importance of the adoption of evaluation for the improvement of macro level management, the reinforcement of the responsibility of the government and public resource utilization, which are shown in the following aspects:

1. Im p r ove the process of decision making more scientif i ca l ly: The utilizat i o n of eva l u a t i o n has optimized the process of decision making. The profe s s i on a l agencies on the evaluation, the utilization of standard evaluation programs

OPENING SPEECH 3 and technical ways have provided operational characteristics, the professional evaluation, plays foundation for the scientific decision making. more and more important roles. In the process of evaluation, the collection of in f o rm a t i o n from mul t i - ch a n n e l s ,d i f fe r ent angles Recently, the central committee of CPC and the ha v e provided more objective and comp re h e n s i v e State Council cle a r ly appointed out in the document references for the scientific decision making. of « Decisions on the Enforcement of Tech n o l o g i ca l The technique based on the objective facts to Innovations, Promotion of hi-tech, and industrial- get rid of personal bias, has provided warran t e e ization of hi-tech, the evaluation agency system on the justice of scientific decision making. should be promoted in the near future. Since the capacity of evaluation is still far short from the 2. Im p r oved the cap a c i t y of adjustment of science req u i r ement of devel o p ment of the socialist econom y . and techn i c al macro- m a n a g e m e n t : The intro- Th u s , we are now making efforts on the building of th e duction of technical evaluation methodology Chinese science and techn o l o g i c al eva l u a t i o n syst e m . has improved the scope and depth of the vi e wing of management departm e n t . The third For the promo t i o n of the healthier devel o p ment of the pa r ty role and standard eva l u a t i o n methodology, course of Chinese science and techn o l o g y eva l u a t i on , make it possible for the management departm e n t a management rules and relevant counter measures to understand the real situation of the eva l u a t e d , ha v e been under res e a r ch and making, as well as the in order to find the problem and key for the te c hnique crit e ri o n and standards , thus to make im p r ovem e n t . The eva l u a t i o n analysis based on ef f o r t on the perfection of the evaluation system, the facts, ha v e provided refe r ences and rea s on s and im p r ove the quality of the institutions profe s s i on a l for the management department for the and quality of their works. i m p rovement of planning management, adjustement and allocation of resources. We believe, experts and scholars from all over the world gathered together to exchange experience, 3. Promoting the innovation on the science which will greatly promote the building of the and technological management system: The capacity of evaluation of China and delegates’s introduction of evaluation methodology has countries. I appreciate very much the active spirit se p a r ated the judgement from decision making, suggested in the handbook of UNDP Evaluation which has restrained various unhealthy trends, Office, that is « Study ». Evaluation is a process of such as curtness, personal favor, the lack of study, a process of fighting for the realization and in f o rm a t i o n and poor data etc. It is also helpful re-study on objective matter. Not onl y the manage- to prevent corru p t i on of the gove rn m e n t me n t , but also the people undertaking eva l u a t i on , de p a rt m e n t s . To understand the objective rea l i t y and the evaluated, all have the opportunity to via the professinal agencies, the government increase their knowledge over the objectives in the de p a r tement may focus on the decision making process of evaluation, from which to acquire new pro c e s s . This has changed the old system from knowledge, thus to improve the capacity to master point to point management to a new triangle the objective and ref o r m the objective. The eva l u a t i o n balance system,which has provided guarantee of science and technology has just started in for the correctness, effectiveness and safety of China, thus we should even emphasized more on the investment from the government. the spirit of study. One is to activel y study the lessons internationally; the second is to cope with the 4. Re i n f o r cement of the authorit y of the making characteristics and reality of the China. Thus to and enforcement of the Nat ional science plan: de velop scientific and democratic prin c i p l e . I believe The establishment of the process of science and it is the same way for the other countries. te ch n o l o g i c al eva l u a t i on , has helped to improve and attach importance to the plan contents for Lad i e s ,g e n t l e m e n , and frie n d s , the autumn of Beijing all levels and aspects, since it is from the third is with a good fame of « Golden Time », the sky is angle, based on the content of the plan,and blue and high. It is a season full of harvest and verified with the reality. This practice helps to prospective future. For a long time, the Chinese pr event people from making false for the govern m e n t , the Ministry of Science and Tech n o l o g y ap p r oval from the government, thus to raise of PRC enjoyed a broa d ,f ri e n d ly and good cooperat i o n the quality of project approva l . It is also helpful with UNDP,the Wor ld Bank and the other countrie s . for promoting the management staff keeping The opening of this meeting is not only a fruit of on study the latest devel o pm e n t , and underta k i n g our past cooperation, but also a new page for the in ves t i g a t i on s . And it is helpful for moni t o ri n g cooperation in the future in the field of evaluation. the enforce m e n t , and for the self-cons c i e n c e n e s s Now,please allo w me to wish success to the symp o s i u m , for the enforcement of the plan accordingly. and wish every delegates to enjoy a harvesting time in the Beijing « Golden Time» together. Generally speaking, in the process of setting up our socialist market economy with the Chinese Thank you.

4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing The New Dimensions in Evaluation and Evaluation Capacity Development 2 Khalid Malik,Director EO, UNDP

When evaluators get together at a meeting, th e r e is an inevitable interest in sharin g information on concepts and good practices. And this meeting in no exception. However, this meeting is also an opportunity for us to collectively “push up” in the national and international agenda the relevance and value of evaluation concerns as an integral part of decision making and strategy setting. It should allow us to reinforce the need for evaluation capacity development at the country level, by building professional alliances and by broadening our understanding of the range of institutional capacities, strategies, resources and directions.

The dialogue started here should also help identify where genuine demand exists for evaluation to assess effectiveness and relevance of public action. It will also set the basis for exploring various modalities for the donor community to support ECD programmes in Asia.

THE CASE FOR EVALUA T I O N

The 1999 UNDP Human Development Report is on globalization. While documenting the diffe rent forces promoting globaliza t i on and its likely consequences, it also makes a point that a lot of the momentum we see today in the world is the result of conscious decisions taken to promote an open trade and competitive environment. In a rapidly globalization and increasingly uncertain world,taking correct decisions and learning from experience become essential for survival of the firm or organization or indeed of the nation. Evaluation and the role of evaluators become more, not less important. But to grasp these new challenges, the field of evaluation itself has to develop and become more relevant to the need of the day.

As many speakers are going to talk about at this confe re n c e , the field of eva l u a t i o n is changing rapidly, partly as a consequence of dealing with the new demands being placed on it. Increasingly, evaluation is being recognized as an essential part of good governance and public sector reform,and in particular in promoting tra n s p a re n c y and accountability. Some go further and see it as part of res p on s i b l e civic action and civil society development.

Let me in this paper focus on probably one of the most promising areas which has rec e n t l y emerged, an d , that is the interface between results-based management and evaluation. This is the future and the areas which represent enormous technical and methodological challenges for all of us.

Generally, it is fair statement to make that taxpayers, whether they finance do mestic or interna t i o nal agencies, in donor or recipient countrie s , ar e ultimately

The New Dimensions in Evaluation and Evaluation Capacity Dev e l o p m e n t 5 interested in results. Public sector performance also try to connect individual efforts with the has been an issue among citizens of industrialized ma c r o picture, i . e .o u t c om e s , the larger purpo s e countries for some time - more during the 1980’s of why we are doing things, whether it is to and 1990’s, as taxpayers have challenged govern- reduce pove rty or promoting science and ments to demonstrate value for money in the t e ch n o l o gy. This appro a ch raised many provision of public services. And, increasingly, me t h o d o l o g i c al issues such as those conc e rn i n g taxpayers are questioning the very relevance of attribution, when assessing the contribution institutions themselves and their mandates in a of different groups to the realization of a world for rapid change. Are police forces actually specific outcome. leading to better safety and reduce crime, how do related socio-economic forces affect the pictures; But the basic idea is to move away from a pre- are R&D institutions actually leading to products occupation with individual operations towards and services in the market place, etc. assessing the outcome and eve n t u a lly the impact of development interventions. This At the organizational level, similar questions can approach puts the notion of results in the also be raised: what is the value of a project or centre of planning and management efforts programme or indeed the entire organization; and and therefore “managing for results” emerges are the expected results being attained. as a consequence of this appro a ch , with This enormous pressure on demonstrating results performance systems geared to delivering on has led to the introduction of results-based man- results rather than an introverted risk-averse agement in many domestic agencies,mostly in the cu l t u r e that stresses the cont r ol and management DAC countries, and increasingly international aid of inputs and activities. agencies. UNDP is no exception. This in turn has produced a sea of change in the methodologies 2. Strategic Partnerships: Producing outcomes within UNDP by which we evaluate and assess require the concerted effort of many actors in performance. Benchmarking, indicators,measure- society and institutions - the government, the ment are becoming familiar buzzwords in the civil society and relevant aid agencies. If our evaluation community. focus is now on outcomes, then clearly we have of necessity to invest in partnerships The challenge for international aid agencies has which in turn requires us to have a much been to revisit their M & E arrangements, looking clearer idea of our own strength and the con- for ways to develop effective learning systems and tribution we can make to the partnership. to support their partner countrie s . Ef f o r ts to res h a p e This approach fits in well with the idea that the devel o p ment assistance business are also putting development is a broad-based multi-faceted pre s s u r e on evaluators to re-think and upgrade their exercise. Importantly, it provides a specific work in capacity building in programme countries. purpose to the need for partnerships.

In some ways, we are really moving towards the notion of “shared accountability” which T O WARDS A COMMON FRAMEWORK recognizes that valuable roles are played by many actors in society, from NGOs to local Evaluation is changing and is likely to change even as s o c i a t i o ns and so on. Th e r e are methodological more. A shared framework is necessary so that challenges here, for instance how do we access eva l u a t i o n efforts can build on syne r g y and promo t e “shared accountability” to the issue of how broad knowledge sharing. Drawing upon ongoing best we promote national capacity building in work, some thoughts for such a framework could evaluation that includes the representatives of be highlighted. The key building blocks may cover : civil society, indeed as part of our aims of strengthening transparency and accountability 1. Moving towards the notion of outcomes so in society. that the larger developmental purpose could be adequately addres s e d . In some ways, at least 3. Treating evaluation as a key part of good internationally, the global conferences sup- governance. I will not elaborate on this since ported by the UN provide a strong basis for several other speakers are talking about it. follow-up and assessment at the country level. But, a brief comment may still be in order, evaluation is a natural analogue to strategic More broadly, the focus on outcomes implies planning and management and the basis for that we not only look at the micro picture but sound decision-making.

6 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing 4. And this brings me to my fourth point, we separate the two. In the UN, we feel M have to provide a better fit between evaluation goes with E, since the focus is ver y muc h on and decision-making through what is being learning.Objective, discrete evaluations, referred to in the literature as real time or just ar e part of this search for learning benefiting in time assessments. For too long, we have from a structured approach to interacting not adequately responded to the criticisms with different stakeholders and trying to that evaluations are done late and are as such form a considered response to the famous not of much relevance to decision making and evaluator’s ‘so what’ question, you may strategy setting. A large part of this is due to have carried out a project well, but ‘so the fact that the traditional focus of evaluation what’, what does it amount to. has been on projects. We have to move beyond this and perhaps venture into areas c) Link micro assessment with the macro that hitherto have been seen as off limits to questions. Traditional evaluation tries to evaluators. To become relevant, evaluators form an image of effectiveness in terms of have to take on additional, more difficult ef f i c i e n c y, su s t a i n a b i l i t y and the achi e vem e n t re s p on s i b i l i t i e s , of tra cking management of objective s ,e s s e n t i a lly at the micro- l e vel . responses and contributing to the conditions Moving to next level may not be easy, which may lead to the implementation of but it is increasingly becoming necessary. evaluation recommendations. We have to determine the best ways of co l lecting and assessing empiric al evi d e n c e 5. Refining evaluation methodologies in light of so that future policies and programmes the emerging new role of evaluation in a draw upon the right lessons and that “results-based environment”: organizations are more focussed on doing the right things even as mandates change a) Asking the right questions:The questions and we all try to adapt to a fast changing that evaluators ask may well determine environment. the answers- this may be obvious at the end, but not always at the beginning. So Since we are in China, let me close with a quotation how do we construct an approach, which from the sage Confucius: “If concepts are not clear, ensures that the important questions do wo r ds do not fit. If wo r ds do not fit, the day’s work can n o t get asked? be acc o m p l i s h e d . If the day’s work cannot be acc o m p l i s h e d , morals and arts do not flourish, punishments are not b) Trying to find the right balance between just. If punishments are not just, the people do not M&E. There has been a tendency to know where to put hand or foot” (Analects XIII).

The New Dimensions in Evaluation and Evaluation Capacity Dev e l o p m e n t 7 ASIAN PERSPECTIVE IN EVALUATION CAPACITY DEVELOPMENT: The Challenges of the New Millennium 3 Adil Khan, Senior Advisor, M&E,UNOPS/UNDP, Sri Lanka

I . I N T R O D U C T I O N

The nexus between Evaluation Capacity Devel o p ment (ECD) and good devel o pm e n t management cannot be ignore d .H ow ever one defines it, the emerging globalizat i o n process presents a new context for all activities including ECD. The Human De vel o p ment Report 1999 stressed that as an outcome of globaliza t i on ;“ Sh ri n k i n g space, shrinking time and disappearing boarders are linking people’s lives more de e p l y, mo r e intensely, mo r e immediately than ever before” .T h e re f o re, the dictum “when a butterfly flutters its wings in Shanghai, it creates a typhoon in San Francisco” is probably more valid than ever before - meaning that the current situation imposes on each country the complex responsibility of tracking and monitoring changes, both global and local with utmost alertness and developing capacities that can help responding to these changes with equal promptness. The effects of globalization have been both good and bad for every country including that of the Asian economies. However, the recent upheavals in some of the Asian economies drive one home truth - that every country’s ability to tackle the emerging cha l lenges will ver y muc h depend on how well they can organize within themselves a good learning mechanism - a mechanism that is capable of tracking both progress as well as crises and the effective way in which the lessons learnt are feedback to the planning, policy and reform processes of a government.

The world today has more opportunities for people than ever before.Child death rates have fallen by half since 1965 and a child born today can expect to live a de c ade longer than a child born then. Dur ing the last three deca d e s ,s chool enrol m e n t in developing countries has doubled. Adult literacy has increase from 48% in 1970 to 72% in 1997.

The world is definitely more prosperous, and a better place to live now than ever be f o re . In the past 50 yea r s , ave r age per capita income trip l e d . The share of people en j o ying medium devel o p ment has risen more from 55% in 1995 to 66% in 1997 and the share of low human devel o p ment fe ll ,d u ring the same peri o d ,f rom 20% to 10%.

THE CONTEXT

The current globalization process is indeed a very important context of evaluation. As has been explained above, with the promise of gains, globalization has also increased uncertainties,making it imperative the need for continuous monitoring and assessment of emerging situations . Li k ew i s e , the nagging problem of pover ty which consistently challenges the gains of development presents the development context of evaluation. In this context, evaluation is seen as an important tool to learn lessons of experience and help ensure growth with equity. Similarly, post

ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 9 i m p l e m e n t a t i on sustainability of deve l o pm e n t The story is not very different with the other projects and weak sustainability management of major re g i onal deve l o pment bank either- the i nvested re s o u rces present another important Asian Development Bank’s (ADB) recent studies context of evaluation. It has been argued that con f i r m that despite experiencing more success with post implementation monitoring and evaluation di s b u r s e m e n t s , successes with project sustainability of completed projects has the potential to are less satisfactory (Asian Development Bank, g re a t ly enhance their sustainability. Problems 1993).In fact, recent trend shows that less and less of accountability, c o r ru p t i on etc. p rovide the projects are being sustained. The implication of governance context of evaluation. the latter is that while the debt burden of the recipient countries has been rising their capacity to It is evident that despite the phenomenal progress earn income on these borrowings and service debts made during the last two or three decades, these on borrowed resources have been falling, leaving a benefits of globalization did not spread evenly.The mounting debt burden on the future generation. 1999 HDR reports (UNDP, 1999): ■ Nearly 1.3 billion people do not have access to These and the lingering problems of poverty, clean water. falling governance standards and degradation of ■ One in seven children of primary school age env i r onment make it imperat i v e that more attention are out of school. About 840 million are be given to the aspects of: (i) improved planning; malnourished. (ii) prudent and tran s p a r ent use of existing res o u rc e s ; ■ An estimated 1.3 billi o n people live on income s (iii) creation of a social infrastructure, allowing of less than $ 1 (1987 PPP $) a day. freer and wider participation of people in the decision making; and (iv) to complement the In addition to the above, it is also becoming apparen t ab o ve, establishment of an institutional fram ew o rk , that many developing countries in their quest for an evaluation system, that enables achievement of rapid econo mic devel o p ment have also been brin g i n g good governance and sustainable development and about, perhaps inadvertently, the most wanton allows continuous research, lessons learning and destruction to their environment. These disap- management feedback. In a recent paper, Mr. R. Picciotto (1999) of the Wor ld Bank echoes pointing trends which are concurrent to some of si m i l a r sentiments and presents the following as the more positive outcomes of globalization are the con t e m p o ra r y context of eva l u a t i o n - “Acc o u n t a b i l i t y, not only compromising the gains made so far, but transparency, participation and the rule of law have even threatening their sustainability, which indeed become generally the standards of economic gov- indicate that serious efforts are needed to review ernance” and according to him, these elements constantly its dynamics.The concomitant changes, form the basic framework for evaluation. Late both positive as well as negative need to be ana- Mahbub Ul Haq, the architect of the concept of lyzed constantly and that timely and appropriate “human governance” extends implicitly the context feedback given.It is expected that such continuous of evaluation to equity issues and states, “Every monitoring of progress will help ensuring on the governing institution, every policy action should one hand the sustainability of the gains made and be judged by one critical test. How does it meet on the other, mitigate the losses incurred therein. the genuine aspiration of the people” (Human Development Centre, 1999). It is only through Against this backdrop one has also to recognize evaluation that such a “critical test” can be installed that while worldwide the investible resources are and eventuated. d e cl i n i n g, demand for services are incre a s i n g implying, continuous need for prudent use of available resources and maximization of return on MONITORING TO EVALUATION: the invested ones. However, several recent studies THE CHANGING PARADIGM OF ECD suggest that large amount of resources invested in development projects did not always meet with Until recently, however, most donors as well as the expected results. Some facts of life (World government attention on ECD seemed to have Bank 1993): focussed more on monitoring than on evaluation. ■ a recent World Bank study indicates that, This has been due (i) firstly, to unsatisfactory du r ing 1981-1991, the prop o rt i o n of successful experience with implementation;and (ii) secondly, projects has fallen from 85% to 63%, globally. the belief that early implementation of projects ■ the same study reports that the proportion of w i ll autom a t i ca lly con t ribute to genera t i on of major problem projects had increased from expected results. However, all available evidences 11% in 1982 to 20% in 1991. suggest that while monitoring did help speeding

1 0 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing up implementation, these did not automatically In summary, concerns with efficient use of public p roduce desired re s u l t s . On the con t ra ry, in res o u rc e s ,t ra n s p a re n c y and accountability,cha n g i n g some cases these actually acted predatorily - some conditions with globalization, lingering problems development projects made things worse for the with poverty and concerns with good governance poor and in some cases, the environment got e t c ,a re making it increa s i n g l y obvious that efficient de g ra d e d . 1 In these countrie s , absence of an institution and effective evaluation of social and economic of evaluation deprived the decision-makers from dynamics and of public sector programmes and learning the truth and correcting the situation policies are a sin qua non to good development before it was too late. Further with globalization management and an important tool to help facing and with integration of the national economies to the new challenges of the coming millennium. the global system, many assumptions based on which projects and programmes are formulated tend also to become increasingly invalid - making E V A L UATION CA PACITY DEVELOPMENT (ECD): THE CONCEPT it imperative the need for continuous information ga t h e r ing and lessons learning and making adjustments ECD can be defined as an activity or a set of activities to project conditions. that contribute to the establishment of evaluation There are thus, a variety of reasons why more than capacities within the development administration ever before evaluation should be taken seriously structures of developing countries. Development and be treated as an important tool in devel o pm e n t of such capacities involves evolution of systems management. Such as: and methodologies that assist lessons learning from on-going or past projects and programmes I. With increasing democrat i s a t i o n of the societies, and, through these lessons, adjust projects and individuals are becoming more con s c i o u s programmes in such a manner that they achieve about their rights and demanding better servi c e s their planned objectives or improve the quality of and more value-for-mone y from public expen- design of similar projects in future. di t u re . There is only one way a government can know whether what they are doing, are in The World Bank argues that ECD enables its fact, measuring up to the expectations of the member countries to develop suitable tools to people - through evaluation. ensure “quality of its loan portfolio”(World Bank, 1994: iv).The Bank thus sees ECD as a part of an II. Increase in population is creating increasing overall exercise in public sector reform. It justifies demand for more services, but resources are the need for strong ECD by stating that: not increasing at the same rate. Evaluation helps to save money and ensure best use of “Ef fe c t i v e eva l u a t i o n - both “con c u r re n t ” ( i . e .w h i l e existing resources. a prog r amme or project is in prog r ess) and “ex - p o s t ” (i . e . af ter the prog r amme or project is completed) - III. With globalisation and liberal i s a t i o n of nationa l enhances the quality of public investments and of economies, institutions, relationships, pattern the Bank’s portfolio.Well focused and properly of production, pattern of economy and social timed ev a l u a t i on can provide the inform a t i on behaviour etc. ar e cons t a n t l y cha n g i n g , ma k i n g needed to bring about mid-course corrections in it extremely difficult for the governments to p ro g rammes and pro j e c t s , a n a lyze and re s o lve con t r ol and manage events to its own advantage. s ystematic policy issues, the design of future Constant vigilance through monitoring and o p e ra t i on s , and increase country ow n e r s h i p” eva l u a t i o n and res e a r ch of the changing even t s are seen as some of the important imperatives (World Bank, 1994: iv). of good management in the new millennium. Most other donors generally agree with the Bank’s IV. Evaluation ensures transparency and account- ECD strategy, although UNDP as well as some ability in public finance, which are also key bilateral donors wish to focus more emphatically elements of good governance. on the equity aspect of eva l u a t i on , and not limit the tool to assess financial and economic sustainability V. Evaluation can also be politically rewarding to of projects and programmes only. In their view, it a government which is benevolent - for, an is important that each eva l u a t i o n looks more can d i d ly evaluating government is also likely to be at the distri b u t i on aspects of project benefits viewed by its citizens as a listening and a and through lessons learnt consistently supports caring government. p ro g rammes and policies that help ach i eving

1 Alauddin and Tisdell (1989) explain how investment in shrimp culture in Bangladesh pushed small and marginal farmers out oftheir land, made rich richer and poor poorer and degraded the environment.

ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 1 1 human development objectives. To ensure that of good gover nance – “the right to know” by people equity is incorporated as an important agenda in might also be guaranteed. These arrangements evaluation research, these donors also stress the functioned well for a while, but with the changing need for incorporating the beneficiaries into the political and moral environment of the country evaluation process.Thus, according to this school, such noble innovation of the bygone leadership participatory evaluation (mainly as a sequel to could not be sustained. In recent yea r s , PEO though participatory development) should form the basis continues to undertake evaluation work, weak of all future ECD work. However, as corruption political commitment to its works seemed to have and inefficient use of public funds are also becomi n g made some of its initiatives somewhat less tenable. concerns of many donor agencies, the aspects of Nevert h e l e s s , the PEO model leaves one importa n t transparency and accountability are being regarded guidance which is worth paying attention to, the as important subjects of evaluation as well. provision of wider dissemination of evaluation information has indeed the potential to assure maximum feedback, an aspect which continues to remain a problem in many evaluation systems built I I . THE ASIAN EXPERIENCE in Asia.How e ver ,ar ound this time another importa n t trend in India to which a parallel similar to that of AND THE EMERGING ISSUES growth of evaluation could also be drawn.

As in many things in devel o pm e n t , Asia also played While with the eroding idealism and falling standards a pioneering role in setting up formal evaluation in gover nance many institutions of che c ks and balance practices in many of its countries. Interestingly started to become weak in India, the institution en o u g h , so me of these initiatives came from within of Judiciary emerged as the only institution that and not from outside. Presented below are some of continued to serve the interest of the people, the key lessons of ECD in Asia. in d e p e n d e n t l y and fairly. But what made it possible? Many believe that two things worked in favor of Ju d i c i a r y - one , the democratic envi r onment within THE PIONEERS: THE INDIAN AND THE MALAYSIAN EXPERIENCE which the institution operated allowed transparent expression of its views, and secondly, the political Perhaps the most radical of all evaluation capacity and administrative immunity that the institution building initiatives came from India (Khan,1989). received from or been endowed with, by the Inspired by the post-colonial idealism,the political constitution also helped the institution to operate leadership of the day, led by Pandit Nehru set freely and openly. By drawing lessons from this, about to form a benevolent governance structure in many tend to argue that evaluation being a vital the body politic of the country. Search for truth tool of check and balance and indeed, of develop- and anxiety to learn from facts to help improving mental justice, as well as that of public sector the conditions of its citizens, especially that of the accountability, its institutional arrangement cannot poor and the disadvantaged came as a natural simply be left to the monopoly of governmental choice. As early as 1950, a Programme Evaluation whims. Rather, evaluation should be seen as an O r g a n i za t i on (PEO) was created within the integral part of good governance if not, of human Planning Commission of India. Its mandate was de vel o p ment and thus, its institutional arran g e m e n t to undertake post-evaluation of completed projects be hoisted in such a manner that it enjoys legislative and give feedback to the planning process. Among immunity.2 However, prevalence of a democratic other things, the most impressive aspect of this environment in the country is equally important to initiative has been the way the arrangements were ensure free and fair dissemination of information. made to ensure feedback. To guarantee maximum attention, transparency and response to evaluation M a l ays i a’s mon i t o ring and ev a l u a t i on ca p a c i ty findings, provisions were made to circulate the bu i l d i n g , wh i c h also came from within, was starte d evaluation reports to the widest possible audience, in 1960, under the direct patronage of the then including the Public Accounts Committee, the Prime Minister Tun Abdur Razak (Khan, 1999). Parliament Library and the Press.It was envisaged Established mainly to speed up deliver y of servi c e s that wider circulation of evaluation findings of in the rural areas, the Malaysian M&E became government programmes would not only ensure attractive to many Asian governments, who like maximum response, but by making provision to their Malaysian counterpart were also grappling circulate these reports to the Press, a key element with the problems of implementation. In course of

2 Reportedly, the evaluation organisation of Brazil has been established with constitutional backing and statutory funding which is helping it to operate independently and consistently, regardless of changes to governments and their attitudes.

1 2 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing time, government also developed a separate agency However, results of these ECD initiatives have for eva l u a t i o n (the Soc i o - E c on o mic Research Unit - been somewhat mixed. In many Asian countries SERU), but this has since been disbanded and evaluation continues to receive low priority. In eva l u a t i o n as an activity has now been re- e s t a b l i s h e d some, evaluation has a stand-alone position with as a small unit within the Econo mic Planning Unit little effect on the planning process. Lessons learnt (EP U ) , the central planning agency of the govern m e n t . are not fed into formulation of new policies and programmes. Nor, the evaluation and the planning Presently, not much is known about Malaysia’s units are institutiona l ly linked to enable forma l i za t i o n M& E . As far as moni t o r ing is conc e r ned most times of an evaluation feedback. are spent, almost on a daily basis, on monitoring those projects deemed important by the Prime CIVIL SOCIETY INVO LVEMENT IN EVALUATION: Mi n i s t e r .Th e r e is,how e ver , a Comp u t e ri z ed rep o rt i n g THE SRI LANKAN INITIATIVE sy stem to which all projects reg u l a r ly rep o r t physi ca l and financial prog re s s . B u t , it is not know n , Despite these disappointments, in some Asian whether the current system goes beyond physical co u n t ri e s , for example the civil society in Sri Lan k a , and financial monitoring and assesses the results. seemed to have taken initiatives to broaden the institutional framework of evaluation beyond the government and took initiatives to establish the DONOR SUPPORT TO EVALUATION INSTITUTION BUILDING Sri Lanka Evaluation Association (SLEA) in September this year. In addition to these two pioneering initiatives, other Asian countries that established fairly With support from UNICEF and UNDP, several pe r manent institutions of moni t o r ing and eva l u a t i o n University teachers, government researchers and are -Indonesia, the Philippines, Sri Lanka, Nepal, pr ivate sector consultants took initiatives to establish Thailand, China, Bangladesh and South Korea. the Association. Dedicated to developing more of Some of these countries also received a variety of a culture than a mere system of evaluation, the ECD inputs from a range of donor agencies. For SLEVA among others, gave itself the following example UNDP intervention in 1991 saw the key objectives: (i) to function as an advocacy group cre a t i o n of the Central Moni t o r ing and Evaluation for growth of evaluation culture in the country; D i v i s i on (CMED) in Nepal in the Na t i on a l (ii) to help establishing ethics and standards in Planning Commission. The ADB intervention in ev a l u a t i on ; and (iii) to establish national and 1992 in Sri Lanka saw the establishment of the in t e rn a t i o nal netwo r king for informa t i o n excha n g e Performance Evaluation Unit within the Ministry and capacity development. of Plan Implementation and Pa rl i a m e n t a ry A f f a i r s . In 1993 Wo rld Bank supported the Establishment and nurturing of groups such as Ministry of Finance in China to establish a broad SLEA has to be the most exciting innovation in promotion of an evaluation culture in a society. based institutional fra m ew o rk for undert a k i n g App e a r ing as an advocac y group such an association evaluation in the country. During early to mid has the potential to provide vital link between the 1990s both UNDP and the Asian Development civil society and the state and pursue evaluation on Bank continued to provide ECD support to 3 demand basis. Su ch groupings which enlist v a rious Asian countri e s . While the UNDP memberships of individuals and institutions of su p p o r t focused mainly on results based moni t o ri n g both the gover nment as well as the non- g overn m e n t and ev a l u a t i on and pursued intro d u c t i on of and academic institutions may equally help in participatory methodology in evaluation, ADB establishing a collegial link between the govern m e n t introduced what it calls the Project Performance and the civil society and in many cas e s , help rem ov i n g Management System (PPMS).4 In those cases the misgivings associated with evaluation. where ADB felt that the countries were making significant progress in ECD, it followed up in those countries with second and third generation TRAINING: BROADENING THE OPTION assistance. The Philippines, Sri Lanka, Thailand, Nepal and China seemed to have fallen under Another useful lesson learnt in ECD in Asia is these categories of countries. that while these initiatives helped training a large

3 UNDP initiatives in preparation and publication of Country Monographs on National Monitoring and Evaluation Systems equally helped in sensitising and informing the key stakeholders about the situation ofM&E in their countries.This analytical e xercise provided useful foundation for future ECD. 4 PPMS basically means management of the Project Cycle, with inputs from evaluation.

ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 1 3 number of officials in evaluation methods and are also contentious arguments with regard to techniques,these gains could not be sustained fully “indicators” of success. It has been stated in a over time. Three main reasons are given for this recent study that in the Philippines many eva l u a t i o n lapse – (i) frequent turnover of staff from M&E studies - mostly undertaken by academics - are institutions to other agencies; and (ii) secondly, rejected by prog r am managers due to disagree m e n t s poor staff quality; and (iii) thirdly,absence of facilities with one or more of the above factors (Callanta in conducive to application of evaluation skills learnt Khan,1989). Callanta suggests that the absence of through ECD. Moreover, training in evaluation a wider pre-evaluation stakeholder analysis and continues to be an ad hoc, oneoff activity. Further, lack of adequate consultation with regard to the most government training institutes that received need, objectives and methodologies of evaluation ECD inputs, tend generally to operate in is o l a t i o n risk both the credibility as well as the ownership fr om the other non- g over nment acad e m i c institu- and the use of evaluation.5 tions and the Universities, particularly from those (overseas institutes) that are engaged in in n ov a t i v e In recent years the introduction of rapid and res e a r ch in eva l u a t i o n (e.g. pa rt i c i p a t o r y eva l u a t i o n participatory methods in evaluation has caught the et c . ) . Absence of this institutional linkage tends to attention of many and indicates an increasing make it difficult for the government M&E train- willingness to apply rapid methods to conduct ing institutes to maintain required standards. It is, ev a l u a t i on . Evaluators of developing countri e s therefore important that future ECD work in require extensive training in these methods. But, training consider on the one hand, creating better at the same time, they also need to bear in mind linkage between the government and the non-gov- that the rapid method is not a substitute for ernment training institutes and, on the other, the rigorous method, rather, one complements the involve the national level universities and non- other and the choice of one method against government research institutions to incorporate the other would very much depend upon the courses within their curricular, and build res e a r ch eva l u a t i o n questions that one would need to answer.6 capacities in ev a l u a t i on .M a ny believe that incorpo - rat i o n of Universities in ECD is likely to yield fol- In the matter of methodology training in ECD, lowing important gains:(i) Firstly, the Un i ve r s i t i e s , some donors tend to introduce their own in-house unlike the gover nment training institutes will have jargons which seem to create some confusions built-in incentives to run and improve courses as among the developing country practitioners. we l l as undertake res e a r ch and publicat i on s in evaluation; and thus ensure better the aspect of sustain- Recently, in the arena of evaluation methodology ability of ECD inputs; and (ii) secondly, continu- th e r e are discussions about invo l ving the beneficiarie s ous training through its graduate and post-gra d u - into the evaluation process. This undoubtedly is a ate courses shall guarantee stream of grad u a t e s positive initiative and has the potential to enhance trained in evaluation, thus consistently filling the ownership of eva l u a t i o n at all level s . Suc h empha s i s stock of qualified evaluators in a country. will also enable gathering of more qualitative information. However, in societies where beneficiaries are divided by unequal power structure and METHODOLOGIES AND TECHNIQUES: where socio-economic relationships are unevenly M OVING TO NEW DIRECTIONS structured, the ability of the evaluators to access the targeted beneficiaries may prove somewhat Suitable methodologies to measure substantive difficult. In such situations some non-threatening aspects of a project - that is, project effects and interactive methods of data gathering may become impacts, remain a matter of highest concern in all the obvious choice. Training in interactive and eva l u a t i o n work. First of all ,t h e re are complaints that non-threatening methods of data gathering can be evaluation research is too costly and that it takes another aspect of future ECD activities. too long to be of any benefit to management. Sec on d ly,debates also occur with reg a r d to sampling With reg a r d to parti c i p a t o r y eva l u a t i on , Ch a m b e r s and research designs that are capable of making (1997) talks about “em p ow e r ment eva l u a t i o n” wh i c h “be f o re - a ft e r ” and “wi t h - w i t h o u t ” com p a ri s on s .T h e r e involves incorporation of the beneficiaries right

5 Yee (1999) talks of similar difficulties when his team wanted to evaluate the impact of Asian Development Bank assistance in China (The study was called, Country Assistance Programme Evaluation -CARE) and concludes that wider participation ofprogramme stakeholders and adoption of qu a l i ta t i ve assessment principles help assessing impacts in a more “p l au s i bl e ”m a n n er. The study also introduced a agreed-upon simple development effectiveness index to assess success. 6 A recent concept paper, Impact assessment ofUNDP Country Interventions,argues that qualitative studies (quite often the, rapid and participatory studies) are useful pre-requisite to quantitative studies. However, the paper cautions that, “by relying upon qualitative analysis alone, one risks taking perception for reality” .

1 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing WHAT IT TAKES TO MAKE IT W O R K

Monitoring and evaluation (M&E) systems - especially evaluation systems are neither easy to introduce nor are they easy to sustain, especially a good quality system. A recent United Nations Development Programme (UNDP) study - Generic Issues in Monitoring and Evaluation. What Works and Does Not - reflect that success and sustainability of an M&E system depends on a variety of factors, including (Khan, 1993):

■ political support and its continuity. ■ suitable institutional framework with adequate feedback linkage. ■ staff training and logistics support. ■ demonstrated benefits of evaluation. ■ democratic governance environment. ■ develop a “culture” rather than a system - Link civil societies.

The study argues that all of these factors are equally important and that weakening of any one of these factors may has threaten the success of the entire system. For example,the study reports that while political support at the introduction may assist in introduction and institutionalization of a system, if that support is not maintained (by way of use and backing), its operational efficiency and effectiveness will also ultimately weaken. Similarly, a well laid out institutional arrangement will not yield the desired result if it is not backed up by good quality staff and a good feedback arrangement. The study concludes that for M&E to be successful,all the factors that have been identified as key to the success of the system must be equally emphasized and provided with. A lapse of any one element, the report repeatedly argues, will lead to the collapse of the entire system.

at the planning stage where they also become indicate that more emphasis should be given to the principal evaluators and through evaluative rapid methods and innovative research designs. participation, empower themselves to drive the I nv o lvement of Unive r s i t i e s / re s e a rch institutes implementing process to targeted results.7 in evaluation research may also be seen as an important part of future ECD activities. The past ECD works in Asia have also identified a vari e ty of opera t i onal problems that affe c t administration of efficient evaluation. Principal FEEDBACK: DEMOCRATISING THE PROCESS among these is the lack of basic project data. In ma n y countri e s ,p roject offices are either dismantled Feedback relates to methods of linking evaluation or reduced in size after a project has been comp l e t e d findings to the decision-making process of the and most of the staff get shifted to another project. govern m e n t , pa rt i c u l a r ly its planning proc e s s . In These changes affect undertaking of quality eva l u a t i on . Asia different types of methods and channels are Mo ving staff away not onl y means project knowl e d g e currently being used to disseminate evaluation is lost but in many cas e s , even the project documents information. The most commonly used methods get lost. Furt h e r ,the lack of a Project Implementation are: (i) reports/returns; (ii) review meetings; (iii) or Comp l e t i o n Report (PCR) equally cons t r ains the Workshops/ seminars; and (iv) newsletters and opportunity to capture updated data at the time of computer networking. As a result of ECD inter- com p l e t i o n of implementation of a proj e c t . Ef f o rt s ven t i on s ,s o me countries have tried to institutiona l i s e to build new evaluation capacities in developing fe e d b a ck arra n g e m e n t s , without mu ch success. countries must consider ways of collecting and Fu rt h e r, m a ny of the ev a l u a t i on organisation s c o llating project inform a t i on in one place. seem to continue to function in isolation from the Establishment of a documentation centre with brie f s planning institution.8 This institutional gap seems of completed projects will grea t l y assist in allev i a t i n g to be defeating the very purpose of evaluation. It this problem. Further, active involvement of bene- was stated earlier how India had a system of sharin g ficiaries into the evaluation process,is also likely to ev a l u a t i on inform a t i on with a wide audience help retention of data at the grass-root level. including the Parliamentarians and the general p u b l i c . How e ve r, f a lling gove rnance standard s , In summary, it appears that as far as methodology weak gove rnment commitment and politica lly is concerned, there appears to be a lot of room for vu l n e r able institutional arrangement seemed to have improvement and problems with time and money weakened that resolve. Increased democratisation

7 In Bangladesh,BRAC’s (an NGO) Rural Development Programme seemed to have incorporated this methodology quite successfully. 8 For exa m p l e , in Sri Lanka the Perfo r mance Evaluation Unit of the Ministry of Plan Implementation is not institutionally linked to the planning process. New policies and programmes continue to be developed without much consultation with the Performance Evaluation Unit.

ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 1 5 of societies, legislative enshrinment of evaluation is so vital to good governance. institutions,involvement of civil societies and mo s t ■ i n s t i t u t i onal deve l o pment in ev a l u a t i on im p o rt a n t l y, de vel o p ment of democratic culture in a capacity has been viewed by many as purely an country seem to represent the important elements, aid management ritual and not as a tool for if not the key challenges of evaluation capacity good development management. To ensure development and feedback in the new millennium. sustainability of evaluation, its institutional Im p r ovement to eva l u a t i o n fee d b a c k must go hand- development (both as an organization as well in-hand with other democratic ref o r ms in a country. as a function) may req u i r e legislative backi n g . Prevalence of a democratic environment in a Al l in all, while recent ECD works made importa n t society may also be seen as another important con t ri b u t i o ns in introducing and initiating eva l u a t i o n element of evaluation capacity development. pr actices in many Asian countri e s ,f u rther works are ■ limiting evaluation training purely within the ne c e s s a r y to make these interven t i o ns institutiona l ly government training institutes encountered better structured and operationally more efficient. usual difficulties associated with many Th e r e is indeed a need for holistic thinking on ECD gover nment institutions of this nature - limited and further work on this aspect of development research, limited exposure to outside world, management should weigh carefully the reform low incentives for high standards etc. risked pr ocess within which each society is evo lv i n g .Wor k quality and sustainability. on ECD should contextualise the work on democrat i c ■ poor staff quality often contributed to poor ref o r ms that are curren t l y being undertaken in many products coming out of evaluation, affecting countries and support institutional development in the credibility of the organisation and even evaluation in such a manner that the activity itself the merit of the practice. becomes both an agent of change as well as the ■ i n a p p ro p riate and costly methodologies beneficiary of such a charge. con t r ibute to low rel i a b i l i t y and poor acceptance of evaluation. ■ inadequate or poor fe e d b a ck remains a nagging problem. I I I . CHALLENGES OF ECD IN THE NEW MILLENNIUM THE OPTIONS

Overall, it is reasonable to state that the most The new ECD measures must consider these impressive aspect of ECD interventions has been difficulties seriously and come up with solutions, the recognition to and establishment of separate which are consistent with the changing political evaluation units in most developing countries, and economic environment of the world. It is in especially in Asia. There are also evidences that this context following suggestions are put forward: due to improved ev a l u a t i on ,p roject perfo r mance in so me of these countries has also improve d .H ow ever , ■ More emphasis should be given to the devel- despite these gains, practice of evaluation as an opment of an evaluation culture rather than ob vious and untempered element of good govern a n c e just building systems within gove rn m e n t is yet to be fully realized. The new challenges institutions.The former would require linking of ev a l u a t i on stem from a vari e ty of factors. the civil societies - the Universities, research Broadly, these are institutional related, but there institutions, evaluation societies etc. - to the are several organizational matters which also need evaluation process of the government. to be looked at. In this context it is strongly suggested that THE CHALLENGES future ECD measures should include in its va r ious elements the aspect of providing support to the growth of evaluation societies in each As has been observed , a varie t y of cha l lenges still rem a i n : ■ and every country and through this foster tentativeness with evaluation structures make a collegial link between the civil society, and these units vulnerable to changing govern- the government. ment moods. ■ co nfining institutional devel o p ment of eva l u a t i o n ■ Not just for making eva l u a t i o n effec t i v e, but to pu re l y within gover nment stru c t u re s ,l e a ves out en s u r e growth of good govern a n c e , con t i n u o u s a gap and tend to exclude the most importa n t efforts should also be made to systematically actor, the civil society - from an activity that de m o c r atise all societies, cr eate an envi r onm e n t s

1 6 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing of free and fair exchange of information and Governments should be made aware of this. give the ordinary people the opportunity to exercise their “right to know and right to tell’. The governments need to remove from themselves It is only in an environment of openness the fear of evaluation. They need to realise that that evaluation and for that matter, all other evaluation is one tool that will help them doing the institutions of checks and balance and lessons right thing at the right time. They need to be learning can grow. shown that it is an inseparable element in the j o u rn ey to good gove rnance and sustainable ■ In community development or in poverty human development. alleviation activities, Patton’s “Development E v a l u a t i on” ( w h e re ev a l u a t i on is used to continuously monitor change and with lessons R E F E R E N C E S learnt make adjustments to achieve goals) and Alauddin, M. and Tisdell,C. (1989), “Poverty Resource Distribution Ch a m b e r ’s “Em p ow e r ment Evaluation” (w h e r e and Security. The Impact of New Agricultural Technology in Rural Bangladesh”,Journal of Development Studies,Volume 24 beneficiaries are included in the evaluation Number 4, Pages 510 -570. process to drive project process to intended The Asian Development Bank (1993) Country Synthesis Report on results) methodologies should be given high Post Evaluation findings. The Asian Development Bank. Manila, 1993. 9 priority in ECD agenda. Chambers,R. (1997) Whose Reality Counts? Putting the Last First. London Intermediate Technology Publications,1997. ■ Fut u r e ECD inputs in training should serio u s l y Khan, M. Adil (1989) A South Asian Region Study on Current consider (in addition to capacity building of Thoughts and Practices in Monitoring and Evaluation. The government training institutions) providing Economic Development Institute. The World Bank Washington D.C. 1989. su p p o r t to the Univer s i t i e s / re s e a r ch institutions Kh a n , M. Adil (1991) Initiatives in Efficiency: Experience of Malays i a in building capacities in evaluation training in Monitoring and Evaluation. Central Evaluation Office. Unites and research. Nations Development Programme. New York. 1991. Khan,M. Adil (1993) Generic Issues in Monitoring and Evaluation: Evaluation can be a highly political exercise. It is a What Works and What Does Not. Central Evaluation Office. game of truth. It can be both pleasant as well as United Nations Development Programme,New York,1993. u n p l e a s a n t .H ow ever , tr uth must be told trut h f u l ly, Patton,M. S. (1999) “Development Evaluation” Evaluation Practice, but constructively. Evaluators need to be sensitive 1994,15 (3),311-319. to the difficulties of implementers and be cons c i o u s Picciotto, R. (1999)” The New Challenges of Evaluation Capability of the environment within which they operate. It Development Programme” Evaluation Insights. Evaluation Co- operation Group Newsletter. June 1999 Volume 1,Number 1. must be remembered that role of evaluator is not United Nations Development Programme. Human development that of a development police but that of a develop- report,1999. UNDP. ment facilitator. More and more, evaluation should The World Bank (1993). Evaluation results for 1991. Operations be seen as a resource, a reform agent and as an Evaluation Department. The Wor ld Bank. Washington D.C. 1993 . arbitrator of development justice. Constructively Yee, Y.L.(1999). “The ADB’s first Country Assistance Programme used, evaluation can be very effective in delivering Evaluation”. E valuation Insights. E valuation Co-operation both equity and justice as well as efficiency. Group Newsletter. June 1999 Volume 1,Number 1.

7 These approaches also provide the general methodological framework for results-based M&E – which basically imply: (i) that the process must be seen as dynamic and flexible and not rigid;and (ii) that evaluation should be treated as a continuous planning tool and not merely a post- facto activity and institutionally linked as such.

ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 1 7 AID EVALUATION: A Donor Perspective 4 Niels Dabelstein, Head of Evaluation Sec re t a r iat DA N I DA ,O E C D / DA CWP

In this intervention I would like to address four issues:

The DAC Evaluation Group; Aid evaluation from a donor’s perspective; the move towards partner country responsibility and, what donors can do.

Bi l a t e r al and mul t i l a t e r al aid agencies have a long trad i t i o n of eva l u a t i o n beginning in the fifties and becoming institutionalised and systematic in the mid-sixties.In most donor countrie s , de vel o p ment aid agencies/ministries established forma l i s e d evaluation procedures long before domestically oriented public institutions.

In order to enhance the quality and utilisation of eva l u a t i o ns the Devel o p ment Assistance Co mmittee (DAC) of the Organisation for Econo mic Co-operat i o n and Devel o pm e n t (OECD), established a Working Party on Aid Evaluation already in 1982.

The purpose of the Working Party is to: i. Strengthening exchange of information and experience in order to: improve eva l u a t i o n activities; en c o u r age standardi s a t i o n of methodological and conc e p t u a l frameworks; and improve co-ordination in planning major evaluations; ii. C on t ributing to improved aid effe c t i veness through the synthesis of evaluation lessons; iii. Examining the possibility of launching joint studies of aid effectiveness; iv.Seeking ways to promote developing countries’ own evaluation capabilities.

Today, it has developed into the leading forum for exchange of information and ex p e r ience between donor agencies; for devel o p ment of methodologies; st a n d a rd i s i n g of terminology; and co-ordination of joint evaluation efforts.The WP-EV today co nsists of rep re s e n t a t i v es of eva l u a t i o n units in bilateral donor agencies of OECD member countries and the European Commission. In addition, UNDP, the World Bank, IMF, and the regional development banks (AsDB, AfDB, IADB) hold status as observers and participate as such in WP-EV meetings twice a year.

The Working Party has established an evaluation definition widely recognised and adopted by its members. Evaluation is:

“An examination as systematic and objective as possible of an on- g oing or comp l e t e d project or programme, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, developmental efficiency, ef fe c t i ve n e s s , impact and sustainability. An eva l u a t i o n should provide informa t i o n that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors.”

Ge n e ra l ly, aid eva l u a t i o n perfo r ms sever al functions : it provides lessons for managers and staff of ongoing aid programmes thereby leading to improvements in implementation; it establishes information and documents experience that can be used in planning and designing future aid prog ra m m e s ; it is used by managers, st ra t e gy - and policy makers for revising existing — and for devising new — aid strategies and policies; and finally aid evaluation delivers information on efficiency and

AID EVALUA T I O N : A Donor Perspective 1 9 effectiveness of aid programmes, and thus ensures ori e n t e d . The need for accountability, pa rt i c i p a t i on , accountability towards politicians and the public. transparency, rule of law has only recently been considered necessary for improving governance Go ver nment financed devel o p ment aid is expenditure pra c t i c e s . Mo re o ver , the demand for eva l u a t i o n come s of taxpayers money outside national borders. In more often from the donors community (which addition, aid is often spent through partners or co- has limited influence), than from the partner countrie s ’ o p e rating gove rnments over which the donor Parliament or taxpayers. gover nment has no formal cont ro l . This is a fea t u r e distinguishing aid evaluation from evaluation of It appears that credible evaluation is a function of most other publicly funded programmes. For that good governance i.e. demand for accountability reason alone, the accountability function may very mo r e than of eva l u a t i o n institution and profe s s i on a l well be the most important for aid evaluation. Not cap a c i t y. Recent yea r s ’ focus on public sector ref o rm , the least in periods of aid fatigue and pressure on on public expenditures management, on democrat i c gover nment budgets,co ntinued public support for aid ref o rm s , and on devel o p ment effec t i v eness has, I hope, ex p e n d i t u r e is contingent upon aid agencies’ ac c o u n t - increased the demand for evaluation as a necessary ability and their ability to document aid results. and useful element of good govern a n c e . Th e r e are many wa ys in which donors can support eva l u a t i o n cap a c i t y: D eve l o pment aid ev a l u a t i on has con t i n u o u s ly de veloped from an early focus on individual activities ■ Promote an agency ECB support policy or and projects to broader evaluations of sectors, s t ra t e gy, p a rt i c u l a rly in view of new aid pro g ra m m e s , and cross cutting issues or themes such forms being introduced,including programme as gender, environment, institutional development assistance for institution and capacity building and sustainability. Aid evaluations are no longer as part of good gove rnance initiatives at simply about evaluating efficiency, effectiveness national and sector levels. and impact of aid in terms of technical, economic, s o c i o - c u l t u ra l , i n s t i t u t i onal and env i ron m e n t a l ■ Advocate and stimulate the evaluation issue in factors, but about evaluating the combined effects co u n t r y dialogues and sector prog r amme assistance. on developing societies of the complex of political, economic and technical interventions by donors. ■ Provide techn i c al advice and advise on trai n i n g facilities and materials on evaluation issues. The emergence of new types of aid interventions and the inclusion of novel themes in development ■ aid pose new challenges to aid evaluators. Some of Su p p o rt the establishment of tw i n n i n g those cha l lenges are methodologica l :h ow to eva l u a t e ar r angements between other domestic eva l u a t i o n support to good governance, human rights, civil institutions and host country institutions. service reform, and privatisation. Others are wider ■ ranging: the current trend to move away from Develop consistent evaluation methodologies project aid and to more varied and flexible modes and terminology. of assistance within a sector framework frequently ■ addressing key policy and institutional problems. Co - o r dinate their eva l u a t i o n prog r ammes with host countries and other donors in order to These new trends and cha l lenges need to be addres s e d . optimise the use of res o u r ces and the cons t ra i n e d Donors alone should not bear the responsibility for cap a c i t y of recipient countrie s ’ eva l u a t i o n syst e m s . eva l u a t i o n alon e .M o re emphasis on joint eva l u a t i on s ■ would be one way. Another is increased emphasis Arrange joint-evaluations with a genuine par- on evaluation capacity building. Through such ticipatory approach, where the needs of both e m ph a s e s ,m a ny of the future eva l u a t i o n cha ll e n g e s parties are incorporated from the start and can be addressed. where the capacity building element is taken into account specifically. The developing countries, in their own interest, should take on these challenges and further devel- In my opinion joint evaluation is one of the most op evaluation capacities to assess the performance concrete tools to enhance evaluation capacity. To of the public sector to which development aid is, ensure that evaluations become efficient learning after all, a small input. t o o l s ,p romote good govern a n c e , enable the partn e r s to be fully accountable, and are cost effective they Evaluation institutions exist in many developing must be planned and executed jointly with the countries, but most have little impact on policy recipients. Ultimately, both the donors and the making and management decisions,partly because recipient authorities are jointly accountable both th e r e is little demand for independent and tran s p a re n t to the taxpayers in developing countries and to the evaluation. Governments are not yet fully results people of the developing countries.

2 0 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PART II Evaluation of Performance Management in the Public Sector

E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing RESULTS BASED MANAGEMENT IN THE DEVELOPMENT COOPERATION AGENCIES: A Review of Experience 5 Annette Binnendijk,Consultant,UNDP

OV E RV I E W

This paper provides a broad overview of some key concepts and elements of results based management (also called performance management), which has been a central feature of recent public sector reforms in the OECD countries. While most of the remarks made and issues raised apply generally to government agencies’ efforts to implement results based management, the particular focus taken here is from the perspective of donor agencies. Experience and examples are drawn primarily from USAID, although other donors’ experiences shared at a Workshop on Performance Management and Evaluation held in New York in October 1998 are also taken into account.

PUBLIC SECTOR REFORMS

Dur ing the 1990s, ma n y of the OECD countries have undertaken extensive public sector reforms in response to economic, social and political pressures. Budget deficits, structural problems, growing competitiveness and globalization, lack of public confidence in government, and growing demands for better and more responsive services and for more accountability have all been contributing factors. Popular catch phrases such as “Reinventing government”, “Doing more with less”,“Demonstrating value for money”,etc.describe the movement towards public sector reforms that have become prevalent in the OECD countries. Often, gover nment-wide legislation or exec u t i v e orders have driv en and guided the public sector reforms.

While there have been variations in the reform packages implemented in the OECD countries, there have been many common aspects. For example: ■ Focus on performance issues and on achieving results ■ Devolution of management authority and responsibility ■ Orientation to customer needs and preferences ■ Participation by stakeholders ■ Reform of budget processes and financial management systems ■ Application of modern management practices

RESULTS BASED MANAGEMENT

Perhaps the most central feature of the reforms has been the emphasis on improving performance – that is, on ensuring that government activities achieve de s i r ed res u l t s . A recent study of the experiences of ten OECD Member countrie s

RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 2 3 with introducing perf o rmance management Most OECD governments now see evaluation as showed that it was a key feature in the reform p a rt of the ove ra ll perf o rmance management efforts of all ten. (See In Search of Results: Public framework, although perspectives concerning the Management Practices, OECD, 1997). degree of integration and independence var y. The vi e w of eva l u a t i o n as a separate or specialized function, Pe rf o rmance management, also re fe r red to as but integrated into performance management has results based management, can be defined as a been gaining mome n t u m . This is reflected in PUMA’s broad management strategy aimed at achieving Best Practice Gui d e lines for Eval u a t i o n (O E C D ,19 9 8 ) im p o r tant changes in the way gover nment agencies which was endorsed by the Public Management operate, with improving performance (achieving Committee.The Guidelines state that “evaluations better results) as the central orientation. must be part of a wider performance management fra m ew o r k”. Less emphasis is placed on independence, A key component of results based management is and evaluation is seen as one of many instruments performance measurement, which is the process of used in the ove ra ll perf o rmance management o b j e c t i ve ly measuring how well an agency is framework. However, some degree of independent meeting its stated goals or objectives. It typically evaluation capacity is being preserved; such as the i nv o lves seve ral ph a s e s : e . g. , a rticulating and evaluations conducted by central evaluation offices agreeing on objectives, selecting indicators and or performance audits carried out by audit offices. setting targets, mon i t o r ing perfo r mance (colle c t i n g data on res u l t s ) , and analyzing those results vis-a-vis Evaluation is viewed as complementary to — and the targets. While performance measurement is in some respects superior to — routine perfo rm a n c e concerned more narrowly with the production or measurement techniques. For example, evaluation supply of performance information, performance allows for more in-depth study of performance, management is broader. It is equally concerned can cover a longer time period, may incorporate with generating management demand for perfo rm a n c e factors such as outcome or impact too difficult or information — that is, with its uses in policy, expensive to assess through on-going monitoring, program, and budget decision-making processes and can analyze causes and effects in detail. and with establishing organizational procedures, Nevertheless, some concerns over the future of mechanisms and incentives that actively encourage its use. In an effective performance management the ev a l u a t i on function re m a i n s . Establishing p e rf o rmance measurement systems are taking sys t e m ,a ch i e ving results and continuous improvem e n t considerable effort and resources, often without based on performance information is central to the additional funds being allocated for this purpose. management process. In situations of diminishing overall budgets, this may well lead to a competition for resources between performance measurement and evaluation THE ROLE OF EVALUATION IN activities within OECD government agencies. PERFORMANCE MANAGEMENT

The role of ev a l u a t i on vis-a-vis perf o rmance USES OF PERFORMANCE INFORMATION management systems in many cases was not i n i t i a lly cl a ri f i e d , and still remains som ew h a t Performance should not be measured for measure- vague. In part, this is because evaluation was well ment’s sake. Performance information, both from established in many OECD governments before performance monitoring and evaluation sources, the introduction of performance management and should be used. One key use is for transparent the new approa c hes did not necessaril y incorpo ra t e reporting on performance and results achieved to eva l u a t i on . New perfo r mance management techn i q u e s external stakeholder audiences. In many cases, were developed partly in response to perceived government-wide legislation or executive orders failures of evaluation; for example, that the uses of have recently mandated such reporting. Moreover, evaluation findings were limited relative to their such reporting can be useful in competition for costs. Moreover, evaluation was often viewed as a funds by convincing a skeptical public or legislature specialized function carried out by external experts that the agency pro g rams produce significa n t or independent units, w h e reas perf o rmance results and provide “value for money”. Annual m a n a g e m e n t , w h i ch inv o lves re f o rming core pe rf o r mance rep o r ts are often directed to ministers, management processes,was essentially the respon- p a rl i a m e n t , s t a k e h o l d e r s , c u s t om e r s , and the sibility of managers within the organization. general public.

2 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Performance information should also be used for 1. Identifying in clear, measurable terms the in t e r nal purp o s e s ,s u ch as for management decision- ob j e c t i v es or results being sought and devel o p i n g making and identifying areas for improvement. a conceptual framework for how the results This requires that performance information be will be achieved. i n t e g rated into key management systems and processes of the organization; such as in policy 2. Selecting indicators that will be used to measure f o rmu l a t i on , in pro j e c t / p ro g ram planning and progress towards each objective. management, and in budget allocation processes. 3. Setting explicit targets (i.e., planned results to be Performance budgeting is the term generally used achieved by specific dates) for each indicator to refer to the infusion of performance information that will be used to judge performance. into resource allocation processes. The concept of performance budgeting is essentially the process of 4. De veloping perfo r mance moni t o r ing systems to linking budget levels to expected (or actual) reg u l a r ly collect data on actual results achi e ved . results, rather than to inputs and activities. Many OECD countries are experimenting with a variety 5. Analyzing and reporting actual results vis-a- of approaches to more closely associate expected vis the targets (or other criteria for making performance with requested funding levels, as part judgements about performance). of their broader reforms efforts to become more re s u l t s - o ri e n t e d .( See OECD, Bu d g eting for Results: 6. In t e g r ating eva l u a t i o ns to provide comp l e m e n - Pers p e c t i ves on Pub lic Expenditur e Managem e n t , 19 9 5 ) . tary information on performance not readily A recent report by the U.S. General Accounting available from perfo r mance moni t o r ing syst e m s . Office reviews some of the many challenges facing U.S . gover nment agencies attempting to implement 7. Using performance information (from both p e rf o rmance budgeting, i n cluding a vari e ty of p e rf o rmance mon i t o ring and ev a l u a t i on performance measurement issues, cost accounting so u r ces) for internal management learning and sh o rt c om i n g s , and the essentially political nature of decision-making and for external reporting to budget allo ca t i o n pro c e s s e s .( See GAO, Perfo rm a n c e stakeholders on results achieved. B u d ge t i n g : Initial Age n cy Experiences Provide a Foundation to Assess Future Directions, July 1999) The first three phases or steps generally relate to a re s u l t s - o riented planning appro a ch , s om e t i m e s When performance information is used in internal refe r r ed to as strategic planning.The first five steps, management processes with the aim of improving together, are usually included in the concept of performance and achieving better results, this is pe rf o r mance measurem e n t .A l l seven phases comb i n e d often referred to as managing-for-results. Such ar e essential to an effec t i v e results based management actual use of performance information has often system. That is, integrating complementary infor- been a weakness of performance management in mation from both evaluation and performance the OECD countrie s . Too ofte n , gover nment agencies monitoring systems and ensuring management’s have emphasized performance measurement for use of this information are viewed in this paper as external reporting only, with little attention given critical aspects of results based management. to putting the performance information to use in internal management decision-making processes. These two major uses of performance information OTHER COMPONENTS OF m ay not be com p l e t e ly compatible with on e another, and may require different types of results RESULTS BASED MANAGEMENT data and collection methods.Satisfying both needs or uses without over-burdening the performance Other reforms are often associated with results management system remains a challenge. based management systems. Often, these other c om p onents act to stimulate or facilitate the use of performance information. Some of these organizational changes include:

KEY PHASES OF ■ Ac c o u n t a b i l i t y — instituting new mecha n i s m s RESULTS BASED MANAGEMENT for holding agency managers and staff account- ab l e for achieving results within their sphere Some key elements or phases of results based of control or influence (e.g., results-oriented management include: management cont r acts and personnel apprai s a l s ) .

RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 2 5 ■ Decentralization — delegation of authority effective and results-oriented. “Aid fatigue”, the outs to the field and down to the management public’s perception that aid programs are failing to level that’s being held accountable for results. produce significant development results, declining In other word s , e m p ow e ring managers at aid budgets, and the gover nment-wide ref o r ms have ap p ro p r iate levels with flexibility to shift res o u rc e s all contributed to the donor agencies’ recent efforts from poorer to better performing activities. to establish results-oriented management systems.

■ Client focus — consulting with beneficiary Thus far, the donor agencies have gained most groups concerning their preferences and satis- experience with establishing performance meas- faction with goods and services provided, and urement systems — that is, with the provision of being responsive to their needs and desires. performance information — and some experience with external reporting on results. There is less ■ Pa rti c i p a ti o n — including partners and documented experience with the actual use of stakeholders in all aspects of performance performance information for internal management me a s u r ement and management processes (e.g., decision-making. jointly setting objectives, defining indicators, c o ll e c t i n g, a n a lyzing and rev i ewing data, Results based management and measure m e n t conducting evaluations, and using the infor- processes may take place at various organizational mation for learning and decision-making). levels within the donor agencies. The first level, which has been established the longest and for ■ Reformed policies and procedures — new which there is most experience, is at the project policy and procedure directives for changing level. However, some agencies such as USAID the way the agency conducts its business (e.g., ha v e also rec e n t l y established more strategic country new requirements, roles and responsibilities level systems within their country operating units. for strategic planning, for perf o rmance Moreover, establishing performance measurement measurement and evaluation, and for use of and management systems at the third level — the pe rf o r mance informa t i o n in external rep o rt i n g corporate or agency-wide level — is now taking on and internal decision-making processes). urgent importance due to increasing public pres s u re s and government-wide mandates requiring annual ■ Supportive mechanisms — various ways of rep o r ting on agency-wide perfo r mance and res u l t s . assisting managers to effectively implement Effectively linking and aggregating performance performance management, such as providing me a s u r es across these various levels remains an issue. re e n g i n e e ring tra i n i n g, t e ch n i cal assistance services, new performance information data- The donor agencies face special cha l lenges and issues b a s e s , g u i d e b o ok s , tips and best pra c t i c e s in establishing their performance management series, and other management tools. and measurement systems, that are either unique or more pronounced than those confronting the ■ Cultural change — equally important for domestic government agencies. This can make successful implementation of results based establishing perf o rmance measurement sys t e m s management is tran s f o r ming the organizat i on a l mo r e complex and costly than norma l . For example, c u l t u re and attitudes (e.g. , re o ri e n t a t i on these agencies: towards achieving results rather than imple- ■ Wor k in many differ ent countries and cont e x t s . menting inputs and processes, an openness to ■ Have a wide diversity of projects in multiple learning from failures as well as successes, and sectors. a commitment to objective and transparent ■ Often focus on capacity building activities, performance reporting). wh i c h are harder to measure than direct servi c e delivery activities. ■ Are moving into new areas such as the envi- ron m e n t , w h e re there’s little perf o rm a n c e SPECIAL CHALLENGES measurement experience. ■ Often lack standard indicators on results that FACING THE DONOR A G E N C I E S can be easily aggregated across projects. ■ A re usually on ly minor actors affe c t i n g As has been the case more broadly for the public i m p a c t s , with consequent problems in sector of the OECD countries, the donor agencies attributing them to their agency’s activities. h a ve faced con s i d e rable external pre s s u res to ■ Typically rely on outcome and impact data reform their management systems to become more co l lected by partner countrie s , who have limited

2 6 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing te ch n i c al cap a c i t y and res o u rc e s , with cons e - used to attain them. (Se e quent quality, co ver age and timeliness prob l e m s . Figure 1). The logframe FIGURE 1: is based on a five-level PROJECT LOGFRAME In particular, these factors can complicate donor h i e ra rchy model with HIERARCHY LEVELS A N D agencies’efforts to aggregate results across projects assumed cau s e - e f f ect rel a - TYPES OF INDICA T O R S and prog r ams to higher organizat i o nal and agency- tionships among them, wide levels. with those at the lower Coal l evel of the hiera rchy con t r ibuting to the attain- Impact Indicators ment of those above . Thus, inputs are used to PERFORMANCE MEASUREMENT undertake project activi- AT THE PROJECT LEVEL ties that lead to the Purpose de l i ve r y of outputs (goo d s / Outcome Indicators Performance measurement at the project level is services), that lead to the concerned with measuring both a project’s imple- attainment of the proj e c t m e n t a t i on pro g ress and results ach i eve d . Tw o purpose that contributes broad types of project performance measurement to project goal. Outputs might be distinguished. (1) Im p l e m e n t ation measurem e n t Output Indicators is concerned with whether or not project inputs and activities are in compliance with design budgets, 2 . SELECTING INDICA T O R S wo rk p l a n s , and sche d u l e s , and (2) results measurem e n t is concerned with whether or not actual results are Next, performance indi- Activities achieved as planned. Results are usually measured cators are selected for Process Indicators at three levels — immediate outputs, intermediate m e a s u ring pro g ress in outcomes and long-term impacts. implementing activities and in achieving results. Wh e reas tra d i t i on a lly the emphasis has been The logframe provides Inputs mostly on implementation concerns, with the rise a hierarchical structure of results based management the focus is increa s i n g l y around which the indi- Input Indicators on measurement of results. Moreover, emphasis is cators are typically con- shifting from more immediate outputs to longer- st ru c t e d .I n d i c ators specify term outcomes and impacts. what to measure along a scale or dimension in orde r to gauge progress (e.g.,number of workshops held, Measuring performance at the project level can be percentage of farmers attending demonstration divided into five elements or phases, as briefly sessions, changes in crop yields, etc.). The relative importance of indicator types is likely to change outlined below. The importance of participatory or co ll a b o ra t i v e approa c hes in all phases of perfo rm a n c e over the project’s life cycle, with more emphasis on input and process indicators at first, then shifting measurement is stressed — that is, involving not to output, outcome (purpose-level), and impact on ly donor agency project managers but also ( go a l - l evel) indicators later on as the pro j e c t representatives from the implementing agency, the matures.Also, different management levels tend to p a rtner gove rn m e n t , the intended beneficiary place emphasis on different indicator types. For groups, and other stakeholders. This helps build example, project field staff will find input and ag r eement around the proj e c t ’s objectives and com- process indicators of most use, whereas project mitment to the perfo r mance measurement proc e s s . managers will be more interested in achievement of project outputs and purpose/outcomes. Senior 1 . FORMULATING OBJECTIVES agency officials will be interested in the longer- term and broader social and economic impacts of the project, which may not be evident until after As part of project planning, the project’s objectives the project is completed.These different intended should be cla r ified by defining precise and measurab l e uses and users need to be kept in mind when statements concerning the results to be achieved selecting indicators. (outputs. purpose, and goal) and then identifying the strategies or means (inputs and activities) for Often checklists of criteria are devised, against meeting those objectives.The project logframe is a which proposed indicators can be judged and favorite tool used for conceptualizing or modeling s e l e c t e d . For example, s ome com m on ly used a project’s objectives and the strategies that will be indicator selection criteria include:

RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 2 7 FIGURE 2: PROJECT PERFORMANCE MONITORING PLAN

(Format for Recording Key Aspects of Data Collection)

Type of Indicators and Data Frequency Responsibility Indicator Definitions Data Sources Collection and Schedule for Data Methods of Collection Acquisition

Impact Indicators

Outcome Indicators

Output Indicators

Process Indicators

Input Indicators

Risk Indicators

■ Valid — Does the indicator directly represent 3 . SETTING T A R G E T S the result it is intended to measure? Once indicators have been identified, actual baseline ■ Objective — Is the definition precise and values should be collected for each, ideally just unambiguous about what is to be measured? before the project gets underway. This will be important for gauging whether progress is being ■ Reliable — Is the data consistent or compara- made later. Often agencies also set explicit targets ble over time? for their indicators. A target specifies a particular ■ Practical — Can data be collected easily, on a value for an indicator to be accomplished within a timely basis and at reasonable cost? given timeframe. (For example, 200 workshops to be held before September 2001, contraceptive ■ Useful —Will the data have utility for deci- prevalence rate increased to 65% by 2003.) Targets sion-making and learning? help clarify exactly what needs to be accomplished by when.It represents a commitment and can help ■ Owned — Do stakeholders agree that this orient and motivate project staff and mangers to indicator makes sense to use? the tasks at hand.

Tradeoffs among these indicator selection criteria A natural tension exists between setting targets ma y exist. Prob a b l y the most importa n t , overa r chi n g that are high enough to make project managers con s i d e ra t i o n is that the indicators provide managers and staff stretch to achieve them, and yet low with the information they need to do their job. enough to be realistic and achievable. While on the one hand, indicator data should be of sufficient quality to be credible and ensure the If they are set unrea l i s t i ca l ly high and unattainable, right decisions are made, on the other hand it confidence and credibility will suffer and may even should be practical — timely and affordable. Care set in motion perverse incentives to hide or distort should be taken to limit the number of indicators the figures. Any information that helps to ground selected to the minimum needed to adequately a target setting exercise and ensure its realism is capture the key dimensions of a result. Keep the us e f u l . For example, it is useful to establish a baseline, performance measurement system simple to avoid identify histori cal tre n d s , seek implementing overb u r dening managers and staff with unnecessary agency views, survey expert opinion about what is data collection responsibilities. p o s s i b l e, rev i ew re s e a rch findings, or identify

2 8 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing benchmarks (i.e., compare what results have been patterns (e.g., typical variations in data collection achieved by similar projects with a reputation for s o u rc e s / m e t h o d s , f re q u e n cy of coll e c t i on , a n d high performance). assignment of responsibility) are summarized in Figure 3. As one moves to higher and higher l evels of the logf rame hiera rchy, t h e re is the 4 . MONITORING PERFORMANCE (COLLECTING DATA) tendency for data collection efforts to become more expensive, time-consuming, and technically Once indicators are selected and targets are set, c om p l e x .A l s o, th e r e is a tendency for data colle c t i o n actual data for each indicator is collected at regular ef f o r ts to be conducted less freq u e n t l y.The placement intervals. Implementation monitoring involves the of responsibility for data collection also tends to frequent, on-going recording of data on project shift from the implementing agency at the lower operations — e.g.,tracking funds and other inputs, levels to the donor agency and/or to the partner and processes. It involves keeping good financial government at the higher levels. accounts and field activity records, and frequent checks to assess compliance with workplans and Data on project inputs, processes, and outputs are budget. Results monitoring involves the periodic generated mostly by project staff and are based on co ll e c t i o n of data on the proj e c t ’s actual achi e vem e n t simple reporting systems updated frequently. Data of results — e.g., its short-term outputs, medium- on outcomes are generally collected periodically term outcomes, and long-term impacts.This type (e . g. ,a n n u a l ly) from low-cost rapid appraisal methods, of monitoring demonstrates whether a project is mini-surveys or consultations with project clients. moving towards its objectives. M e a s u ring impacts usually re q u i re con d u c t i n g expensive sample surveys or relying on already Project managers have found it useful to prepare existing data sources such as national surveys, p e rf o rmance mon i t o ring plans to re c o rd key censuses, registration systems, etc. Impact data are aspects of data collection,such as providing defini- usually only collected every few years or at the tions for each indicator, source and methods of projects beginning and end (or ex post). data collection, frequency/schedule for collection, and assignment of responsibility for collection. Data collection at the higher levels — especially Fig u r e 2 illu s t r ates a matrix format that USAID has at the impact level — is often considered beyond found useful for recording summary information the scope of the implementing agency’s normal about data collection plans. responsibility. Donor agencies will need to make special arrangements with partner country organi- Data colle c t i o n approa c hes vary according to level s zations with statistical/data collection expertise for of the project logframe hierarchy. These common conducting or adding-on to planned surveys. Since

FIGURE 3: CHARACTERISTICS OF PROJECT DATA COLLECTION EFFORTS

(By Logframe Hierarchy Levels)

Type of Data Collection Frequency of Organizational Indicator Method Data Collection Responsibility

Impact Indicators censuses and surveys, multi-year partner governments, national statistics donor agencies

Outcome Indicators customer surveys, annually donor agency and rapid appraisals, implementing agency consultations

Output Indicators project records quarterly, biannually implementing agency

Process Indicators project records weekly, monthly implementing agency

Input Indicators project records, weekly, monthly implementing agency financial accounts

RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 2 9 several donor agencies working in the same sector results (example: p e rcentage of students may share needs for similar impact-level data, it attending project schools who are female). would be useful to consider coordinating or jointly supporting these data collection efforts, to avoid ■ Customer satisfaction — how well project duplication of effort and to share costs. Moreover, o u tputs corre s p ond to client pre fe re n c e s to ensure valid and reliable data, su p p o r ting cap a c i ty - (example: percent of clients satisfied with building efforts may be called for as well. health services delivered).

At what level should the focus of performance ■ Ef fe c ti v eness — the extent to which results — monitoring be placed? Concentrating on just one outputs, outcomes, or impacts — are being le vel of the logfr ame hierar chy may have unintended, achieved as planned (targeted). even dysfunctional, consequences. For example, concentrating only on the output level may result ■ Attribution — the extent to which outcomes in “doing the wrong things well”. Concentrating and impacts can be attributed to outputs from only on higher outcome and impact levels may a particular project. lead to lack of basic monitoring information about project activities and services, and result in poor ■ im p l e m e n t a t i on . The answer appears to lie in taking Co s t - e f fe c ti ve n e s s — the rel a t i o nship betwe e n as comprehensive and balanced an approach as is pr oject costs and results attributable to the proj e c t . possible, within reason/practicality. ■ Sustainability — the capacity for results to Developing a more comprehensive performance extend beyond the formal life of the project. monitoring system that recognizes the need for performance information at various levels is least ■ Relevance — the continued appropriateness likely to lead to distortions. Moreover, as already of a project’s results to the needs of the target discussed, different stakeholder groups and man- p o p u l a t i on , the partner country’s nation a l agement levels will have varying interests in these de vel o p ment prio r ities and to the devel o pm e n t le vels of res u l t s , so satisfying ever yone means having agency’s corporate-level goals. a comprehensive system. Periodic reviews of performance data by project management will help alert them to problems or 5 .A N A LYZING AND REPORTING PERFORMANCE DATA shortcomings vis-a-vis plans and targets, which m ay lead dire c t ly to taking actions or signal Per iodic management rev i ew s ,a n a lysis and rep o rt i n g the need for more in-depth studies focused on of project perfo r mance moni t o r ing data most typi ca l ly specific performance issues. Routine performance emphasizes effectiveness in achieving targets, by m on i t o ring alone may not be adequate for c om p a ring actual results with planned re s u l t s . addressing some of the performance issues listed However, analysis of performance monitoring data above(e.g., cost-effectiveness, attribution), which may address a broad variety of issues. For example: be c ause of their long - t e r m nature and/or comp l e x i t y ■ m ay re q u i re special in-depth assessments or Economy — the relationship between costs evaluation studies. and physical inputs (i.e., an organization is ec on om i c al if it is purchasing inputs as che a p l y A number of donor agencies have established as possible). performance rating systems whereby managers, ■ Efficiency — the relationship between costs drawing on data from performance monitoring and outputs (example: cost per kilometer of s ys t e m s , judge their pro j e c t’s perf o rmance by road built). assigning a rating along a scale (e.g. , h i g h ly satisfactory, satisfactory, unsatisfactory, or highly ■ Productivity — relationships between inputs u n s a t i s f a c t o ry ) , against a number of cri t e ria and outputs (example: number of demons t ra t i on s (e.g., effectiveness, efficiency, relevance, sustain- handled per extension worker). ability, etc.). These performance ratings or self- assessments are typ i ca lly re p o rted to agency ■ Excellence/quality — producing high quality h e a d q u a rters in standard re p o rting formats at outputs (example: percent of units produced specific times, such as at project completion or in that meet technical standards). annual reports. A particularly useful characteristic of project performance rating systems is that they ■ Eq u i t y — the extent to which needy/disadvan- enable consistent comparisons and aggregation taged sub-populations have equitable access to across the project portfolio.

3 0 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from an International Contference in Beijing PERFORMANCE MEASUREMENT A conceptual tool that is being used by USAID for strategic planning and performance measurement AT THE COUNTRY LEVEL at the country prog r am level is the results fram ew o rk . A results framework is a graphic display of the A few donor agencies have developed performance strategies necessary and sufficient for achieving a measurement systems for broader country pro- significant or strategic development objective in a grams — defined as sets of related projects or developing country. The results framework relies activities sharing the same development objective on objective tree conc e p t s , and diagrams the logical within a partner country, us u a l ly at a national sector cause-effect relationships between activity outputs level. USAID pioneered this approach during the at the bottom, intermediate results or outcomes in mi d - 1 9 9 0 s ,a b a n d o ning its previous focus on proj e c t s the middle , and the strategic devel o p ment objective and moving towards more strategic and results- at the top. Thus, it embodies the development oriented country programming approaches as part hypotheses underlying multiple partners harmon i ze d of its broader reengineering reforms. st r ategies for achi e ving a shared devel o p ment objective.

The country program approach is a much more See Figure 4 for a hypothetical example of a results com p re h e n s i v e and strategic approa c h to perfo rm a n c e fra m ew o rk . Results fram ew o r ks should be devel o p e d management and measurement than the project via colla b o ra t i v e processes invo l ving all donor agencies approach. It focuses on a significant development and other development partners working towards objective within a country, usually a sector, sub- a shared development objective, ideally under the sector, or a crosscutting objective.Thus, the unit of leadership of the partner country government. an a l ysis is not a single project but a country prog ra m that typically includes many projects or activities Results frameworks are useful as strategic planning implemented by differ ent donor agencies and partn e r and management tools as well as structures for organizations over a relatively long time period. p e rf o rmance measure m e n t . T h ey help identify what prog r am strategies are necessary and sufficient Perf o r mance measurement fram ew o r ks and syst e m s to achieve a significant development objective, and developed at the country program level are thus then enable colla b o r ating partn e r s ,w o r king in harmo ny, com p re h e n s i v e, lon g - t e rm , mul t i - a c t i v i t y and mul t i - to sort out their individual re s p onsibilities or site endeavors that usually include many projects contributions to the overall strategy. This can help and diffe rent organiza t i onal actors/con t ri b u t o r s donor agency operating units to better align (focus within a given country sector or sub-sector. and concentrate) their assistance activities into

FIGURE 4: HYPOTHETICAL ILLUSTRATION OF A COUNTRY PROGRAM RESULTS FRAMEWORK

(Development Objective and IntermediateOutcomes)

Increased availability of food in domestic market

Markets constructed More efficient transportation Increased food production (Ministry of Agriculture) of goods to market

Improved Better More credit Improved Upgraded production tech- Irrigation available to farm to transport nologies available (Ministry of farmers market roads vehicles to farmers Agriculture) (DANIDA) (World Bank) (USAID) (CIDA)

RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 3 1 those program strategies for which they have taken many development partner actors — e.g., various res p on s i b i l i t y, rather than just have a diverse portf o l i o donor agencies, the NGO community, and of of seemingly unrelated pro j e c t s . The country course the partner country government. Some of development objectives and intervention strategies the tools developed for country program level selected by a unit usually have to be in line with strategic planning and performance measurement the donor agency’s overall corporate goals and should be particularly well suited to new modes of areas of comparative advantage. de vel o p ment assistance based on joint mul t i - d on o r / partner sector programs in which investments and The fram ew o r k is also a perfo r mance measurem e n t activities are harmon i z ed to achi e ve shared country tool — providing a structure for measuring and development objectives. monitoring progress towards the achievement of those results for which the unit is responsible. While this approach holds considerable promise, Performance data from the monitoring system is its actual use thus far may be falling short of its used to alert managers when actual results are not po t e n t i a l . For example in USAID, co u n t r y operat i n g meeting targets as planned,indicating the need for units have too often tended to develop results adjustments to be made in relevant projects and frameworks in relative isolation and from their ac t i v i t i e s . It may be useful to occas i on a l ly supplement own agency perspectives. mo r e routine revi e ws of the perfo r mance moni t o ri n g data with complementary strategic, program-wide While there is typically some limited participation evaluations that assess the relative effectiveness by their implementing agency partners and immediate and cost-effectiveness of alternative strategies and stakeholders,the focus is usually on USAID’s own activities for achieving the development objective. programs and strategies, rather than placing equal focus on all relevant partners’ programs. Whereas the project approach puts equal weight on moni t o r ing all elements of the logfr ame hierar chy, and may even trad i t i on a l ly have favored implemen- ta t i o n monitoring, the country program results PERFORMANCE MEASUREMENT f ra m ew o rk puts the higher-order deve l o pm e n t ob j e c t i v e and intermediate outcomes at center-stage. AT THE AGENCY LEVEL It is less concerned with defining the individual project means (inputs/processes) and outputs, and Largely driven by domestic public pressures and muc h more conc e r ned with measuring and achi ev i n g government-wide legislation for annual reporting the higher-level results. The shift from individual on agency performance, the donor agencies — like projects to programs also implies a different time- gover nment agencies more general ly — are cla ri f yi n g fr ame dimension ,f reed from the confines of a single their overall goals and seeking ways to summarize project’s life cycle. By focusing on country level their achi e vements vis-a-vis those go a l s .M e a s u ri n g de vel o p ment objectives and intermediate outcome s , and reporting on results at the agency-wide level the timeframe now becomes lon g e r - t e rm ,o u t l i v i n g poses a significant challenge for the development the comings and goings of individual project activities. ag e n c i e s .T h e y face a number of obstacles in attempting to aggregate results, some of which are either Individual project activities tend to be less well defined unique to or comp l i c ated by the nature of devel o pm e n t in this approach,allowing for more flexible designs cooperation work (See previous discussion). and implementation, rather than rigid “blueprint” approaches. Moreover, in USAID, headquarters Agencies such as USAID have recently developed no longer approves projects. Instead, authority is and issued policy papers or strategic plans that delegated to operating units in the field so they can clearly articulate the agency’s overall mission and s h i ft course mid-stream if results mon i t o ri n g the key development goals or priority areas on information indicates certain activities are not which they will concentrate. Usually the agency working well. Nevertheless, it is important to be goals are sector-oriented (e.g., better education, able to link individual project activities, outputs improved health, etc.) although some may be and their associated costs within the bro a d e r crosscutting special concerns (e.g.,gender equality, results frameworks. partnerships) or internal management efficiency goals. These statements about agency goals serve This country prog r am level approa c h puts a prem i u m to articulate to external audiences what the overall on partnerships and more colla b o ra t i v e approa ch e s , aims of the devel o p ment assistance prog r am are, an d since achieving a strategic,long-term development provide a framework or structure for gathering and objective is clearly dependent on the activities of reporting data on overall agency results achieved.

3 2 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing This is viewed as important in an era of declining contribute towards achieving and how it intends aid budgets, increasing competition for funding, to contribute. The hierarchies serve as detailed and growing public skepticism about the effective- st ru c t u r es for rep o r ting on agency results at severa l ness of development aid. Clarifying agency-level levels. For example, a typical three-level hierarchy goals has also been useful as an internal management structure might include agency goals, sub-goals, tool for strategic planning — that is, for focusing and supporting program approaches. USAID has and concentrating the agency’s assistance portfolio also found it useful to present their strategic and resources within priority goal areas.In the case framework as graphic diagrams or visual displays, of USAID, country operating units have been using objective tree concepts. Figure 5 provides an asked to align their country strategic objectives example for the environmental goal from USAID’s and program activities within the new USAID strategic framework. goal structure. C o llecting data for agency-wide perf o rm a n c e USAID has further elaborated its agency level assessments and reporting may take place from goals into seve ral sub-ca t e go ri e s , f o rming a several basic sources;(1) from existing internation- multi-level framework or hierarchy of objectives. al sources/databases that maintain country level Such a multi-level strategic framework can serve statistics on sector development trends, (2) from to clarify even further what the agency aims to the proj e c t / p ro g r am perfo r mance measurement systems

FIGURE 6: HYPOTHETICAL ILLUSTRATION OF A COUNTRY PROGRAM RESULTS FRAMEWORK

(Development Objective and Intermediate Outcomes)

Agency Goal 4 Environment managed for long –term sustainability

Agency Agency Agency Agency Agency Objective 4.1 Objective 4.2 Objective 4.3 Objective 4.4 Objective 4.5

Biological Global climate Sustainable Increased Sustainable diversity change threat urbanization provision of natural resource conserved reduced promoted environmentally management and pollution sound energy reduced services

Agency Program Agency Program Agency Program Agency Program Agency Program Approaches Approaches Approaches Approaches Approaches

1) Improve 1) Reducing 1) Increasing 1) Increasing 1) Managing management greenhouse access to energy forests of protested gas emissions water and efficiency sustainability areas from energy sanitation 2) Increasing 2) Managing 2) Promoting use services use of water sustainable 2) Reducing 2) Promoting renewable resources use of net green- improved energy 3) Practicing biological house gas urban 3) Introducing agricultural resources emissions management innovative sustainability 3) Supporting from land use 3) Promoting clean 4) Managing ex-situ 3) Assisting pollution technologies coastal conservation adaptation prevention zones of genetic to climate and control sustainability diversity change

RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 3 3 FIGURE 5: USAID’S STRATEGIC FRAMEWORK FOR THE ENVIRONMENTAL GOA L

Core Set of Indicators Core Set of Indicators

Economic well-being Reducing extreme poverty 1. Incidence of extreme poverty The proportion of people living in extreme poverty in 2. Poverty gap ratio developing countries should be reduced by at least 3. Poorest fifth’s share of national consumption one-half by 2015 4. Child malnutrition

Social development Universal primary education 5. Enrolment in primary education There should be universal primary education in all 6. Completion of primary education countries by 2015 7. Adult literacy rate

Gender equality Progress towards gender equality and the empowerment of women should be demonstrated by eliminating gender disparity in primary and 8. Gender equality in education secondary education by 2005 9. Gender equality in adult literacy

Infant and child mortality The death rates for infants and children under the age of five years should be reduced in each developing 10. Infant mortality rate country by two-thirds the 1990 level by 2015 11. Child mortality rate

Maternal mortality The rate of maternal mortality should be reduced 12.Maternal mortality ratio by three-fourths between 1990 and 2015 13.Births attended by skilled health personnel

Reproductive health and population Access should be available through the primary health care system to reproductive health services for all individuals of appropriate ages, no later 14.Contraceptive prevalence rate than the year 2015 15.HIV prevalence rate

Environmental sustainability and regeneration Environment There should be a current national strategy for 16.Countries with national environmental plans sustainable development,in the process of 17.Access to safe water implementation, in every country by 2005, so 18.Intensity of fresh water use as to ensure that current trends in the loss of 19.Biodiversity: Land area protected environmental resources effectively reversed at 20.Energy efficiency both global and national levels by 2015 21.Carbon dioxide emissions

maintained by the agency’s country operating units, results data across pro j e c t / p ro g ram port f o l i o s . or (3) from the agency’s evaluation reports. In Results of these agency-wide analyses of aggregate most cases, these data are entered and stored in pro j e c t / p ro g r am perfo r mance and results are usually automated, central agency databases to facilitate reported in annual performance reports. agency-wide analysis and reporting. Donor agencies have a number of basic options to Co mputer databases and softwa r e prog r ams facilitate co nsider for aggregating or summing up perfo rm a n c e data sorting, aggregation, statistical analysis and and results achi e ved at the agency-wide or corpo ra t e gra p hic pres e n t a t i o n of re s u l t s .T h ey can grea t l y aid level. At the two extremes — project outputs and the work of analyzing large amounts of perfo rm a n c e / country-level sector statistics — aggregation of

3 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing indicator data may be relatively easy. But in the ag e n c i e s . On the other hand, th e r e are some serio u s case of outputs, the question “so what?” may be issues with using this approach, especially in the raised. With country level statistics, it is rarely c ontext of re p o rting on agency perf o rm a n c e . possible to link changes cre d i b ly to a single Attempting to link and attribute these country- agency’s interventions, especially on a year-to-year le vel and global-level socio-econo mic improvem e n t s basis. In the middle are project outcomes, which to the activities of a single donor agency is a wide should be both significant yet have clearer linkages stretch of the imagination that many will question. to agency activities than national statistical trends. Donor agencies using this approach would be The problem here is that often there is great diver s i t y advised to adopt the goals and indicators from among in projects’ objectives and in their performance those agreed to by the international community (as me a s u re s , so aggregating across standard indicat o r s articulated in the DAC report, Shaping the 21st is often not possible. Some agencies have overc om e Century: The Contribution of Development Co-oper- this by developing rating systems that score a proj e c t ’s ation, 1996). Moreover, their reports might clarify success in meeting its objectives and then summing that these results are the consequence of many across projects the numbers and percentages that partners’ contributions, and cannot be attributed we r e successful or unsuccessful in achi e ving outcome s . to individual agencies. (See Figure 6). Another complication is that agency performance reporting These three basic options for aggregating results is usually required annually, whereas data on coun- are discussed below. try development trends is often only available at intervals of several years apart. Moreover, even if it was available annually, the long-term nature of 1 . SELECTING THE PROJECT OUTPUT LEVEL development improvements at this level means FOR AGENCY-WIDE REPORTING ON RESULTS year-to-year changes may not be significant.

Outputs of projects (such as number of units of goods and services delivered or numbers of 3 . SELECTING THE PROJECT/PROGRAM OUTCOME LEVEL beneficiaries/clients reached) are relatively easily FOR AGENCY-WIDE REPORTING OF RESULTS. summed up across similar types of proj e c t s ,a s s u m i n g they are comparable. For development agencies In between project outputs and macro-statistics, with fairly centralized structures and a standard set there’s the level of intermediate outcomes. of project approaches with comparable outputs ( go o d s / s e rvices) across country settings, t h i s A major advantage is that performance monitoring approach may be feasible. However, reporting at sy stems at the project or prog r am level are general ly the output level will only be valuable to the extent a l ready established and thus data on pro j e c t / that the intended external audiences/stakeholders pro g r am outcome achi e vement should be available. will be impressed with this level of results. If the A key problem with aggregating project outcomes response is “Is that all we’re achieving?”, summing for agency-wide reporting is the typically great and rep o r ting on outputs may be counterpro d u c t i v e di ve r s i t y of outcomes and their indica t o r s ,e s p e c i a lly in terms of defending the aid program before in decentral i z ed agencies such as USAID. Wit h o u t parliament or the taxpaying public. standard indicators of project/program outcomes, direct aggregation is not possible. Agencies may deal with this problem in differ ent ways. For example, 2 . SELECTING LONG-TERM SECTOR DEVELOPMENT T R E N D S by developing standard outcome indicators for FOR AGENCY-WIDE REPORTING OF RESULTS common “program approaches” (groupings of like projects). However, this approach is only advisable Another option for reporting on results achieved for more centralized agencies with fairly structured at the corporate level is to report on long-term pro g r am approa ch e s . Another approa c h for getting social and economic improvements at the country around this incomparability problem is to devise sector and global levels using interna t i o nal statistical rating systems that score a project’s success in d a t a s e t s ,w h i ch have some measure of comp a ra b i l i t y meeting its objectives . The agency can then aggreg a t e a c ross countri e s . Advantages of this appro a ch across projects within an objective or program area in c lude its appeal in terms of rep o r ting on significan t with statements like “85% of projects or programs impacts that matter to stakeholders, (e.g., alleviate aimed at improving child survival have successfully poverty, reduce infant mortality, achieve universal met or exceeded their outcome targets”. Issues primary education) and the ready availability of with this approach may include the extent to i n t e rn a t i onal indicator datasets cove ring many which standard criteria for making judgements (not all) of the sector concerns of the development about scores are applied across projects, and the

RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 3 5 rel i a b i l i t y of “se l f - a s s e s s m e n t ” ratings especially when managers’ project/program performance self- managers may fear the consequences of poor scores . ratings, etc.) Typically performance data are presented in standard, comparable formats None of these options for aggregating results to that can be easily entered into databases and the agency-wide level appears to be ideal, and the analyzed across the portfolio. They are meant donor agencies face considerable challenges in to provide consistent types of information their current efforts to summarize and report covering a broad range of performance issues performance at the corporate level. Given this and res u l t s , but without great depth of analysi s . state-of-the-art, agencies may benefit from mixing Evaluations, on the other hand, usually are or balancing the various appro a ches available. less standardized and follow individual scopes How to best co-ordinate, synthesize and integrate of work.Moreover they tend to focus on fewer findings from ev a l u a t i on re p o rts into annual p e rf o rmance issues but analyze them in a g e n cy perf o rmance re p o rts is another issue greater depth. needing attention. ■ Performance monitoring reports focus mostly on whether or not results were achieved as p l a n n e d , w h e reas ev a l u a t i ons can better THE ROLE OF EVALUATION VIS-A-VIS explain why and how they were achieved or no t . In other words , eva l u a t i o ns seek to analyze PERFORMANCE MEASUREMENT and understand the projects or pro g ra m’s context and factors influencing performance, Perf o r mance measurement and eva l u a t i o n are general ly both internal (within managers’ control) and viewed as two distinct but complementary sources external (beyond managers’ control). of performance information, both of which are necessary for effective results based management. ■ Routine performance monitoring can ser ve as an early warning system to alert managers when In USAID, performance monitoring is defined as there are performance shortfalls. However, a process of collecting and analyzing data to measure they do not assess the causes of the shortfalls the performance of a program, process or activity nor make recommendations for appropriate against expected results (targets), wh e r eas eva l u a t i o n management actions, as do evaluations. is defined as a relatively structured analytical effort un d e r taken selectivel y to answer specific management ■ Because of timing as well as the need to use questions regarding programs or activities. Some more rigorous methods and in-depth analysis, further distinctions often made by agencies follow: some performance issues, such as long-term i m p a c t , a t t ri b u t i on , c o s t - e f fe c t i ve n e s s , a n d ■ Pe rf o rmance mon i t o ring re p o rts are self- su s t a i n a b i l i t y, can prob a b l y be better addres s e d assessments by project or program managers, by evaluation than by routine performance whereas evaluations are typically conducted by monitoring reports. larger evaluations teams, often comprised of external evaluators that can provide an inde- Thus, evaluations and performance measurement/ pendent judgement about pro j e c t / p ro g ra m monitoring can be viewed as distinct but comple- performance. However, trends towards more me n t a r y functions . Both are management tools. Bo t h participatory forms of evaluation in some are important sources of performance information agencies may make this less of a distinction. that together can cont r ibute to management learni n g and decision-making processes and to external ■ Performance monitoring reports are typically performance reporting requirements. mandatory for larger projects or programs and thus provide a reasonably complete coverage However, there is some concern that performance of the overall portfolio, whereas evaluations measurement and reporting tasks, often required are often conducted on a much more selective by government-wide law or executive orders, may (i.e., occasional, optional) basis for projects/ be “cr owding out” eva l u a t i on s . That is, th e y may be programs of particular interest or concern. co mpeting for the same, in c re a s i n g l y scar e staff and other res o u rc e s . For example, a survey of eva l u a t i o n ■ Perf o r mance rep o r ts invo l ve rel a t i ve l y strai g h t - offices within the U.S . fed e r al gover nment agencies forward presentations of performance data found this to be a concern (See U.S. General (e.g.,actual results achieved vis-a-vis expected Accounting Office, Program Evaluation: Agencies res u l t s , actual expenditures data vis-a-vis budgets, Challenged by New Demand for Information on

3 6 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Program Results, April 1998). Typically, agencies A special type of decision-making that perfo rm a n c e have not been given additional funding to establish information is increasingly being called upon to their performance management and measurement influence is resource allocations. In USAID, for systems, while overall agency budgets have often example, country operating units must submit been on the decline. annual reports (called Results Reports and Resource Requests) that ties their request for funds to the These overall trends and concerns are evident in results they expect or plan to achieve with those USAID.For example, since the mid-1990s when res o u rc e s . The rep o r ts also contain self-assessments ree n g i n e e r ing ref o r ms mandated the establishment by operating units of their actual prog r am perfo rm a n c e of performance management and measurement (extent to which results targets were achieved) over sys t e m s , the size of USAID’s central eva l u a t i o n office the previous year. Their performance ratings are staff and resources has declined rapidly. Moreover, then ranked across countries and programs, and the number of evaluations conducted by USAID’s this information further influences the budget country operating units also declined sharply, from allocation process across countries and programs. 489 reports in FY1994 to 70 reports in FY1998. On the margin, better performing programs tend to get more funds, and poorer perfo r ming prog ra m s get less. However, performance remains a compar- atively minor factor in these allocation decisions, KEY USES OF PERFORMANCE INFORMATION outweighed by predominant foreign policy criteria and constrained by Congressional earmarks. In results based management sys t e m s ,p e rf o rm a n c e information (drawn from both performance measurement and evaluation sources) serve two primary EXTERNAL REPORTING (ACCOUNTABILITY-FOR-RESULTS) uses or purposes. One use is as an internal management tool for making program improvements; The second key use of performance information is the second is for external accountability reporting. to rep o r t agency perfo r mance to various stakeholder audiences. Donor agencies, like other domestic gover nment agencies, ar e accountable for achi ev i n g INTERNAL MANAGEMENT IMPROVEMENT and reporting results to the taxpaying public and ( M A N A G I N G - F O R - R E S U L T S ) their elected representatives, and to designated central oversight agencies. Often there are now This first intended use of perfo r mance informa t i o n government-wide legal requirements for reporting is for continuous feedback to managers about the results, at certain times and in specific formats, results they are achieving, so they can then use the which are being audited by oversight agencies. information to improve their performance even Moreover, overall agency accountability for results m o re . Sometimes discussions of this internal is increasingly being devolved and translated into management use are further sub-divided into two accountability at lower organizational levels (e.g., related aspects or processes — promoting learning op e r ating units, te a m s , or even individual managers). and facilitating decision-making. USAID for example is experimenting with management contracts and personnel appraisal systems (a) Promote Le a rn i n g. Pe rf o rmance informa t i o n that specify what results are to be achieved, when, promotes continuous management learning and by whom. In addition to being responsible to about what results are being achieved by their domestic stakeholders, the donor agencies are also pro j e c t s / p ro g r ams and why — i.e., what factors accountable to their partner country governments are influencing good or poor performance. and ultimately to their intended beneficiary grou p s . I m p roved knowledge is a pre requisite for better decisions. POTENTIAL CONFLICTS AMONG USES (b) Facilitate Decision-making. Management’s le a r ning in turn facilitates their making approp ri a t e As experience with using perfo r mance informa t i o n de c i s i on s .C o ntinuous perfo r mance informa t i o n grows, the potential for conflict between its two about progress towards results and about the key intended uses is emerging. Managing-for- factors influencing performance will facilitate results implies a shift in focus in from inputs and good decision-making and timely action. Les s on s processes to outputs, and from outputs to even from experience can help agency managers higher outcomes and impacts. Not only is it to con t i n u a lly improve their deve l o pm e n t important to know what results are being achieved assistance projects and programs. at these different levels,but also to understand the

RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 3 7 cause-effect linkages between them, e.g., why an science research. Moreover, it would favor a self- activity is successful or not, which approaches assessment appro a ch to ensure management’s wo r k better, and under what cond i t i o ns or cont e x t s . ownership and a first-hand learning experience, Emphasis on accountability-for-results, however, and also would encourage more part i c i p a t o ry may ironically shift focus back down to outputs, methods and stakeholder invo l vem e n t . In con t ra s t , which can be more easily attained and attributed an audit/accountability approach might either call to agency activities, and for which data can be easily for more independent assessments or for a system co l lected on an annual basis. Managers have rel a t i ve l y of spot-ch e ck s , rev i ews and ve ri f i ca t i on of greater control over outputs and thus are under- management self-assessments. s t a n d a b ly more com f o rtable with being held accountable for this lower level of results than for These conflicting aims present a dilemma for outcomes or impacts over which they have less donor agencies, as it does for other government i n f l u e n c e . M o re ove r, o u t c omes and especially agencies implementing results based management. impacts are longer-term changes that may not Both uses should be kept in mind when establishing show improvements quickly or annually. Since performance measurement and evaluation systems. p e rf o rmance re p o rting is genera lly con d u c t e d To the extent possible, the systems will need a n n u a lly, this further encourages managers to to address both uses and mix or balance data focus and report on lower-level results that will collection and analysis approaches to satisfy both sh o w changes faster. Furt h e rm o r e, th e r e is a growi n g interests. For example, an independent series of concern among auditors and oversight agencies central impact evaluations might be undertaken with attributing results to agency interventions. to address auditor’s concerns about attribution, Since demons t r ating attrib u t i o n becomes increa s i n g l y while managers in the field might be encouraged difficult for higher-order outcomes and impacts, to conduct self-assessments employing more rapid this also acts to encourage managers to focus and appraisal and participatory techniques. re p o rt at lower results leve l s . Fu rt h e rm o re, ac c o u n t a b i l i t y rep o r ting tends to empha s i z e measurin g Another potential conflict among performance what is being achi e ved (and comp a r ing it to pre- s e t information uses is more unique to the donor ta r g e t s ) , rather than analyzing why or how it is being agencies. Donor agencies are accountable not only achieved.In contrast,a management improvement to domestic stakeholder audiences but also to the approach is equally concerned with analyzing the partner country stakeholders. context and factors influencing performance, and with drawing lessons for improving performance. To the extent that donor agencies’ performance m e a s u rement and re p o rting systems may vary Ac c o u n t a b i l i ty re p o rting versus management considerably for one to the next, partner country i m p rovement uses also implies diffe rent data governments will have to deal with trying to collection and analysis approaches. For example, coordinate, compare and make sense of widely a t t ributing outcomes and impacts to specific different donor agency approaches, frameworks, agency interventions requires rigorous designs and indicators, data, etc. Harmonization among donor data collection methods. It also implies extensive agencies of their performance measurement and attention to data quality, validity and reliability, reporting systems,particularly at the country level, and to independent verification. On the other would lessen the burden on partner country organ- h a n d , a management improvement appro a ch izations. However, the extent to which this can be would tend to emphasize more rapid and low-cost accomplished may be limited given the variation in data collection/appraisal techniques, with data of government-wide performance reporting systems sufficient quality for decision-making needs but that have evolved in different OECD countries not necessarily up to standards required for social that may dictate the donor agencies’ approaches.

3 8 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing EVALUATION CAPACITY DEVELOPMENT IN THE PEOPLE’S REPUBLIC OF CHINA: Trends and Prospects

6 Ray Rist,Evaluation Advisor,The World Bank Institute

I N T R O D U C T I O N

Evaluation is relatively new in the People’s Republic of China (PRC). Indeed, be f o r e the early 1980s, it was unknown there. This unfamiliarit y with eva l u a t i o n reflected the orientation of the social science at that time, the virtual absence of any evaluation literature published in Chinese, and the lack of systematic contacts by chinese with those practicing evaluation in other parts of the world. Some activities under way, however, whithin the PRC did come to resemble evaluation, in c luding some policy analys i s ,e c on omic and management studies, su r vey res e a r ch, pr oject comp l e t i o n revi ew s , and what was broa d ly termed “ex p e r ience summariza t i o n”. But these efforts were not called evaluation, nor were they systematic or focused on the issues now encompassed inour understanding of evaluation.

In the 1950s, the PRC established policy and economic research institutes in national and ministerial, provincial and even some county governments. Over the past nearlyt 50 yea r s , these institutes have undertaken a wide varie t y of studies using an array of analystic methods. Early work was largely economic, but the later work has branched out into studies of financial systems, social affairs, environmental protection, and sustainable development, to name four.

Although there are few formal studies of the use of material from these institutes in national planning, a general consensus is that the institutes have been helpful. The policy community is aware of their work and sees them increasingly as sources of pertinent data of analysis. The material is largely anecdotal, but some important findings from the institutes have directly affected government decision making, especially after the economic reforms and openings to the world in 1978. As the ref o r m movement in the PRC has grown over the past 20 yea r s , a number of studies in agriculture, fiscal and tax policy, financial policy, foreign tra d e , and enterpr ise management have cont r ibuted signican t l y to the formul a t i o n of reform policies.

As the impetus for development grew from the 1950s to now, the PRC built methodological experience in how to study development projects and programs. It focused on the technology, engineering, and cost-effectiveness of development initiatives such as dams, highways, bridges, power plants, construction projects, irrigation, heavy industry, and railroads. Studies of these projects examined cost and quality cont ro l , financial benefit, and compliance with devel o p ment objectives .

As the 1978 reforms took hold, more capital and development assistance came into the PRC. As res e a r chers built contacts elsewh e r e, com p re h e n s i v e and reg u l a r evaluations were more often undertaken. The past 18 years have seen a growing capability in and understanding of technological and engineering analysis,

E V A L UATION CA PACITY DEVELOPMENT IN THE PEOPLE’S REPUBLIC OF CHINA: Trends and Prospects 3 9 e nv i ronmental impact analysis and modeling, undertaken evaluations, of many large nationally social impact analys i s , env i r onmental analys i s , entities and line ministri e s . The CIECC is su s t a i n a b i l i t y analysi s , and implementation studies, systematically studying the policies and methods though in paractice not all of these are equally of evaluation, training staff from mulitple and conducted in a satisfactory manner. Building skill handbooks on evaluation practices. and experience in these areas for the PRC has been undertaken via a two-way flow of information and By June 1998, the CIECC had completed eva l u a t i on s expertise — many persons from Cchina have gone of 58 national government projects and also a elsewhere to study these techniques, and many l a r g e - s cale ev a l u a t i on study of the projects consultants and experienced researchers in these constructed in accordance with the eighth state areas have traveled to China. fi ve - y ear devel o p ment plan (spanning from 1990 to 1995). It is now assisting the Asian Development Bank (ADB) to evaluate the perf o rmance of ADB’s whole portfolio in China. These studies THE PRESENT SITUATION: can be characterized as generally a combination of performance measurement,financial analysis, co s t - THE MAJOR A C T O R S benefit assessments, implementatin moni t o ri n g , and techn i c al and engineering analysi s . The CIECC The driving force for evaluation in China is the to date has done few systematic ex-post or impact ma s s i v e and sustained surge in national devel o pm e n t studies of large-scale national projects. and econ omic growth (the annual GDP has increased by 8% per year for the last 8 years). The A recent significant development is thatt, during attention and capability of the country to address the current government restructuring, the SPC evaluation questions comes form this concer with has been re s t ru c t u red as State Deve l o pm e n t development. Although most evaluation questions Planning Commission (SDPC) and the CIECC is c omes from this con c e rn with deve l o pm e n t . transferred from being the executing arm of the Although most ev a l u a t i on is ex-post pro j e c t SPC into an advisory group of the State Council. assessment, there is increasingly recognition that Compared to the SPC, the SDPC focusses much mo r e on macro econo mic reg u l a t i o n and long - t e r m evaluation issues are also embedded in all stages state deve l o pment pro g ra m m i n g, and less on of the development project cycle. For this reason, annual investment planning. As a matter of fact, there is a growing awareness within China that the the annual investment plan of the SDPC covers evaluation function is applicable at all stages of only state financed large projects. the project cycle. There is now interest in linking evaluation to project and program formulation and In ev a l u a t i on ,h ow ever , the SDPC has an importa n t implementation, and some ongoing evaluation has n ew duty of mon i t o ring and supervising the already been undertaken, though comprehensively implementation process of state key projects. The doing so still infre q u e n t . ( c f.Va l a d ez and Project Sup e rv i s i o n Office of the SDPC, com p ri s i n g Bamberger, 1994). mainly senior officials,is set up exclusively for this ob l i g a t i on . It has not yet been determined whether Before the reorganization of the government that the SDPC will conduct evaluation as the SPC did is now under way, the major national agency in before, but it is clear that if it will, only state China for project and program evaluation was the financed projects will be focused, and those proj e c t s State Planning Commission (SPC), whose major involving financing from bilateral and multilateral fu n c t i o ns also include formulating macro reg u l a t i o n sources. It will not evaluate any projects wholly policies,preparing what large and medium projects financed by local governments with no funding need to be cons t r ucted each year all over the country. from foreign sources. This means that from now The SPC organized evaluation studies through on,local projects will need to be evaluated by local governments themselves. its Ke y Cons t ru c t i o n Departm e n t . The SPC or g a n ize d eva l u a t i o n studies direc t i ve s , es t a b l i s h e d The new functions of CIECC have not yet been policies and guilines for eva l u a t i on , and planed what defined. But it can be presumed that its role in pro j e c t s or pro g r ams need to be evaluated and when. evaluation will by no means be weakened,but only The major executing agency for the SPC in project st re n g h t e n e d . In fact, the CIECC has now a better a p p raisal and ev a l u a t i on was the China i n s t i t u t i onal loca t i on than before in terms of International Engineering Consutling Company carrying out evaluation. Operating under the (CIECC). The CIECC, located in Beijing, has direct supervision of the State Council, it now

4 0 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing enjoys much more political clout and is almost evaluations for a number of projects in their own wholly independent of administrative agencies. As sectors,as required by either the SPO or their own a result, it can now access more easily information p l a n s , with the assistance of their affiliated and data relevant to evaluation and can report research or designing institutes. di re c t l y to the State Council any eva l u a t i o n findings and recommendations. It is essentially now in the The ministries and their institutes activel y parti c i p a t e position to monitor the performance of the public in (and even sponsor) training seminars and works h o p sector (cf. Mayne and Goni, 1997). Further, its of study tours to foreign countries on evaluation evaluation conclusions will hopefully be taken p ro c e d u res and methodologies. Some of the m o re seri o u s ly by administra t i ve agencies and ministries have developed their own evaluation project entities. However, there will be conflict of guidelines and directives, such as the Ministry of interest which will harm the objectivity of its communications, Ministry of Railway, Ministry evaluation work if the CIECC continues to do of Agriculture, Ministry of Health, and Ministry project appraisal. (Dr. HONG: WHY IS THIS of Water Resources. SO? CAN YOU EXPLAIN??) Ad m i t t e dly, the ev a l u a t i on ca p a c i ty in the Another important agency, the State Au d i t provinces is still relatively weak, but a number of Administration (SAA), which in recent years has provinces have had in recent years good exposure made significant progress in the evaluation field, to eva l u a t i on . Th e y have conducted self-eva l u a t i on s also merits increased attention. Re p o r ting direc t l y as required by SPC and some international organ- to the prem i e r , the SAA conducts audit of financial i za t i on s , p a rticipated in training courses on compliance and funds utilization as required by the ev a l u a t i on methodology organized by centra l State Audit Law. The top management of the agencies, and some of them even have prepared SAA attaches great importance to eva l u a t i o n work. their own evaluation guidelines. It will be a They have had extensive exposure to international si g n i f i c ant cha l lenge for China in the coming yea r s practice and have a good understanding of the to effectively design and implement an inter- linkage between eva l u a t i o n and auditing. The SAA governmental evaluation system that will allow has to date conducted considerable training for evaluation work to be linked and coordinated at their staff on evaluation methodology. They have the various administrative levels (cf. Rieper and also finished audits of financial and econo mic benefits Toulemonde, 1997). for 21 national key projects that were partially financed by lending from interna t i o nal organizat i on s . One sector significan t l y influencing the devel o pm e n t The SSA is now preparing guidelines,establishing of evaluation in China is banking. Many banks in data banks, and designing indicators for preparing China also evaluate projects they themselves have guidelines, and designing indicators for evaluation fi n a n c e d . Th e y have explicit req u i r ements reg a rd i n g operations. According to its plan, the SSA will what percentage of their projects should be eva l u at e d , have conducted a formal evaluation in 1998 and when to conduct an eva l u a t i on , what indicat o r s to 1999. (DR. HONG. DOYOU KNOW HOW use, and what kinds of data are to be collected. MANY EVALUATION THEY PLAN TO DO Since many banks have been extensively commer- IN 1998?). The significant advantages of doing cialized during the past few years, their demand evaluation in the SSA is that it promises almost for evaluating their own lending operations is 100 percent independency from administrative tremendous. The Construction Bank of China agencies, holds the highest political clout, and is (CBC),formerly named the People’s Construction close to the top state leader. In its audit reports, Bank of China, for example, has developed syst e m a t i c the SSA often makes critical but objective recom- guidelines and evaluated about 240 projects within mendations regarding what needs to be improved its portf o l i o . The State Devel o p ment Bank (SDB), in investment management. China,will now, with as a non-commercial bank,has also issued an array the evaluation capacity of the SSA,be able for the of eva l u a t i o n policies and guidelines. It also cond u c t s first time to link up the budgeting proces with the ev a l u a t i on re g u l a rly. In the banks, how e ve r, eva l u a t i o n process (cf. Gr ay, Je n k i n s , and Seg s w o rt h , ev a l u a t i ons largely assess financial benefits, 1993). The SSA can collect information and data management performance, compliance with terms quickly and effectively, and provide good feedback of loans, and risks on loan recall. There is little in on its findings to the State Council and relevant the way of evaluation outcomes or impacts,studies administrative agencies. of stakeholder satisfaction , and sustainability analysis. The banks evaluate against criteria they There are evaluation activities in the line ministrie s have as lending institutions, evaluation policies as well. Many line ministries have conducted largely independently of the SPC.

E V A L UATION CA PACITY DEVELOPMENT IN THE PEOPLE’S REPUBLIC OF CHINA: Trends and Prospects 4 1 The last key set of actor in evaluation within the information, (d) find appropriate means to convey PRC is made up of the bilateral and multilateral evaluation findings to the policy community, and d eve l o pment organiza t i on s , most notably the (e) create the development of evaluation as a World Bank, the Asia Development Bank, and sustainable effort. China is building the foundation. the United Kingdom’s Overseas Deve l o pm e n t However, there is no grand edifice in place. In the Adm i n i s t ra t i o n (ODA) . These three organizat i on s , Chinese gover nmental struc t u r e and administrat i v e in particular, have demonstrated sustained support hierarchy, several key tasks appear necessary at this for the development of an evaluation capability time if evaluation is to continue to develop. within China with their support , China has undertaken the following activities to strenghten Fir s t , it is vital for the PRC to have a strong central its evaluation infrastructure: organization for overall evaluation management ■ It has conducted studies of how other countrie s and coordination. Such a central organization in both the developed and developing wo rl d would carry out independent evaluations for state ha v e established their national eva l u a t i o n sys t e m financed key projects and state deve l o pment and what are the possible options China to programs and portfolio, report relevant evaluation establish its own evaluation system; findings to the State Council, and disseminate ■ It has drafted evaluation guidelines, manuals, evaluation results to relevant stakeholders and even and handbooks; the interested public. Furt h e r , gi v en the magnitude ■ It has financed senior Chinese officials to tra ve l and diversity of the national administration, the to both developing and developed Co u n t ri e s to or g a n i za t i o n would also set comp re h e n s i v e nationa l study differ ent eva l u a t i o n syst e m s ; evaluation policies and guidelines so that any ■ It has provided training in eva l u a t i o n to about one na t i o nal organizat i o n unit undertaking an eva l u a t i o n thousand officials, pro fe s s i on a l s , res e a r che r s ; an d will have a common understanding of policies and ■ It has organized a ministry-level seminar on methods necessary. In addition, there is a strong evaluation. need for alignment among the key organizations inv o l ved in eva l u a t i o n so that they share a comm o n Many of these activities have been carried out understanding and build a com m on set of within the past four to six years, and there have approaches to different evaluation tasks. This already been notable results. First, the CIECC coordination has not existed to date, but many in and the SDB have both set internal units to focus these different sectors see the need to do so. ex clu s i ve l y on eva l u a t i on . Sec on d , the CBC, si m i l a r banks,and a few sector ministries have established Second, it is important that the PRC establish evaluation divisions. Third, the SDPC has set up formal evaluation units, policies, and guidelines in a unit responsible for supervision of project imple- the ministries and banks. Such units would be me n t a t i on . Fou t h , the SAA has begun to incorpo ra t e independent of their operational departments and evaluation into its benefit audit function, and adequately staffed and they might be required to Fifth, the Ministry of Finance has established a evaluate not only their own projects, but also fu n c t i o n of cost-benefit analysis for state key proj e c t s . expand the analysis to include programs portfolios. Finally, in the area of knowledge dissemination, Such evaluations are an important instrument for some of those trained have now begun training the ministries and banks to maintain the quality of others on their own; and other central agencies, their portfolios as well as a basis for discussions major universities have incoporated on evaluation and decisions at the central level as to whether into their relevant modules, and some graduate further evaluations are required. students have chosen ev a l u a t i on as the main subject of their degree dissertations. Third, it seems to be an appropriate time for the provincial-level and local governments to start building their own evaluation capability. The local governments have now almost 100 percent respon- WHERE TO GO FROM HERE s i b i l i ty for their own investment activities, whithout interference from the central authorities. China has begun to (a) establish an evaluation Their investement portfolio are huge. Therefore, culture that legitimates serious inquiry into public the local gover nments need to establish an eva l u a t i o n sector perfo rm a n c e , (b) build the necessary experti s e authority operating directly under the governor’s to undertake ev a l u a t i on studies, (c) cultivate su p e rv i s i on . Suc h an authorit y would be res p on s i b l e understanding and recognition in the program and for evaluating key projects and programs, drafting policy communities of the utility of evaluation ev a l u a t i on policies and guidelines, organizing

4 2 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing evaluation training, and disseminating evaluation of large state owned enterprises. It is important to information. It appears essential to build a formal f u rther disseminate such practices to the line evaluation function in the provincial operational ministries and local governments. departments as well, especially for those that have a large investment portfolio. Some departments Finally, the PRC needs to develop a supply of h a ve already conducted ev a l u a t i ons and have w e ll - t rained evaluators for the many national relevant expertise and experience, but they do not ministries, provinces, and banks moving into the have a specialized evaluation unit and their evalu- evaluation arena. The supply at present is far too at i o n cap a c i t y needs to be strenghtened cons i d e ra b l y. small to meet demand. Indeed, demand will probably outstrip supply for some considerable Fourth, the PRC needs to set up in the SAA an time in the future. Building training centers, auditing process of the evaluation function so that re c ruiting talented students and gove rn m e n t there can be ongoing oversight and auditing of the employees, legitimating the professional status of evaluations undertaken within the line ministries evaluators, creating means of sharing experiences and evaluation policies and guidelines issued by and developents through journals and conferences, the central evaluation organizations, the relevant and linking evaluation skills to subject matter ministries, provinces, and banks. Evaluation units ex p e r tise will take cons i d e r able time and effort. But if need to be accountable and must adhere to standards China is to build the necessary cap a c i t y and experti s e , of high performance, and the SAA has the right all these efforts need careful, sustained attention. institutional location and legitimate authority to supervise this.

The oversight responsibility would be much like that performed by the U.S. General Accounting IN THE SHORT T E R M : Office, The Netherlands Court of Audit, the Swedish National Audit Office, and the Office of One task viewed as an important initial step to the Auditor General in Canada. the achievement of the evaluation goals set out above is the creation of a special central office, Fi ft h , the PRC needs to develop advanced which would have oversight responsibilities in the following areas: ev a l u a t i on methods across these units and ■ organizational entities. Much evaluation work to Drafting of guidelines and procedures for the ministries in the area of evaluation. date has not been performed by such methods. ■ Am ong the aspects that mostly need to be C o o rd i n a t i on of tra i n i n g, d eve l o pment of strenghtened are environment and social impact materials, and use of consultants; ■ a n a lys i s , s u s t a i n a b i l i ty analys i s , i n s t i t u t i on a l Development and maintenance of data banks assessment, program and portofolio evaluation, for its own work, the sector ministri e s ,p rovi n c e s , macroeconomic analysis, and evaluation of social banks, and the SAA; ■ sector projects (cf.Valadez and Bamberger, 1994). Building of evaluator netwo r ks and establishment Learning new approaches and bringing them into of a national evaluation association; ■ evaluation requires training, building curricula, Building the institutional cap a c i t y in univer s i t i e s translation of materials,and mentoring from those and research centers in order to ensure for the more experienced in evaluation. future supply of persons trained in evaluation; and ■ Sixth, the PRC needs to introduce the monitoring Dissemination of evaluation findings. and superv i s i on function into its investment management agencies. Mon i t o r ing and supervi s i o n Given the movement in China toward a national provides direct examination and control of the evaluation system, the creation of such an office is quality of project implementation. Further, it an essential first step.At present, there is no helps collect data that are needed for evaluation. national coordination or oversight to ensure the Without good monitoring and supervision, it is best use of resources, prevent duplication, and set difficult to conduct good eva l u a t i on . The PRC has high standards. begun to recognize the importance of monitoring and supervision. Apart from the SDPC,the State Since the CIECC is now the advisory organizat i o n Council has set up a supervision office in the of the State Council and has rel a t i ve l y good experti s e Ministry of Personnel specializing in monitoring in eva l u a t i on , su c h a central office could be well and supervising the performance of the managers placed in the CIECC.

E V A L UATION CA PACITY DEVELOPMENT IN THE PEOPLE’S REPUBLIC OF CHINA: Trends and Prospects 4 3 IN THE LONGER T E R M : Tracking the changes and assessing the impacts associated with the development will necessitate The experiences of developed and deve l o p i n g an evaluation infrastructure, considerable expertise c o u n t ries suggest that establishing national in both the subject matter and evaluation methods, evaluation system is a long and difficult process. It and development of routes for conveying eva l u a t i o n takes system coordination and alignment that is findings into the policy communities. The second not easily achieved, even in small countries with s o u rce of demand comes from deve l o pment relatively sophisticated public sectors. China is assistance given by the bilateral and mul t i l a t e ra l not small, and its public sector is still developing. le n d i n g organizations. The loan portofolio of the PRC now represents billions of dollars. China is China can best appro a ch this undertaking of the one of largest borrowers from the World Bank evaluation capacity building through a number of and the Asian Development Bank. The emphasis pilot projects targeted at sector ministri e s , by both the lenders and the Chinese government provinces, and banks with a strong emphasis on u p on portfolio perf o rm a n c e, i n d i ca t o r s , and understanding what ca reful ev a l u a t i on work evaluation of outcomes drives still more need for requires. This select group could test activities evaluation capability. related to capacity strenghtening, the building of methofological skills, development of reporting Finally, as China moves into the international formats, and training. The emphasis would be on arena, its policy communities will encounter new piloting different approaches to the institutional- evaluation developments,approaches,and forms of ization of evaluation in the public sector. In the utilization that will apply to China. There will be b e g i n n i n g, the emphasis would be more on encouragement to bring these developments home o r g a n i za t i onal learning than on organizat i on a l and test them out in the Chinese context. accountability, there may be a reluctance to test out new approaches-something essential in a pilot The major concern in the Chinese public policy phase. As lessons emerge from these pilot efforts, community is whether the initiatives and efforts the system can expand ministries and levels of d e s c ribed in these pages can be sustained. govern m e n t . Over time, the balance between learni n g Building a national evaluation system in China and accountability can be worked th ro u g h , and an will take decades. The infrastructure has to be eva l u a t i o n culture can be devel o p e d . built piece by piece across ministries, provinces, and local jurisdictions. The first steps we have The demand for evaluation in china is great. described above are being taken, but sustaining Demand is being driven by social, technological, this marathon effort will take financial support, and environmental development of the country, technical assistance, the perseverance of officials, which is moving in a high and sustained level. and patience.

4 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing EVOLUTION OF EVALUATION IN THE PEOPLE’S REPUBLIC OF CHINA

7 Peter C.Darjes,Operations Manager ADB

The process of building performance monitoring and evaluation capacity has been evolving in the PRC since the early 1980s. It was,however, not until 1992, when the China National Audit Office (CNAO), the Ministry of Finance (M O F ) , the State Planning Comm i s s i on , and other key central agencies, de c i d e d to develop specific proposals for recommendation to the State Council. The World Bank and the Bank have supported this process by providing a series of technical assistance for institutional strengthening and training. The World Bank supported a steering committee composed of representatives of the major public agencies under MOF leadership. This group drafted a proposal with various alternatives for an evaluation system. In 1994, the Bank provided a small-scale TA to develop project performance monitoring and evaluation capacity in the PRC.1 Under the TA, evaluation methodologies were developed and training was provided to selected Government officials. Similar objectives were pursued through TA to the State Development Bank.2 While the two TAs focused on selected agencies,3 the third Bank TA looked at eva l u a t i o n as a gover nance function and therefore more as a generic activity.4 The TA provided a useful overview of the existing institutional arrangements,policies,systems,and practices for monitoring and evaluation in the PRC.The main objectives were to develop a plan of action to build long-term evaluation capacity, improve the institutional arrangements for better coordination among evaluation agencies, and harmonize the evaluation system, methodology, and practices to be used by all evaluation agencies.

I.INSTITUTIONAL SET-UP

1. The evaluation system of the PRC involves many institutions. While CNAO has an oversight function for instituting evaluation at all government levels, its evaluation practice is confined to financial audits. Evaluation units have been established in most key central agencies. For example, the State Development Bank, Ministry of Construction, the China Engineering Consulting Corporation (CIECC), a Government owned consulting firm with close links to SDPC, and others have set up evaluation units. In all these cases,the evaluation units are being located at the highest levels of each organization to ensure independence from operational line responsibilities.

12TA No. 2133-PRC:Developing the Performance Evaluation Capability ofthe People’s Bank ofChina,for $100,000, approved on 9 August 1994. 13 TA No. 2664-PRC:Institutional Strengthening ofthe State Development Bank ofChina,for $500,000, approved on 16 October 1996. 14 The agencies covered by the TAs included the China National Audit Office (CNAO); State Planning Commission (SPC); China International Engineering Consulting Corp o ration (CIECC ) ; Sta t e Development Bank (SDB) 15 TA No. 2 8 2 1 - P RC :S trengthening of Ev aluation Capac i t y in the Peo p l e ’ s Republic of C h i n a ,f o r$500,000, approved on 4 July 1997.

E VOLUTION OF EVALUATION IN THE PEOPLE’S REPUBLIC OF CHINA 4 5 2. SDPC is at the apex of the country’s planning urgency in the PRC. To stimulate domestic and performance evaluation activities. It is demand and to counter deflationa r y tendencies directed by the State Council to issue the in the econo my the Gover nment has embarke d regulations and to commission evaluation of on a massive program of infrastructure invest- public sector programs. In the process of a me n t s . At the same time, poor implementation major reorganization in May 1998, SDPC results of large public-sector projects have has established an independent ev a l u a t i on heightened the awareness of the Government d e p a rt m e n t , the Key Project Inspectora t e for more rigorous performance management Office (KPIO). Currently, 85 KPIO staff sys t e m s . In Feb ru a r y 1999, a national confe re n c e monitors about 110 projects countryw i d e . As on project quality was prompted by a series of the number of projects will grow to eventually severe accidents and project failures relating to 700, the staff strength is projected to increase major investment projects. to about 200 in the medium term. SDPC’s evaluation work has been prim a ri l y focused on 6. In the PRC, despite the efforts exerted in the ex-post eva l u a t i o n of projects. Thus far, very p a s t , ev a l u a t i on is still in its infancy. little work is being done in mon i t o ring Therefore, the human resource requirements activities and evaluating operational perform- are significant and the need for both,qualified ance during project implementation. staff and training must be met as a matter of p ri o ri ty. Most KPIO staff has undergon e some training at various PRC universities and in the context of external assistance. KPIO I I . FUTURE CHALLENGES management has, however, expressed concern that this training is inadequate given the tasks 3. While proposals have been made from time to ahead. Previous training activities are either time to devolve evaluation responsibilities to con s i d e r ed to be too short or not comm e n s u ra t e provincial public sector agencies, no formal with best practices. As KPIO is mandated links have yet been established. This is the with the training of future evaluators and challenge to be addressed by SDPC in the pr oject managers, it must first focus on building f u t u re . The significance of perf o rmance up a nucleus of qualified trainers within its management and ev a l u a t i on needs to be folds. Currently, there is no clear notion of the assessed in the context of effec t i v e govern a n c e . c ompetencies and skills that would be It emanates from the delegation of authority req u i r ed for project perfo r mance management. to subordinated line agencies that will be held The formulation of job descriptions will be a accountable for the accomplishments of tasks key task of the proposed TA. assigned to them.Th u s , eva l u a t i o n is a necessary corollary to results-based resource allocation. In the PRC, this thinking is slowly evolving. C u r re n t ly, the emphasis of perf o rmance management is on inspection and control by I I I . PROPOSED ADB the central government. T E C H N I CAL A S S I S T A N C E 4. At the provincial government level, project A . O B J E C T I V E S performance management and evaluation is al l but unknown . Apa r t from the central agencies 7. The overall objective of the TA is to continue mentioned above, there is no authority at the provincial level that has a mandate or an the process of building perfo r mance management appropriate organization to undertake project capacity within the PRC administration. The performance evaluation. In the process of focus will be on the central government level. d e c e n t ra l i za t i on , the Provincial Pl a n n i n g This will mean to improve the ability of C om m i s s i ons were given project approv a l KPIO to develop its capacity and culture of authority. As a result, there are now a large evaluation. In this regard, the TA will act as a number of projects under provincial management. catalyst to enhance KPIO’s abilities,skills,and How e ver , even at this level it is being increa s i n g l y knowledge, procedures and attitudes that will recognized that there should be a linkage enable it to improve the effec t i v eness of public between performance evaluation and project sector investments on a sustainable basis. planning, design and approval. Spe c i f i ca l ly,the TA will foster project perfo rm a n c e m a n a g e m e n t ,i n cluding diagnosis, mon i t o ri n g , 5. During the past two years or so, project and impact evaluation. A logical framework performance evaluation has assumed renewed for the TA is given in Appendix 1.

4 6 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing B . S C O P E coordination and guidance for all evaluation activities in the country. In the absence of 8. The scope of the TA will follo w the rec om m e n - these prec on d i t i on s , it is difficult for eva l u a t i o n da t i o ns and the action plan prep a r ed under the methodologies to be harmonized and refined, previous Bank TA. As such, the TA will focus an evaluation culture to be formed,and for the on human resource development of KPIO. eva l u a t i o n profe s s i o n to be prop e r ly rec o g n i ze d . Towards this end, the scope will include: KPIO’s mandate includes all facets of impact ev a l u a t i on , i n cluding rev i ews of pro j e c t (1) Forming a partnership with one or more design, monitoring of benefit development, international partner agencies that have a bidding procedures, construction schedules, comparable mandate. This would most quantity and quality of physical progress as likely be a government department. It is well as costs, prices and fund utilization. considered that the most effective way of KPIO has shown a keen interest in adopting transferring know-how would be through the Bank’s Project Performance Management such arrangements; System (PPMS) and has translated the Bank guidelines into Mandarin. KPIO would (2) Identifying skills and key competencies appear as the most promising entry point req u i r ed for KPIO staff.The foreign partn e r for transmitting that system to the PRC wi l l help introduce approp r iate proc e d u re s evaluation administration. for project performance management;

(3) Based on the above activities, pre p a ra t i o n of GETTING THE PROVINCES ON BOA R D a training curriculum and a training prog ra m ; 10. The world over, the significance of evaluation (4) Implementation of training activities: as an element of governance has evolved in Training will center on seminars and the process of delegating responsibility and workshops to be conducted in the PRC. ac c o u n t a b i l i t y from central gover nment planning A limited amount of the TA funds will be l evels to line agencies and subord i n a t e d provided for overseas training. All train- provincial administrative levels. For KPIO to ing activities will be programmed and become effective, the provinces must over carried out in close association with the time match KPIO’s efforts in undertaking foreign partner institution; p roject perf o rmance management. T h i s should, as a start, include all projects under (5) Prep a ra t i o n of project perfo r mance eva l u a t i o n central Government financing executed by the guidelines.The guidelines should include p rov i n c e s , and should later cover pro j e c t s the use of teaching methods and tools; financed by provincial gove rnments alon e . The lon g e r - t e rm need to appro p ri a t e ly (6) Evaluation of selected Bank-financed including the provincial administrative level projects as case studies; in performance management is addressed by the proposed scope of the TA. (7) Prep a ra t i o n in outline form of a prog r am of activities for a follo w-up phase of cap a c i t y building TA,w h i c h would focus on sprea d i n g LONG-TERM PERSPECTIVE pr oject perfo r mance management prac t i c e s to provincial governments, and; 11. Building an effective evaluation capacity in a country is a long-term developmental process (8) Provi s i o n of office and training equipme n t . and the process may take decades.The process has been evolving in the PRC since the early 1980s. During this relatively brief time span, remarkable progress has been achieved in the PRC.However, given the size and complexity I V. KEY ISSUES of the country, particularly with its ongoing e c on omic tra n s f o rm a t i on , a major effort APPROPRIATE ENTRY POINT would still be required to build an effective project and program performance evaluation 9. Institutionally, there is not a high-level central system covering all concerned entities in the au t h o ri t y in the Gover nment to provide overa l l Government.

E VOLUTION OF EVALUATION IN THE PEOPLE’S REPUBLIC OF CHINA 4 7 TRANSFERRING THE REQUIRED KNOW-HOW on tech n i cal assistance, it will largely be intended to assist staff to understand and 12. Project management will have to consider the master a process. Organisations interested in optimum source of technical assistance for this partnership arrangement will have to show each type of activity under the project. In this ho w they intend to develop a rel a t i o nship based regard, a priority activity for the TA will be on mutual learning, equality and respect, with the selection of one or more organisations to sustainability of process as a long-term goal. enter into a partnership arrangement with KPIO. A partnership arrangement is more The partner agency will have a number of functions , c omplex and demanding than a tw i n n i n g including the provision of technical assistance, a r ra n g e m e n t . Twinning arrangements are professional development support and mentoring often based on an expert-client model. A pa r t- for KPIO curriculum development and in-service nership approach may involve some twinning, staff, as well as the provision of work attachments. but takes a more developmental approach The partner organization(s) would have to be based on an analysis of the current and future selected through competitive bidding. This will skills and needs of both organisations. While not be easy given that the pros p e c t i v e organizat i on s a partnership will have some necessary focus would likely be in the public sector.

4 8 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing LINKING EVALUATION TO POLICY FORMULATION AND BUDGET PROCESSES: Lessons Learned from Australia 8 Leo n a r d J. Ea r ly, De p t . of Finance and administrati o n , Aus t ralian Public Sec t o r

I N T R O D U C T I O N

Australia has had a well-developed system of policy evaluation that has been firmly linked to the Budget process. The system developed over many years,but is now in a state of flux – its future is far from assured.The growth and possible future decline of the Australian model can provide some interesting lessons about development and maintenance of evaluation capacity in other countries.

THE HISTORY

There has been a long history of policy evaluation in Australia. A number of policy research bureaux (e.g. the Productivity Commission, the Australian Bureau of Agricultural and Resource Economics and the Bureau of Transport and Commu n i ca t i o ns Econo mics) have had a major impact on policy formul a t i o n in Australia for many decades. 1 In addition, major public enquiries have frequently been com m i s s i o ned to evaluate econo mic and other gover nment policies. For example,the Vernon Committee2 undertook a major review of Australian economic policy in the early 1960s, and the Crawford Study Group3 evaluated industry policy in the late 1970s. Evaluation was given significant impetus with the election of the Hawke Labor Government in 1983.In the period 1947 to 1983, Labor had been in Government for only three years, when Gough Whitlam was Prime Minister, from 1972 to 1975. The Whitlam government never achieved legitimacy in the eyes of many Australians, and it was dismissed in controversial circumstances by the Governor General. The Hawke Government was led by young Ministers with terti a r y qualificat i o ns in law, ec on o mics and comm e rc e , who were determi n e d not to repeat the mistakes of their predecessors. They intended to stay in office su f f i c i e n t l y long to stamp their phi l o s o p hy irrevo ca b l y on Aus t r alian public policy.

The Hawke Gover nment had come to office with a well- a r ticulated public servi c e po l i c y, designed to ensure that Ministers were firml y in cont r ol of the public policy agenda. Upon its election, the Government published a White Paper on public sector reform4 and embarked on a reform program, aspects of which included: ■ The introduction of program budgeting with its focus on program objectives and program performance rather than on inputs; and

1 Par ticipating in other agen c i e s ’ evaluation act iv i t y became a major part of De p a r tment of Finance budget of f i c er s ’w o rk . By around 1993, the Department of Finance had to limit its invol v ement to major eval u a t i o n s . 2 The Hon David Kemp, Minister Assisting the Prime Minister for the Public Service, Ministerial Statement,Reforming the public service to meet the global challenge, Public Service and Merit Protection Commission,1998. 3 This action was unprecedented in the Commonwealth Government in Australia. 4 Within a set of fairly loose controls.

LINKING EVALUATION TO POLICY FORMULATION AND BUDGET PROCESSES: Lessons Learned from A u s t r a l i a 4 9 ■ Devolution of authority and responsibility to potential prog r am improvements – inclu d i n g of program managers to enable them to achieve pro g r am objectives and perfo r mance informa t i on . the desired objectives. ■ I n t e rnal ev a l u a t i on s , u n d e rtaken entire ly within agencies, u s u a lly short exe rcises aimed at improving program management and efficiency. THE INTRODUCTION OF A FORMAL EVALUATION POLICY BEDDING DOWN By the middle of the 1980s, it was clear that furth e r ref o r m was req u i r ed if the Govern m e n t ’s objectives THE NEW A R R A N G E M E N T S we r e to be met. Prog r am objectives and perfo rm a n c e were not well integrated into the Budget process. In the early years,some agencies regarded the need Prog r am res o u r cing did not depend on demons t ra t e d to produce evaluation plans as an unnecessary program performance and, as a result, program bureaucratic requirement. The quality of plans managers did not have the incentive to produce was variable and, in many ca s e s , ev a l u a t i on good perfo r mance informa t i on . Prog r am perfo rm a n c e planning was not well integrated into strategic information was poor, and public service managers planning within agencies. Evaluation capacity in we r e not perfo rm a n c e - o ri e n t e d . Prog r am budgeting agencies was mixed , as was the quality of eva l u a t i on s . had not deliver ed the requisite cultural change in pu b l i c sector management. Into the 1990s, a separate evaluation unit was established in the Department Of Finance to over- Further public sector reform in 1987 and 1988 sight the evaluation strategy, and to assist agencies included the introduction of a formal evaluation with ev a l u a t i on plans, with advice on strategy. This strategy had a number of elements: evaluation methodology and (in some instances) ■ Ea c h policy prog r am was evaluated ever y three with evaluations. That unit delivered evaluation to five years. t raining to hundreds of policy analysts and ■ E a ch ye a r, e a ch portfolio submitted an produced evaluation handbooks. evaluation plan to the Department of Finance spelling out the portfolio’s evaluation program Evaluations were increasingly used and demanded over the next three years. by Ministers in taking Budget decisions. (Indeed, ■ Ea c h new policy proposal submitted for Budget th r ough the 1990s, it became increa s i n g l y comm o n consideration included a proposal for future for the Expenditure Revi e w Committee – the Cabinet evaluation of the initiative. s u b - c ommittee re s p onsible for producing the ■ Evaluations were normally to be published. annual Budget - to refuse to consider policy prop o s a l s if scheduled evaluations had not been completed.) This stra t e gy recognised the full range of Evaluation units (of varying sizes and capacities) evaluation activity. It did not only focus on more were established in most of the bigger portfolios. formal evaluation processes. It encompassed different forms of evaluation, addressing program Most agencies quickly learnt that major Budget appropriateness, effectiveness and efficiency. For proposals were more likely to be accepted if they example, the strategy included coverage of: were supported by sound evaluation (particularly if ■ High level external evaluations (usually on that evaluation had also involved the Department politically more sensitive issues), requiring Of Finance and if that department supported p a rticular expert i s e, c omplex analysis and the evaluation results). As a consequence, the possibly leading to significant change. Department of Finance was routinely involved in ■ Evaluations involving both the Department of steering committees of major evaluations5 and Finance and the line agency; usually taking ev a l u a t i on became firm ly embedded in policy less than one year and typically focused on development and in Budget processes.

4 For example: • The Department of Finance and Administration was formed by amalgamation of the Department of Finance and the Department of Administrative Services. On its formation in October 1997,the Department had 2,300 employees. As a result ofa series of reviews (some of which had been commissioned before the Department was formed) activities have been abandoned, re-engineered or contracted out. As a result,employment in the Department has now fallen to around 700; • The provision of IT services to all government agencies is being substantially outsourced to the private sector; and A new agency, Centrelink, has been created and required to compete with the private sector to deliver employment services to the unemployed.

5 0 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing By 1997, the Australian National Audit Office The government substantially rationalised central was conc luding that eva l u a t i o n plans and the quality controls on agency operations. Agencies became of evaluations was generally satisfactory (although fr ee to set their own wages and terms and cond i t i on s mo r e needed to be done to ensure eva l u a t i o n res u l t s of employme n t ; financial cont r ols were substantially were used to improve program performance). rat i on a l i s e d ; and most central rep o r ting req u i re m e n t s were rationalised or abandoned. However, at the same time, portfolio evaluation plans had exploded in length and complexity, until The winding back of central controls included reports of 120 pages or more were the norm. The abolishing the requirement to produce portfolio Department Of Finance and some agencies were evaluation plans. Agencies were to report (very be c o ming conc e r ned at the cost of this burea u c ra t i c b ri e f ly) ev a l u a t i on plans in Po rtfolio Budget ove rh e a d , and beginning to question whether St a t e m e n t s . In pra c t i c e, the specifica t i on of evaluation had become sufficiently embedded in evaluations in Budget documentation has been public service culture that the reporting burden patchy, with some agencies (most notably the could be significantly rationalised. Department of Finance) paying this requirement scant attention.

The ev a l u a t i on unit in the Department Of THE NEW PHILOSOPHY Finance has been disbanded and the department no longer sees its role as assisting agencies with The conservative Howard coalition government eva l u a t i o n methodology (although it does cont i n u e was elected to power in 1996. The new govern m e n t to sit on steering committees of significant was highly sceptical of the efficiency and effec t i ve n e s s evaluations – often at the request of Ministers or Cabinet Committees). of the public service. It has stated6 that on coming to office it was confronted by: ■ The Government has required managers to review A public service with major inflexibilities in its all their activities to see if they should be discon- ad m i n i s t ra t i v e, in d u s t r ial and financial dealings; tinued, privatised, devolved to another level of ■ Costs of many processes that were high becau s e government or otherwise improved, e.g. by market of a culture of compliance with detailed central testing. This entails a fundamental evaluation of rules and regulations; the appropriateness,effectiveness and efficiency of ■ A Commonwealth bureaucracy that lagged all government operations. There have been some behind other governments and the private significant changes, but most agencies are yet to sector in its efficiency and in the quality of its commence the process. management; and ■ In 1999, the Comm o nwealth gover nment prod u c e d A cash-based accounting system that made its first accrual Budget. The accrual Budget is ac c u r ate assessments and comp a ri s o ns of public produced in terms of outputs and outcomes. This sector activity impossible. provides a link from Budget resourcing through the outputs produced by agencies to the outcomes th e The Government rapidly set out to address these gover nment wants to achi e ve for the Aus t r alian people. perceived deficiencies. It dismissed six of the 17 p o rtfolio secre t a ries (chief exe c u t i ve officers);7 Budget documents detail the proposed pri c e, introduced a Budget which cut $A8 billion (or quantity and quality of agency outputs, and link a round two per cent) from Com m onw e a l t h these to desired outcome s . The documents spell out Government outlays and introduced a sweeping detailed perfo r mance targets and perfo r mance against range of public service reforms. those targets is to be reported in annual reports.

6 Not surprisingly, the quality ofthis material is highly variable in this year’s Budget,with much progress yet to be made. 7 Government activities are classified either as Departmental outputs (the goods and services produced by Departments and agencies on behalf of Government for organisations or individuals) or administered items (controlled by the Government and managed or oversighted by Departments and agencies). Examples of Departmental outputs are policy advice, evaluation activity or the activity of payment of pensions or benefits. The benefits (e.g. unemployment benefits and age pensions) which are determined by Government and that are a statutory right of people who meet the eligibility criteria,are, administered items,as are most financial payments to the States and Territories.

LINKING EVALUATION TO POLICY FORMULATION AND BUDGET PROCESSES: Lessons Learned from A u s t r a l i a 5 1 The concept of prog r am essentially disappears. that departmental expenses comprise only around Thus the requirement to review programs every 10 to 20 per cent of the Commonwealth Budget, three to five years effectively lapses (although it any such development would entail obvious risks. has not formally been repealed).

There is a rolling program of price reviews that e f fe c t i ve ly evaluate perf o rmance in delive ri n g De p a r tmental and agency outpu t s . How e ver , th e r e LESSONS FROM AUSTRALIAN EXPERIENCE is a gap, in that there is no longer any formal requirement or process to regularly review administered items. In implementing an evaluation strategy, Australia st a r ted with some significant advantages, in cl u d i n g : ■ A history of eva l u a t i o n at least of major policy initiatives; ■ A strong central statistical unit producing I M P L I CATIONS OF THE NEW PHILOSOPHY quality data; ■ A well educated bureaucracy, with a good leavening of policy and evaluation skills; Prior to the election of the Howard Government ■ A history of public provision of information in 1996, the efficiency of gover nment arran g e m e n t s on program performance; and had not been subject to muc h scrut i n y or eva l u a t i on . ■ A sound Budget system, including four year The major instrument for improving efficiency fo rw a r d estimates of Budget outlays and reven u e s . had been the efficiency dividend, which had been introduced in 1987. With the efficiency dividend, Ev en so, it took ten years for the eva l u a t i o n strat e g y agency running costs were reduced by one per cent to become firml y embedded in the Aus t r alian public each year.While some agencies found this regime service culture. di f f i c u l t , in fact it was highly ineffec t i v e in improvi n g efficiency. Most agencies had little difficulty in Mackay has identified factors in the success of obtaining new resources through the new policy Australian evaluation policy as: process in the annual Budget, and managers had ■ The crea t i o n of an explicit, whole-of govern m e n t little incentive to re-engineer or restructure their evaluation strategy; operations to im p r ove efficiency or effec t i ve n e s s . ■ The existence of a powe r ful central departm e n t Most govern m e n t op e ra t i o ns were effec t i ve l y sheltered that has been a committed cha m p i o n of eva l u a t i on ; from any effective scrutiny or competitive pressure. ■ Sustained commitment and support the eva l u a t i o n strategy over a decade; and Ev a l u a t i o n activity was mainly focused on prog ra m s . ■ I m p l e m e n t a t i on of related public sector Few ev a l u a t i ons shone the spotlight on the management re f o rms that have given appropriateness,effectiveness or efficiency of those con s i d e r able autono my to line managers and 12 gover nment operat i o ns that deliver ed the prog ra m s . that em p hasise bottom-line results – provi d i n g in c e n t i v es to conduct and use eva l u a t i o n findings. This situation has changed dra m a t i ca lly. Attention is now firmly focused on the public I agree with this analysis as far as it go e s . service and its effectiveness and efficiency. The However, to the list of factors Mackay identifies, intention is to subject the vast bulk of government I would add the existence of a strong supportive activity to competitive pressure – with agency group of key Ministers throughout the period. (I resourcing based on competitive prices of outputs see the consistent demand of the Expenditure rather than on the cost of their production. R ev i ew Committee (ERC) for ev a l u a t i ons as one of the most important reasons that evaluation The additional evaluation of government activities flourished in Australia over this period). is a welcome development. However, the risk is that this entails a shift in the focus of attention I think the Australian experience also illustrates away from the effectiveness of programs. Given the potential fragility of the evaluation culture.

5 2 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing While current Ministers are, if anything, even In my view, there is no doubt that a requirement to more firmly focused on performance than their rep o r t eva l u a t i o n plans and perfo r mance is necessary p re d e c e s s o r s , the assumption is that greater to get the requisite response in Departments and management flexibility, in tandem with increased agencies. However, this reporting requirement market discipline (including by privatisation) will should not be allowed to burgeon into a bureau- underpin performance improvement. cratic monstrosity. Reporting regimes should be kept simple. Evaluation resources in Departments Evaluation has a role in the process,but this role is and agencies should be directed to evaluating, not greatly diminished, in that the regular evaluation to completing lengthy reports for central agencies. of the appropriateness, effectiveness and efficiency Is it necessary for countries to “walk before they of the bulk of Government outlays is no longer can run”? In other words, is it necessary to first guaranteed. focus on prov i s i on of simpler perf o rmance information before moving to implement more A moot point is the extent to which evaluation so ph i s t i c ated analyti c al techniques? I would suggest will continue to flourish in this environment. not – in my view, it would be desirable to move on both fronts simultaneously, albeit the adoption of Developments thus far suggest that the culture more sophisticated analytical techniques would is unlikely to change quick ly in the bigger need to be at a pace that recognised the availability D e p a rtments and agencies where ev a l u a t i on of skills to support the analysis. is firmly embedded – e.g. in those Departments that have established their own evaluation units. Public sector reform typically comprises a focus In other organisations, the stimulus to evaluation on outputs rather than inputs, and greater devo l u t i o n is likely to come from the Department of Finance of authority and responsibility to line managers. and Administration in the annual Budget process. Perf o r mance objectives and perfo r mance informa t i o n However, given that only around two per cent cle a r ly have a role to play in this proc e s s . How e ver , of Government outlays typically come up for the Aus t r alian experience suggests prog r am managers detailed scrutiny in the annual Budget,this implies ar e unlikely to produce good perfo r mance informa t i o n a significant potential re d u c t i on in ev a l u a t i on unless it is an integral part of the Budget process. activity in some areas. However, the use of performance information in the Budget cannot be automatic. There will be a need of judgement in interpreting the performance I M P L I CATIONS FOR OTHER COUNTRIES information. Evaluation is necessary if performance information is to give up its meaning. An interesting question is the extent to which For example, a failure to perform satisfactorily other countries might be able to learn from and could call for: emulate the Australian experience, particularly if ■ A reduction in Budget allocation (e.g. if the their starting environment appears less conducive prog r am is proving too expensive and the outcome to success. does not justify the cost); ■ An increase in allocation (e.g. if the program In my opinion, the Australian experience does is a high priority for the Government and pr ovide some useful guidance. It would suggest that cost is a lesser issue); or introduction of a systematic process of evaluation ■ No change (e.g. if a road program has been is more likely to be successful if: delayed by inclement weather). ■ T h e re is an explicit whole-of-gove rn m e n t policy championed by the Budget agency; The important point is that the appro p ri a t e ■ Ministers strong l y support the eva l u a t i o n policy Budget re s p onse can on ly be determined in and demand and utilise evaluations in the the light of ev a l u a t i on and analysis of the Budget process; and performance information. Evaluation is necessary ■ Public sector reform has given line managers for a proper interpre t a t i o n of the res u l t s . One needs sufficient autonomy to manage their resources to ask questions such as: to achieve specified objectives. ■ Ar e results good because targets were set too low? However, there is no miracle cure. Building an ■ Are results poor, despite good performance, evaluation culture requires sustained effort over an e.g. because the agency is poor at producing extended period. accurate and timely performance information?

LINKING EVALUATION TO POLICY FORMULATION AND BUDGET PROCESSES: Lessons Learned from A u s t r a l i a 5 3 ■ Has there been goal displacement (e.g. low e ri n g rea s o ns for good or bad perfo r mance and eva l u a t i o n service quality in other areas) in order to is the most effective means of shedding light on achieve the target? su c h questions . Sound eva l u a t i o n both req u i r es and in c r eases the value of good perfo r mance informa t i on . In summary, one needs to understand the underlyin g The two go together.

5 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PUBLIC PERFORMANCE EVALUATION AND IMPROVEMENT

Marc Holzer, Executive Director, 9 National Center for Public Productivity, Rutgers University , NJ, USA

PRESSURES FOR PERFORMANCE

“Government must be accountable for results.”“Government must do more with le s s . ” The ‘bo t t o m line’ for gover nment is that such statements, voiced rep e a t e d ly by policy makers, citizens, and the media, are deceptive in their simplicity.

Th e y raise some ver y basic questions : What are the desired “res u l t s ” of govern m e n t ? What does accountability really mean? Faced with competing demands and expectations, and social issues that seem to defy measurement, the “results” of public sector initiatives are difficult, at best, to define. Although we might achieve consensus on broad platitudes, the “devil is in the details.” How can we develop an objective approach to accountability, to evaluation and improvement, which will be accepted as impartial, utilized as a means of adding value to public decisions, and applied to positively impact the quality of the lives of citizens? Our argument is that these multiple objectives can only be achieved if policy makers and citizens are actively involved in the process of performance measurement - the evaluative process of holding agencies accountable for their expenditures, actions and promises.

Until recently, government accountability was largely a matter of financial accounting, hence its name. When public funds were appropriated, the key accountability questions focused on how much money was spent and on what specific items: personnel, supplies, travel, communications, etc. Today, the concept of governmental accountability has taken on a much broader meaning. According to Romzek and Dubnick (30) “Accountability is a relationship in which an individual or agency is held to answer for performance that involves some delegation of authority to act. Accountability mechanisms are the means established for determining whether the delegated tasks have been performed in a satisfactory manner or not.

Thus governments,in an effort to demonstrate accountability, need to show their policy officials and service users:

(1) what they are getting for their tax dolla r s , in terms of services (and some t i m e s products): public hospitals, roads, airports, libraries, water supplies, etc...

(2) how efficiently and effectively their tax dollars are spent: cost per mile of roads built to certain standards, cost per pupil of students graduating from high school or university.

(3) how such expenditures benefit their lives or the lives of those they care about: eradication or suppression of diseases such as encephalitis or AIDS; high sense of security in a neighborhood; safe and reliable water supplies.

PUBLIC PERFORMANCE EVALUATION AND IMPROV E M E N T 5 5 This type of accountability holds gove rn m e n t Performance measures can be quantitative (average responsible not only for its actions,but also for the response time) and qualitative (how safe people results of its actions - the impact of each service on feel in their parks during the day or at night). each citizen. Managers can develop performance measurement systems incorporating a variety of performance In order to emphasize to policy makers and service i n d i ca t o r s , s u ch as those defined by the users what they are getting for their tax dollars, Government Accounting Standards Board(4) in agencies need to be able to evaluate-measure and the United States: report- what they are accomplishing. Measures wh i c h are unre p o rt e d ,s u p p ressed or poorly rep o rt e d Input indicators. These measure the resources are virtually useless in improving performance. expended on a program, such as the amount of We ll re p o rted (i.e. w e ll displayed and widely money spent or the total number of employee- distributed) performance measures emphasize how hours needed to deliver a service. well a government is doing at meeting its citizens’ needs and at living up to its commitment to C a p a c i ty indica t o r. These measure the ability provide quality services for the taxes people pay. of an organization to deliver services. Capacity Performance measures can help officials, public measures helps managers evaluate the level of managers, and citizens evaluate the impact that training (how recent, percentage of employees various expenditures have on the quality of their actually trained, level of training, etc.), the state of l i ves and the lives of those they ca re about. physical facilities (space, comfort,safety, etc.), the Performance measures also enable officials, public readiness of systems (Y2K compliant, computer managers and citizens to see the outcomes, or networking, memory and speed, etc.). results, of specific government services and to place a relative value on each service provided. Output indicators. These report the quantity of products or units of service provided to a service Evaluation and measurement of performance has p o p u l a t i on .T h ey also include “wo rk l o a d ” me a s u re s always been implicit in questions of outputs and that reflect the amount of effort expended to outcomes: Is crime increasing? Are the streets produce a product or provide a service. Examples cleaner? Is the air quality better? How well are our of output indicators include: number of meals children doing in school? In short, is the program delivered;miles of road paved;number of students producing as promised; as expected? What are passing the high school proficiency test. the results? Performance measures provide an op p o rt u n i t y to answer such questions with specific Outcome indicators. These measures report the data for evaluation, rather than the more common results of prog r ams and servi c e s . Out c o me indicat o r s situation: subjective statements based on vague possess quantitative and qualitative aspects. Ex a m p l e s assessments of efficacy: the neighborhood seems of outcome indicators include: the number of sa fe r . “Litter here is worse than ever. ” “The scho o l s individuals employed six months after parti c i p a t i o n in this town are very good!”In the absence of data, in a job training prog r am or percentage of res i d e n t s these statements are merely impressions. who frequently enjoy using a park.

E f f i c i e n cy and cost-effe c ti veness indica t o r s . These measures focus on how a goal is achieved, PART I: DATA FOR rather than what was ach i eve d . Sp e c i f i ca lly, efficiency indicators refer to the ratio of the level E V A L UATING PERFORMANCE of service (tons of refuse colle c t e d , number of meals de l i ve r ed) to the cost,in dollars and labor,of provi d i n g It is helpful to think of performance measurement the services. They measure the cost per unit of an as a process or system of measures and procedures, ou t put or outcome . Examples of efficiency indicat o r s whereby organizations assess how well they are include: cost per meal delivered; cost per ton of doing compared to previous performance, to other garbage collected; cost per pupil educated. organizations (“benchmarking”), and in terms of how well they are achieving their stated goals Prod u c ti v i t y indicat o r s . These measure s ,a c c o rd i n g and objectives. A well-designed evaluation and to Am m on s ( 1 ) , c ombine the dimensions of performance measurement system should clearly efficiency and effectiveness in a single indicator. ar ticulate service goals and objectives , define servi c e For example, the number of meals delivered per outputs and outcomes, and specify the expected hour measures the efficiency; the number of quality levels for these outputs and outcomes. meals delivered on time (and warm) measures the

5 6 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing effectiveness. The unit cost (labor-hours) per on basic programs that need the most managerial time (and warm) delivery measures productivity. oversight and where accountability reporting is most important. Designing a good perf o rmance measurement system may be challenging to public managers not accustomed to measuring or setting performance 2 . STATE THE PURPOSE AND IDENTIFY THE DESIRED OUTCOMES ta r g e t s . How e ver , this effort should not be dismissed as too complex or too difficult. De veloping a syst e m Typically, a government ministry, department or involves an understanding of what the program agency initiates a strategic planning process to is trying to accomplish, who the main users or clarify its mission, goals, and objectives. Through cu s t o mers are, and a basic knowledge of the curren t this process, the agency can identify the outcomes level of service. The Government Performance or results it wants to achieve through its programs. Results Act of 1993 in the United States rep re s e n t s A manager can only measure the performance of a an effort to institutiona l i z e perfo r mance measurem e n t program if its purpose is clearly stated. Preparing systems at the federal level(8a). The act requires a well-articulated statement of purpose is a very federal agencies to: im p o r tant first step. Id e a l ly, a clear mission statement is the starting point. If that is not available, a thorou g h ■ establish top-level agency goals and objectives , program description is a good place to begin. For including annual program goals; ex a m p l e , GASB (12) offers the follo wing statement of purpose for a public transportation system: ■ define how they intend to achi e ve those goa l s ,a n d ; The basic purpose is to provide safe, dependable, ■ de m on s t r ate how they will measure agency and convenient,and comfortable transportation services at pro g r am perfo r mance in achi e ving those goa l s . minimum cost to the citizens,including special clients groups such as the handicapped and the elderly. A good evaluation and performance improvement system should include the following seven-step system developed by the National Center for 3 . SELECT MEASURES OR INDICA T O R S Public Productivity at Rutgers University-Campus at Newark: A good system uses a few selected indicators to me a s u r e outcomes and perfo rm a n c e . Most govern m e n t p ro g rams that have established perf o rmance 1 . IDENTIFY THE PROGRAMS TO BE MEASURED measurement systems incorporate the indicators described above: input, capacity, output, outcome, To start with, programs to be measured must be e f f i c i e n cy and pro d u c t i v i ty. For example, in a clearly defined. Programs are groupings of routine sanitation department the input measures might activities aimed at providing support for specific be the amount of labor hours; the operating budget; public services. Groupings of individual activities the number of veh i c les on the roa d . Out put measures make up a program. For example, the following could include tons of refuse collected; number of activities-street resurfacing; street patching; seal households served; number of missed collections. coating; and curb repair-constitute a program that Efficiency measures could include labor-hours per is traditionally called street maintenance. Usually, to n of garbage collected or household serve d ;d o ll a r s programs are defined by governments and are spent per 1,000 households; and pro d u c t i v i ty g e n e ra lly listed on an organiza t i onal ch a rt indicators could include measures such as the cost contained in the operating budget. Programs relate per mile of clean streets; the cost per household of directly to the organizational structure and the twice weekly refuse collection. managerial areas of responsibility.

Choosing what programs to measure is a matter of 4 . SET STANDARDS FOR PERFORMANCE AND judgement. On the one hand, programs should OUTCOMES (TARGETS FOR A C C O M P L I S H M E N T ) not be too few, so that only a small portion of services are covered or the information collected is Under this step, public managers should specify in s u f f i c i e n t . On the other hand, too muc h rep o rt i n g the conditions under which program goals and can be excessively costly, or overwhelming and objectives are met. Managers need to determine impractical. Generally, performance measurement what service effectiveness and quality means for a s ystems work best when they con c e n t rate on particular program and explicitly state how they collecting limited but essential information about are going to determine whether the stated terms of

PUBLIC PERFORMANCE EVALUATION AND IMPROV E M E N T 5 7 effectiveness and quality have been met. This planning to reevaluate goals and objectives and to involves comparing actual program outcomes or adjust priorities. Another use is in managing for results against some agree d - u p o n standards ,i n cl u d i n g : results where outcome information is used to p romote continuous improvement in pro g ra m ■ pr evious perfo r mance (the percentage of res i d e n t s operations and results. In terms of analysis for who feel safe in their neighborhoods this year ac t i on , a well- d e veloped perfo r mance measurem e n t compared to last year); and evaluation system will enable managers to spot weaknesses and challenges to program delivery, ■ p e rf o rmance of similar organiza t i ons (the as well as program strengths and opportunities percentage of residents who feel safe in their for improvement. neighborhood compared to the percentage in a neighboring community);

■ performance of the best organizations (the percentage of residents who feel safe in their PART II: PERFORMANCE neighborhoods compared to the percentage MANAGEMENT FOR IMPROV E M E N T of residents who feel safe in their neighborhoods in the recognized “safest communities” in the country); It is useful to distinguish between performance measurement and performance management for ■ pre-set targets, (next year 85 percent of all im p r ovem e n t . Perf o r mance management for improve- residents will feel safe in their neighborhoods, ment is broader than performance measurement. and in three years 95 percent will feel safe.) Pe rf o rmance management for improve m e n t in c o rp o r ates the results of perfo r mance measurem e n t into general management practices. In the U.S. 5 . MONITOR RESULTS the National Performance Review (25a) defined pe rf o r mance management as “the use of perfo rm a n c e Each accomplishment target should be monitored measurement information to help set agreed-upon on a continuous basis. Monitoring provides the pe rf o r mance go a l s ,a ll o cate and prio ri t i z e res o u rc e s , manager with the results needed to decide whether in f o r m managers to either conf i r m or change curren t or not the performance targets have been met. policy or program directions to meet those goals, Systematic and periodic mon i t o ring gives the and report on the success of meeting those goals.” manager an opportunity to keep track of the operation of the program, and take corrective It is essential that performance measurement be action when necessary. Usually, monitoring will considered an inherent and indispensable part of vary depending on the program and target accom- the management for improvement pro c e s s . plishments. For the most important programs and Measurement for measurement’s sake alone is services, monthly data collection and reporting insufficient. Managers can have stacks of data at systems that indicate results will be necessary. their fingertips, but unless they use that data to im p r ove policy and proc e d u r es it is virtu a l ly useless. 6 . PERFORMANCE REPORTING O s b o rne and Pl a s t rik write in their book “Banishing Bureaucracy”(27) “We have not listed performance measurement as an approach because A good performance measurement system reports we do not believe the act of measuring itself has program results on a regular basis. The report enough power to force fundamental change in public focuses on what was accomplished and what it cost the public. R e p o rts should be bri e f, c onvey or g a n i za t i on s . It is crit i c al competent organizat i on s i n f o rm a t i on gra ph i ca lly, and include minimal need it...But some public organizat i o ns have measured explanatory information. Information should be performance for years, with virtually no impact.” pr esented in such a way as to facilitate comp a ri s on s over time, com p a ri s o ns with similar organizat i on s , As a means of achieving measurable impacts, we comparison with best programs nationwide and have found the following system to be an effective against pre-set targets. i n t e g ra t i on of measurement and improvement factors. This conceptualization, or “roadmap,” is based upon our research has to best practices, and 7 . USE OUTCOME AND PERFORMANCE INFORMATION in particular the experiences of award-winning Exemplary State and Local (EXSL) programs I n f o rm a t i on from an effe c t i ve perf o rmance identified by our Na t i onal Center for Pu b l i c measurement system is regularly used in program Prod u c t i v i t y (19). Tod a y, to produce public servi c e s ,

5 8 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing the best public organiza t i ons have developed Although citizen part i c i p a t i on can ultimately mu l t i p l e, re i n f o rcing ca p a c i t i e s . In part i c u l a r, improve the level and quality of municipal service government agencies which have been formally provision by making services more responsive to recognized as high achievers, as state-of-the-art: the needs of citizens,it is often difficult to achieve. ■ apply quality management principles; While citizen inv o lvement typ i ca lly results in ■ use measurement as a decision making tool; effective policy and meaningful public sector, (32) ■ work hard to motivate employees; ac t i v e citizenship is often perce i v ed as burde n s om e , ■ adapt new technologies; and costly and time consuming (25, 33, 34). The ■ develop public-private partnerships. traditional,top-down,hierarchical model of public ad m i n i s t ra t i o n limits the role of citizen parti c i p a t i o n in perf o rmance measure m e n t . Pe rf o rmance measurement systems may be easier to design and PART III: GETTING POLICY MAKERS implement when citizens are excluded and managers are the ones determining what will be AND CITIZENS INVO LV E D measured and how it will be measured. Although easier to implement,su c h perfo r mance measurem e n t Pol i cy - l e vel and citizen invo l vement in perfo rm a n c e systems fall short in terms of measuring what m e a s u rement may help public managers stay matters most in a community. Citizen involvement focused on what rea l ly matters in their commu n i t i e s . increases the social relevance of indicators by Citizen involvement can increase the impact of c ombining facts—hard data—with value-how pe rf o r mance measurement by encouraging managers citizens feel. to look beyond traditional output measures (that often have little meaning to citizens) and instead Tangible benefits can be derived from effective focus on quality of life issues and community citizen involvement and participation not only in goals. When you think about it, why do managers performance measurement, but in public decision car e about gover nment perfo r mance? Th e y want to making as well. Thomas (33) indicated in “Public im p r ove the quality of service deliver y and ultimately Pa rt i c i p a t i on in Public Decision s ” that public the quality of life in the communities they manage. inv o l vement can increase the effec t i v eness of public managers and the decisions they make. For example: The overall goal of involving policy makers and citizens in performance measurement is to build a) Decision quality may improve as citizens and lasting processes that involve citizens in assessing ci t i z en groups add to the informa t i o n available municipal perfo r mance so that gover nment policies for making decision s . That inform a t i on and services reflect the needs of the community. might prevent repetitions of many ill-advised Po l i cy makers, c i t i zens and public managers, public decisions. together, can establish performance measures that b) With citizens involved in making decisions, are meaningful to both citizens and managers. So, acceptance of the decision may incre a s e, for example, instead of just calculating the tons of enhancing the likelihood of successful refuse collected, public managers might also ask implementation. policy officials and citizens to rate the cleanliness of their streets. Instead of counting the number of c) If citizens assist in service deliver y, se r vices may health workers in specific neighborhoods, public become more effective and more effiscient. managers could also ask citizens how well they feel in their own neighborhoods. d) As invo l vement increa s e s ,c i t i z en understanding of gove rnmental opera t i ons may increase The relevance of performance measures increases and criticisms of gove rnmental agencies when managers incorporate citizens’ perceptions. m ay lessen, i m p roving the plight of the A powerful management tool results when public beleaguered bureaucrat. managers combine or compare traditional output me a s u r es with outcome measures that reflect citizen s ’ If we were to revisit the seven-step performance perceptions.Meaningful measures that the average measurement system developed by the National citizen can understand provide citizens with the Center for Public Pro d u c t i v i ty, but this time opportunity to assess and improve government, incorporate citizen involvement in each of the and influence how gover nment services can be made steps, the process would look something like this mo r e res p on s i v e to commu n i t y needs and prio ri t i e s . (italics indicate changes):

PUBLIC PERFORMANCE EVALUATION AND IMPROV E M E N T 5 9 1 . IDENTIFY THE PROGRAMS TO BE MEASURED 4 . EFFICIENCY MEASURES

To start with, programs to be measured must be Set standards for perfo r mance and outcomes (targets clearly defined by citizens. Determining what for accomplishment). programs should be measured is the first step in i d e n t i f ying com mu n i ty pri o ri t i e s . In some Under this step, ci t i z ens and public managers should communities citizens will identify education and specify the conditions under which program goals rec re a t i o n as essential prog ra m s . In other commu n i t i e s and objectives are met. Ci t i z ens and managers need it might be public safety and public transportation to determine what service effec t i v eness and quality that top the list of citizen prio ri t i e s . Choosing what means for a particular program and explicitly state programs to measure is a matter of judgement and ho w they are going to determine whether the stated citizens should have an equal voice in determining terms of effectiveness and quality have been met. what programs are to be measured. Generally, p e rf o rmance measurement systems work best 5 . MONITOR RESULTS when they concentrate on collecting limited, but essential information, about basic programs that Each accomplishment target should be monitored need the most managerial oversight and where on a continuous basis by the management and staff accountability to the citizens is most important. delivering the service and the citizens receiving it. Monitoring provides the citizens and manager with the results needed to decide whether or not 2 . STATE THE PURPOSE AND IDENTIFY THE DESIRED OUTCOMES the performance targets have been met. Systematic and periodic monitoring gives the mana ger and Typically, a government department or agency, in c i t i zens an opport u n i ty to keep tabs on the collaboration with the citizens they serve, initiates operation of the program, and take corrective a strategic planning process to clarify its mission, action when necessary. goals, and objectives. Through this process, the ag e n c y and cli e n t s / c i t i z ens identify the outcom e s ,o r res u l t s ,t h e y want to achi e ve through agency prog ra m s . 6 . PERFORMANCE REPORTING Citizens and managers can only measure the perfo r mance of a prog r am if its purpose is cle a r ly stated. A good performance measurement system reports program results on a regular basis. Reports are more than an internal management tool and are 3 . SELECT MEASURES OR INDICA T O R S made public.They are shared with elected officials, ci t i ze n s , the media and other gover nment watchd o g s . A good system uses a few selected indicators to The report focuses on what was accomplished and me a s u r e outcomes and perfo rm a n c e . Most govern m e n t what it cost the public. Reports should be brief, p ro g rams that have established perf o rm a n c e co nvey informa t i o n graph i ca l ly,and include minimal measurement systems incorporate the indicators explanatory information. Information should be described in the previous section: input, capacity, pr esented in such a way as to facilitate comp a ri s on s o u tp u t , o u t c om e, e f f i c i e n cy and pro d u c t i v i ty. over time, comparisons with similar organizations, Citizens and managers should determine the most comparison with best programs nationwide and against pre-set targets.Performance data should be appropriate measures for each program, rather presented in a way that is meaningful to citizens so than rely exclusively on the data collected by they can understand what is happening in their individual programs. Good program measurement ne i g h b o r hood as well as the commu n i t y as a whole. systems develop citizen surveys to assess citizen satisfaction and perceptions. For example, in a sanitation department the input measures might 7 . USE OUTCOME AND PERFORMANCE INFORMATION be the amount of labor hours; the operating budget; ( A N A LYSIS AND A C T I O N ) the number of veh i c les on the roa d . Out put measures could include tons of refuse collected; number of I n f o rm a t i on from an effe c t i ve perf o rmance households served; number of missed collections. measurement system is regularly used in program Outcome measures might include the percentage planning to reevaluate goals and objectives and to of citizens who are satisfied with refuse collection; adjust priorities. Another use is in managing for the percentage of citizens who rate the cleanliness results, where outcome information is used to of their street as above average. p romote continuous improvement in pro g ra m

6 0 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing op e ra t i o ns and res u l t s . A well- d e veloped perfo rm a n c e 5. Epstein, Paul D. Using Performance Measurement in Local Government: A Guide to Improving Decisions,Performance and m e a s u rement system will enable citizens and Accountability. National Civic League Press,1988. managers to spot weaknesses and challenges to 6. Fi s c h e r, R. J. (1994). “An Overv i ew of Pe r f o rm a n c e program delivery, as well as program strengths and Measurement.” Public Management 76: pp. S2-S8. opportunities for improvement. 7. Glover, Mark. (1992). A Practical Guide for Measuring Program Efficiency and Effectiveness in Local Government, Tampa, FL: Innovation Group Publication. 8. Government Accounting Standards Board, Service Effort and Accomplishment Measures. 1997. Norwalk, CT. FINDING A S S I S T A N C E 8a. Government Performance Results Act of 1993. Washington, D.C.: Government Printing Office. 1993. The field of public sector performance evaluation 9. H a l a c h m i ,A r i e , and Marc Holzer, eds. “Productivity Measurement is changing rapidly. Fortunately, state-of-the-art and Management.” (Public Productivity Review, 44) San Fra n c i s c o , kn o wledge and best practices are now more rea d i l y CA: Jossey-Bass,1987. available. The World Wide Web offers access to 10. Harvey, J. (1987). “Measuring Productivity in Professional dozens and dozens of helpful sites. Of course, all Services.” Public Productivity Review (44): pp. 29-38. such information must be adapted to specific 11. Hatry, H. P., Blair, L., Fisk, D., Greiner, J. M., Hall, J. R. Jr,& Schaenman, P. S. (1992). How Effective Are Your Community national and local needs. Services? The Urban Institute and ICMA: Washington,DC. 12. Hatry, H. P., Fountain, J. R., Jr., Sullivan, J. M., & Kremer, L. At the National Center for Public Productivity (1990). Service Efforts and Accomplishments Reporting: Its at Rutgers University we maintain web pages Time Has Come. Governmental Accounting Standards Board devoted to the improvement of public perform- (GASB): Library of Congress Catalog Card Number: 90-80879. ance, as well as links to best practices worldwide. 13. Ha t r y, Ha r ry P., et al. Efficiency Measurement for Local Gove r n m e n t Services. Washington,DC: The Urban Institute,1979. Those pages and links may be accessed at: 14. Hatry, Harry P., et. al. How Effective Are Your Community http://www.newark.rutgers.edu/~ncpp Services? Washington,DC: The Urban Institute,1977. 15. Hatry, Harry P., et. al. Measuring the Effectiveness of Basic Municipal Services. Washington,DC: The Urban Institute,1974. 16. Ha t r y, P. & Fis k , D. (1992). “Measuring Productivity in the Public Sector”. In M. Holzer (ed.). Public Productivity Handbook. Ma r c e l C O N C L U S I O N Dekker: NY, (pp.139-160). 17. Holzer, M. and Halachmi,A. (1996). “Measurement as A Means While it may be costly and time-consuming of Accountability.” International Journal of Public Administration, to include citizens or their re p re s e n t a t i ves in 19 (11&12),(pp. 1921-1944). the measurement of gove rnment perf o rm a n c e, 18. Holzer, Marc, (ed.). Public Productivity Handbook. Marcel ultimately the performance measurement system Dekker, 1992. developed will be more useful and meaningful. 18a.Holzer, Marc. “Focus: Productivity Improvement in New York State—the Science and Art of Capacity Building,” in Managing The data collected will have an impact on policy New York State, Number 1,1988. Albany, New York: Governor’s and program administration. The performance Office of Employee Relations. The ten steps contained in this m e a s u rement sys t e m , rather than focusing on article are based, in part, on Holzer, et.al., Managing for managerial accomplishments and administrative Productivity, including previous work by Constance Zalk. achievements,will address quality of life issues and 19. H o l z e r, Marc and Kathe Callahan, G overnment at Wo r k , community goals and aspirations. Governments Thousand Oaks,CA: Sage Publications,1998. will be measuring to improve services, to make 20. Ho l z e r , Marc and Vatche Gabrielian. “Per f o r mance Measurement and Improvement.” Newark, N.J.: National Center for Public government more responsive to the needs of the Productivity, 1998. citizens they serve. 21. Kymlicka, W. and Norman, W. (1995). “Return of the Citizen: A Survey of Recent Work on Citizenship Theory,” In Beiner, R. (ed) Theorizing Citizenship. Albany,NY: SUNY Press. R E F E R E N C E S 22. Managers Guide for Improving Productivity. Washington, D.C.: Office of Intergovernmental Personnel Programs, United States 1. Am m o n s , David N. Municipal Benchmarks: Assessing Local per- Office of Personnel Mangement,1980. formance and Establishing Community Standards. S a g e ,1 9 9 6 . 23. McGowan,R. P.,& Poister,T. H. (1985). “Impact of Productivity 2. Co l w e l l , W. L., & Kol e t a r , J. W. (1984). “Per f o r mance Measurement Measurement Systems on Municipal Pe r f o rm a n c e ,” Po l i c y for Criminal Justice: The Federal Bureau of Investigation (FBI) Studies Review, Vol. 4,No. 3: pp. 532-540. Experience” Public Productivity Review VIII (3): pp. 207-224. 24. Milakovich, Michael E. “Total Quality Management for Public 3. Conference Proceedings of the 1993 Managing for Results: Service Productivity Improvement.” in Marc Holzer, ed., Public Performance Measures in Gove r n m e n t . Aus t i n , Tex a s , October 27- 29. Productivity Handbook, New York: Marcel Dekker, 1991. 4. Epstein, Paul D. (1992). “Measuring the Performance of Public 25. Mi t c h e l l , J. (1997). “Representation in Gove r nment Committees.” Service,” in M. Holzer (ed.), Public Productivity Handbook. Public Administration Review. March/April, Vol. 57, No. 2, Marcel Dekker: NY, (pp.161-194). p. 160-167.

PUBLIC PERFORMANCE EVALUATION AND IMPROV E M E N T 6 1 25a.National Performance Review. Washington, D.C.: Government 30. Romzek, Barbara S. and Dubnick, Melvin J. “Accountability.” Printing Office,1997. International Encyclopedia of Public Policy and Administration. Jay Shafritz, Editor-in-Chief. New York: Westview Press, 1998. 26. Os b o rn e , David and Ted Gaebler, 19 9 2 . Reinventing Gove r n m e n t . pp. 6-11. Reading, MA: Addison-Wesley. 31. Schacter, H. L. (1997). Reinventing Government or Reinventing 27. Osborne and Plastrik, 1992. Banishing Bureaucracy. Reading, Ourselves: The Role of Citizen Owners In Making a Better MA: Addison Wesley. Government. Albany, NY: SUNY Press. 28. Performance Measurement: A Guide for Local Elected Officials. 32. Stivers, C. (1990). “The Public Agency as a Polis: Active Washington, D.C.: The Urban Institute,1980. Citizenship in the Administrative State,” Administration and 29. Poister, Theodore H., James C. McDavid and Anne Hoaglun Society, 22 (May),86-105. Magoun. Applied Program Evaluation in Local Government. 33. Thomas,J.C. (1995). Public Participation in Public Decisions. Lexington,MA: Lexington Books,1979. San Francisco: Jossey-Bass.

6 2 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PART III A )Monitoring Evaluation in a Re s u l t s - B a s e d Management Approach B) Evaluation Initiatives in Science and Technology

E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing EVALUATION PARTNERSHIPS 10 Niels Dabelstein,Head of Evaluation Sec re t a r iat DA N I DA ,O E C D / DAC WP

Donor Government financed aid is expenditure of taxpayers money outside national borders.In addition, aid is often spent through partners or co-operating governments over which the donor government has no formal control. This is a feature distinguishing aid evaluation from evaluation of most other publicly funded programmes. For that reason alone, the accountability function may very well be the most important for aid evaluation. Not the least in periods of aid fatigue and pressure on government budgets, continued public support for aid expenditure is contingent upon aid agencies’ accountability and their ability to document aid results.

Generally, aid evaluation performs several functions: it provides lessons for managers and staff of ongoing aid programmes thereby leading to improvements in implementation; it establishes information and documents experience that can be used in planning and designing future aid programmes;it is used by managers, strategy- and policy makers for revising existing — and for devising new — aid strategies and policies; and finally aid evaluation delivers information on efficiency and effectiveness of aid programmes, and thus ensures accountability towards politicians and the public.

While these functions apply to both donors and partn e r s , the scope differs substantially.

Donors are primarily interested in evaluation the effectiveness, efficiency and impact of “their” investment. But even the best evaluations face difficulties when it comes to establishing causal linkages between the individual donor’s efforts and developmental changes. The individual donor’s evaluations, be they of projects, programmes, or sector support, run the risk of ascribing positive development to a limited intervention, when the reality is that development is the result of the synergetic effect of all interventions by the developing countries themselves as we l l as the support provided by dono r s , enhanced or hampered by external factors.

Our partners – the developing countries – have a wider scope: they would focus on overall performance of projects,programmes, and sectors, and even the entire economy. Evaluation is thus an aid to resource allocation and prioritisation, to sound management, and to accountability towards politicians and the public.

In short you could say that donors evaluate development aid while developing countries evaluate development.

Evaluation institutions exist in many developing countries, but most have little impact on policy and management decisions due to a number of barriers: Poor demand; lack of a culture of accountability (often related to ethics or corruption); absence of evaluation, accounting, or auditing skills; and lack of feedback mechanisms into decision making processes.The strongest barrier appears to be the lack of demand: Credible evaluation is a function of good governance i.e. demand for accountability more than of institutionalisation of evaluation and/or professional capacity.

E V A L UATION PA R T N E R S H I P S 6 5 How, then, can donors contribute to enhancing focus, setting new demands for host-country evaluation capacity in our partner countries. evaluation institutions.

Sup p o r t to building eva l u a t i o n cap a c i t y in devel o p i n g ■ R e g i on a l , s e c t o ral and pro g ra m m e / p ro j e c t countries has been on the agenda for several years, evaluations become more useful if they are and the DAC Working Party on Aid Evaluation based on a co-ordinated approach linked to a has promoted the concept actively, through several national evaluation system, particularly with re g i onal seminars organised jointly with the respect to methodologies and data needs. regional development banks (Africa in 1990, Asia in 1992, Latin America in 1993, and in Africa These findings generally apply to both donor and again in 1998) and through the members’ direct recipient agencies. assistance, progress towards establishing credible and capable evaluation functions in developing While the donors can promote and facilitate countries has been slow. In recent years the multi- ev a l u a t i on ca p a c i ty building through coll e c t i ve lateral institutions have been particularly active, as activities, the main support has been, and will witnessed by this seminar. continue to be, channelled through multilateral and bilateral donor agencies. The Study identified In 1996 the Working Party conducted a survey of several areas where donor agencies could strengten do nor support for and experiences from eva l u a t i o n and mainstream ECB work within their agencies capacity building. The Key findings of the study, and in collaboration with development partners: which I believe are still valid were: ■ ■ Promote an agency ECB support policy or Sustainable eva l u a t i o n institutions need support strategy, particularly in view of new aid forms at the highest policy and management levels being introd u c e d ,i n c luding prog r amme assistance and should be able to demonstrate its use to for institution and capacity building as part of these levels. The design of evaluation systems good governance initiatives at national and must also take into account the specific sectoral levels. government and administrative culture in the country/organisation. ■ Advocate and stimulate the evaluation issue in country dialogues and sector programme ■ Political advocacy and senior management assistance. demand should be preconditions for ECB su p p l y activities and must be linked to the issue ■ Provide technical advice to operational units of governance.A long-term strategy is needed for effective interventions in both cases. responsible for ECB support activities and advise on training facilities and materials on ■ Sustainable and effective evaluation systems evaluation issues. must have a legal foundation or a firm statutory ■ organisational regulation. Su p p o rt the establishment of tw i n n i n g ar r angements between other domestic eva l u a t i o n ■ An eva l u a t i o n unit’s independence from line man- institutions and host country institutions. agement is importa n t , as is the securit y of care e r ■ possibilities for evaluation staff and managers. Assist in securing consistent ev a l u a t i on methodologies and terminology in the ECB ■ The scopes of nation a l - l evel perf o rm a n c e support activities of the agency. evaluation and performance auditing systems ■ are moving closer to each other, although the C o - o rdinate their ev a l u a t i on pro g ra m m e s former is likely to be more closely integrated with host countries and other donors in in the planning process, while the latter tend order to optimise the use of resources and the to focus more on accountability to the policy- constrained capacity of recipient countries’ making level. The choice of approach may, evaluation systems. however, depend on other factors, such as political commitment, the legal framework ■ A r range joint-ev a l u a t i ons with a genuine and institutional capabilities. participatory approach, where the needs of both parties are incorporated from the start ■ Development policy and aid tend to shift and where the capacity building element is from a project/programme to sector/policy taken into account specifically.

6 6 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing I will focus on the very last of these possibilities: DAC Member countries also use joint evaluations Joint-evaluation – or in current lingo: Partnership to improve partner countries’ evaluation capacity. in evaluation. The deeper partner country representatives are involved in evaluation process, the more they are In line with the emphasis on deve l o pment exposed to new eva l u a t i o n methods and approa ch e s , pa rt n e r s h i p , lo c al own e r s h i p , and good govern a n c e , and the learning process is smoother as a result. the donors should use the joint-evaluation as a vehicle to assist in developing an “evaluation The degree of partnership between partner and cu l t u r e”. Thus the value of eva l u a t i o n as a management do nor countries will vary according to the objectives tool as well as an instrument for shared lesson and uses of the evaluation. In some cases, DAC le a r ning and accountability could be demons t ra t e d . Members have involved partner country represen- To succeed in this, donors need to co-ordinate tatives in the evaluation process by including them evaluations and ultimately let evaluations be co- in the evaluation team to define terms of reference ordinated by the recipient countries. (i.e. objectives, questions, scope, methods, uses, etc.). On other occasions, they took part in the evaluation process by assuming different roles such Although joint evaluations do take place they are as that of key informant, a member of a focus s t i ll infre q u e n t , and have pri m a ri ly con c e rn e d g ro u p, a re s e a rch e r, an interv i ew e r, a country donors’jointly financed programmes,or been eval- regional expert, or simply by gathering data. u a t i ons of mu l t i l a t e ral agencies by groups of donors. Common to most of these evaluations is In most cases, partner country representatives have that the developing countries have played a minor - been involved in joint evaluations to gather and if any role in their planning and execution. Rarely analyse data. Generally the International Finance ar e the recipient countries invo l ved until an eva l u a t i o n I n s t i t u t i ons (IFIs) have not included part n e r scheduled by the donor is initiated and mostly the countries in the planning and design phases of the recipient gove rnment is inv o lved in prov i d i n g evaluation cycle, while bilateral donors have done information to the donor but not in the analyses so more often. Some [For example, Denmark, the and final assessment of performance. Evaluation Netherlands, New Zealand, Switzerland and the programmes are prepared in response to agency E u ropean Com m i s s i on] have inv o lved part n e r needs for lesson learning and for accountability, representatives more fully in the evaluation process and is geared to the planning and programming by including them in the follo wing pha s e s : pl a n n i n g cycle of the agency, not to the planning cycles of and designing; gathering and analysing data; partner countries. identifying evaluation findings, conclusions and recommendations. Despite these difficulties the donor community considers it crucial to more fully engage with developing countries in a partnership strategy for evaluation.By jointly analysing opportunities both C O N S T R A I N T S donor and partner countries enhance their sense of ownership and feel responsible for results. Then, A1998 review of the DAC Principles for the evaluations of development assistance programmes E v a l u a t i on of Deve l o pment Assistance alre a d y can be effectively carried out with the participation identified some of the difficulties encountered of donor and partner countries, as both would when implementing partnership in ev a l u a t i on . sh a r e the interest in accountability for res u l t s ,l e s s on s These included recipients’unwillingness,or lack of learned and improved programme effectiveness. interest to participate in joint evaluations; time c on s t raints and higher costs; c om mu n i ca t i on Evaluations carried out with the participation of problems and delays; increased complexities and, partner country representatives are more effective oc ca s i on a l ly, po l i t i c al obstacle s . The fact that partn e r as they generally lead to a fuller use of the findings. countries have inadequate knowledge of donor In fact, joint-evaluations can generate multiple co u n t ri e s ’ eva l u a t i o n policies may also be one of the benefit, including: i) a better understanding of the possible impediments to partnership in evaluation. results;ii) greater acceptance or sense of ownership of the res u l t s ; iii) enhanced cred i b i l i t y and validity Lack of partner government commitment to the of the findings; iv) increased sense of responsibly pro c e s s ;d i s c r epancies between the donor and partn e r and incentives to follow the recommendations country’s objectives or reasons for carrying out the formulated by the evaluation;v) possibly improved ev a l u a t i on ; and insufficient ev a l u a t i on ca p a c i ty partner country’s evaluation capacity. within the partner country have been identified as

E V A L UATION PA R T N E R S H I P S 6 7 the most important impediments to effe c t i ve Donor and partner countries’ governments should partnership in evaluation. be committed at senior level to a joint evaluation pro c e s s . At the same time, Pa rl i a m e n t ,s t a k e h o l d e r s , On the demand side, although many partner countrie s beneficiaries and civil society in both donor and have created an evaluation unit attached to the partner countries should be kept informed about planning or finance ministry, they are not yet the results of joint evaluations as so to enhance the s t ron g ly committed themselves to ev a l u a t i on sense of ownership not onl y of the eva l u a t i o n findings activities. Governments are not yet fully results but also of the development programme. ori e n t e d . The need for accountability, pa rt i c i p a t i on , transparency, rule of law has only recently been As participatory processes usually need longer considered necessary for improving governance time frame, it is crucial to plan sufficient time to practices. Moreover, the demand for evaluation carry out a joint-evaluation. It is necessary to plan comes more often from the donors community during the design phase, the number of evaluators (w h i c h has limited influence), than from the partn e r and financial resources needed, bearing in mind countries’ Parliament or taxpayers. that joint eva l u a t i o ns general ly invo l ve more people and call for more co-ordination. On the supply side, di f fe r ences in dono r s ’ eva l u a t i o n and management requirements have hindered the When eva l u a t i o n cap a c i t y is judged to be insufficient, de vel o p ment of eva l u a t i o n cap a c i t y of partner countrie s . it is suggested to carry out some actions to support its devel o p ment in partner countries simul t a n e o u s l y Public sector reforms can open new opportunities to the evaluation. Donors’ agencies dispose of to effec t i ve l y stimulate demand and increase supply different tools to promote the development of of eva l u a t i o n and perfo r mance management syst e m s . evaluation capacity, notably “learning by doing” SI D A has suggested that,in order to develop eva l u a t i o n training (e.g. their involvement in the definition of capacity effectively, development agencies could questions, data collection, methods); or technical also provide financial and technical resources to assistance to personnel, as part of monitoring conduct programme evaluation separately from programme/project, or other kinds of training. donors’projects.This could also create demand for evaluation independently from the donor needs. In sum, it is rare that donor and partner counties undertake joint evaluations in full partnership. The most important factors impeding effective joint- evaluations are lack of governments’ commitment THE FUTURE to partnership in evaluation.Different conceptions of the purpose of evaluations and limited time and It is important to keep in mind that to have resources availability. successful partnership in evaluation partner and donor countries should, at least partially, share I have attempted to point out the advantages of the same objectives. This normally calls for joint joint-evaluations, to identify the main obstacles or elaboration of the goals, and thus partner repre- constraints,and indicated a few ways in which the sentatives’ participation during the programming partners in development may enhance evaluation phases. For these reasons, evaluations should shift capacity through joint-evaluations. their primary objective away from the control and ac c o u n t a b i l i t y functions , to become more a learni n g I hope that during the next 45 minutes we can i n s t rument to improve deve l o pment assistance identify practical ways of moving joint-evaluation performance and to learn lessons. a few steps further.

6 8 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing EVALUATION TOOLS AND METHODOLOGIES: Case Studies 11 Jacques Toulemonde, Director, Centre for European Evaluation Expertise

THE INTERNATIONAL MODEL OF PROJECT EVALUA T I O N

Donor institutions have disseminated the practice of evaluation worldwide through a systematic review of the projects that they fund. This has given birth to a “model of project evaluation” that is typified hereafter. The evaluation fo ll o ws the logical fram ew o r k of the pro j e c t ,w h i ch means that it looks at activities, outputs, outcomes and the problem to be solved. All evaluation criteria are sys t e m a t i ca l ly applied: rat i on a l e , rel ev a n c e , ef fe c t i ve n e s s ,e f f i c i e n c y, and sustainability. Unintended effects are also valued.

Donor institutions play the leading role in the evaluation system. They set up the eva l u a t i o n agenda and the rules of the game. This does not prevent parti c i p a t o r y methods to become more and more familiar.Participation typically extends to national authorities responsible for the project,to the project management and to beneficiary groups on the project site. Project evaluation and monitoring go hand in hand. Project managers periodically report on the progress of implementation, which generates monitoring data. As far as these data are properly filed, project evaluators can rely upon a sound basis of field data. Evaluation therefore consists of a quick external review. Additional collection of field data is typically limited to a few interviews and a site visit.

The evaluation system follows the project cycle. It may include three different exe rc i s e s : ex ante, on- g oing and expost. These three types of eva l u a t i o n res p e c t i ve l y correspond to the phases of project design, implementation and termination.

The description above is a deliberate caricature. In fact, donor institutions have a more complex practice including joint evaluation with national authorities, country evaluations, thematic evaluations, coordination of donors’ evaluations, etc. Is it useful to typify the model of project evaluation in such a simplistic way? For the sake of this presentation, the answer is yes since my purpose is to set up a benchmark that everybody is familiar with.

In order to make the above model workable the project under evaluation must fulfil several conditions. First, it must be reasonably simple. This means that it delivers a single type of output to one target group and that it aims at solving one major problem. Second, the project is supposed to involve one single line of st a k e h o l d e r s ,n a m e l y the line, wh i c h connects headquarters of the donor institution to the recipients on the project site. Th i rd , the project is under the res p on s i b i l i t y of an identified management team that produces periodic rep o r ts and moni t o ri n g data. Finally, the project runs from A to Z with a beginning and an end. Its objectives should be made explicit from the very beginning.

E V A L UATION TOOLS AND METHODOLOGIES: Case Studies 6 9 A EUROPEAN MODEL familiar with evaluation ten years ago. There was therefore a need to support the development of an OF POLICY EVALUA T I O N evaluation culture across Europe. This was the rationale for launching the MEANS program in This paper builds upon a very different evaluation the early 90s. model, which applies to a large policy of the European Union. Through a contrasted view of MEANS is a French acronym for “Methods for these two models, I will describe some problems Evaluating Ac t i ons of a St ru c t u ral Na t u re” that arise when practice shifts from project eva l u a t i o n (“Structural” because the policy aims at improving to policy eva l u a t i on . I will also propose methodological social and economic structures). This program, in recommendations for solving these problems. The which I have been closely involved, allowed for Cohesion Policy is a major stake for the European several years of research to be undertaken on Union. It principally aims at developing those ev a l u a t i on practice in the field of social and regions that lag behind in terms of social and economic development. The main activities of the ec on o mic wealth. Ma n y other goals are also pursued MEANS program were bottom up. It consisted of like reducing unemployment, promoting an envi- identifying problems and of designing solutions, ronmentally sustainable development, etc. This not of giving lessons. policy is the second biggest in terms of cost for the European taxpayer. It is implemented through; The outputs of the MEANS program comprise a hundreds of programs are managed and funded in body of methodological experiences, ideas and partnership with national and regional authorities. innovations which were presented in seven 50-60- page handbooks. The programme also published In this context, the term “program” applies to: (1) methodological reports and quarterly newsletters. a coordinated set of different types of projects (e.g. It organised training seminars and three European training courses, advice to enterprises, subsidies to c on fe rences (the most re c e n t , in Sev i ll e, w a s em p l o yer s ,e t c . ) , of which (2) the objectives are cle a r ly attended by just under 600 practitioners). An stated (e.g. placement of the long - t e r m unemployed ) effort was made to take full advantage of existing without implementation being defined entirely in ex p e ri e n c e and recent innova t i on s , pa rt i c u l a r ly in advance and, fi n a l ly (3) which has a limited budget the follo wing areas: measurement of employment and duration (generally five-years). effects, development of systems of indicators and evaluation of macro-economic effects. Through Within the framework of the European Cohesion the evaluations initiated at the European level and Policy, programs are particularly complex. They the work that has gone into building on these expe- typically encompass dozens of different types of rie n c e s , sufficient knowledge has now been acquired ac t i v i t i e s . Their global objective is like an umbrell a to produce a com p re h e n s i v e publicat i o n accessible to for several operational objectives and priorities. In a wide public. The result is a collection covering all most European regions, programs follow each aspects of the eva l u a t i o n of socio-econo mic devel - other with incremental changes every five years.In op ment policies (European Commission, 1999). this case, the concept of program cycle with a beginning and an end is somewhat artificial. For At the end of my long tour into this large Europ e a n these reasons,I tend to consider that what is under policy, I conclude that it requires an evaluation evaluation has many features of a policy and I will model that is very different from that of project use this term hereafter in order to be better eva l u a t i on . Ex p r essing my concl u s i o n in a provo ca t i v e understood by non European readers. way (that would certainly not be endorsed by European authorities),I would say that in the case The evaluation system of the European Cohesion of the European Cohesion Policy: (1) systematic Policy was created in 1989 and it is now ten years evaluation along the logframe technique is not old.The system is one of the most ambitious and wo rk a b l e , (2) systematic eva l u a t i o n on ever y sites is comprehensive in the world. It makes evaluation not worka b l e , (3) systematic rel ying upon moni t o ri n g compulsory in every European region at ex ante, data is not workable, (4) systematic following of interim and ex post stages. It puts much emphasis the policy cycle is not workable. on independent external evaluators. It establishes an obligation to evaluate that applies to the Union I develop these points in the next section s ,t o g e t h e r itself, but also to the fifteen member states and to with a series of recommendations that build on the hundreds of regional authorities. The least to say European practice and that help making policy is that all these authorities were not that much evaluation workable.

7 0 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing SCREEN THE POLICY WITH A LOGICAL SELECT KEY ISSUES FOR DIAGRAM OF EXPECTED IMPA C T S IN-DEPTH EVALUA T I O N

One of the first steps of any eva l u a t i o n is to understand Evaluation experience of the European Cohesion the cause-and-effects assumptions that connect Policy shows a wide variety of ways of addressing the activities and their expected impacts. This is evaluative questions. Fairly often, the evaluation easily done by using the logical framework if the team is expected to treat ten or more questions activities and expected impacts are in small numbers. within a few mont h s . Other eva l u a t i on s , by cont ra s t , This is not so easy in the case of the European devote a whole year to the in-depth analysis of a Cohesion Policy. One single evaluation is often single question. supposed to assess thirty or more different kinds of activities and ten or more diffe rent kinds of The first option is in line with the model of proj e c t impacts. Such a complex setting is not surprising evaluation in which the donor institution tends to ask ever y possible question. In this case the for those who evaluate national policies. Ho w many evaluator is asked a list of questions like: did the logframes should be developed for describing such various activities produce their intended effects a policy? Certainly too much for this technique to ( e f fe c t i veness)? We re the effects obtained at be workable. reasonable cost (efficiency)? Was it worth trying to obtain this or that effect (relevance)? And what In order to tackle this pro b l e m ,E u ropean authorit i e s about unintended effects? Since the European ha v e first rec o mmended that evaluators and prog ra m C oh e s i on Po l i cy typ i ca lly encompasses many designers establish a hierarchy of objectives as activities and many effec t s , the number of questions represented by an objective tree. This technique is multiplied in such a proportion that evaluators consists of a diagram in which the most global cannot convincingly answer all of them. They objective is placed on the far left, and the most often admit that “available data do not allow to detailed objectives on the far right. Each objective assess the impacts”,or, even worse, they answer the can thus be broken down into sever al sub-objectives questions without any evidence basis. which, in turn,are broken down into objectives on a more and more specific level. In practice, neither On the contrary, the European practice has shown the objectives nor the relations between them can that sound evaluation works tends to address a be articulated so simply. In almost all cases, a limited number of questions . This allo ws eva l u a t o r s given activity may contribute to achieving several to deriv e their concl u s i o ns from an in-depth analysi s , ob j e c t i v es of general interes t . This is cont ra d i c t o r y including collection of field data when necessary. with the structure of a “tree” and this calls for something resembling more a “network” or a “web” In the following example, the evaluation steering of objectives and activities. group considered several questions before deciding to analyze three of them in more depth. The story In order to address this problem, an alternative started in 1997 when public authorities in the technique has been proposed in the MEANS French region of Franche-Comté had to evaluate an industrial regeneration program partly funded collection. It is called: “logical diagram of expected by the European Union. The evaluation steering impacts”. The diagram sketches the main causal group devoted its first two meetings to the identi- links that connect the activities, situated on the f i ca t i on and selection of ev a l u a t i ve question s . l e ft , and their successive re s u l t s , o u t c omes or Seven fields of questioning were identified. These impacts, situated on the right. Contrary to an were then discussed in order to formulate, for each objective tree which draws a simplified structure of of them, a realistic and useful evaluative question. objectives,the logical diagram of impacts attempts to mapping the causal links between the activities These questions drafted by the steering group under evaluation and their ultimate objectives. were as follows: (1) Which categories of projects have the best impact on the attractiveness of the In the European case, this alternative technique territory? (2) Has support to enterprise helped to has provided a workable means for cle a r ly describ i n g create or maintain jobs that are firmly rooted in th e the logic of a complex policy. In addition, the ar ea? (3) Have services for enterpr ises cont ri b u t e d diagram provides a means to quickly screen the s u f f i c i e n t ly to the dive r s i f i ca t i on of econ om i c whole policy before deciding to scope some parts ac t i v i t y? (4) Does training serve to raise qualificat i on s in more depth. or rather to make them more relevant? Which of

E V A L UATION TOOLS AND METHODOLOGIES: Case Studies 7 1 these two expected effects is the most necessary? di f fe r ent activities in terms of cont r ibuting direc t l y (5) Is the probable impact of higher education or indirectly towards a given objective. and research projects sufficient? (6) What is the ef fe c t i v eness of the differ ent cat e go r ies of projects in Such analyses deserve to be selected when a given te r ms of crea t i o n and maintenance of employme n t ? objective is considered as strategic (assuming that (7) Whi c h cat e go r ies of projects guarantee or improve the corresponding issue is most likely to gain from equal opportunities between men and women? an informed debate), or when an objective is controversial (assuming that the corresponding An in-depth study of all seven questions would issue is most likely to gain from the clarification have been too costly. It was therefore decided to of judgement criteria). The notion of thematic limit the number of questions to study in depth to eva l u a t i o n cal ls to mind that of in-depth eva l u a t i on . three. Certain questions were eliminated so that In the framework of the European Cohesion th e r e would not be duplicat i o n with other eva l u a t i on s Policy, some thematic evaluations examine how and studies underwa y in the reg i on . Fin a l ly, qu e s t i on s the same impact (e.g. the com p e t i t i veness of (1), (4) and (5) were judged priorities. SMEs) is obtained by various activities (type 3 of in-depth analysis) or how interventions financed The first type of in-depth analysis applies to a in a specific domain such as Research & cluster of similar activities (as for questions 2,3,4,5 Development produce several kinds of impact ab o ve) . In this cas e ,evaluators can easily implement (type 1 of in-depth analysis). s u rveys of beneficiary gro u p s . Pri m a ry data obtained in this way provide indications relative to In the fram ew o r k of the Cohe s i o n Pol i c y, th e m a t i c gross effects. The analysis of causality may also be evaluations have proven to be the most useful ones carried out, confounding factors may be identified, for policy-makers at European level . This conf i rm s deadweight may be estimated, evaluation design that policy eva l u a t i o n should focus on a few strat e g i c m ay include com p a ri s on with non - b e n e f i c i a ry issues instead of asking all possible questions. groups. This type of analysis focuses on the success or failure of a given activity through its many different impacts. I N VO LVE ALL RELEVANT PARTNERS Suc h an in-depth analysis deserves to be underta k e n for expensive activities in terms of budgetary IN AN EVALUATION STEERING GROUP appropriations (assuming the stakes involved in decision-making increase in proportion to the In those European regions where the culture of budget). It is also relevant for new or innovative public managers was old fashioned, the evaluation activities (considering that they are experiments, practice has first started as a bureaucratic exercise. which need to be validated by an evaluation). The A typical behavior was to hastily contract out second type of in-depth analysis focuses on a ev a l u a t i on works at the lowest possible cost particular group of recipients, such as the long- ( s ometimes 0.01% of the expenditures under term unemployed,creators of new enterprise, small eva l u a t i o n) and with the lowest possible own e r s h i p . farmers or women (as for question 7 above). The A current practice was to use standard terms of evaluation is performed from the point of view of re fe re n c e, s ca rc e ly orientated tow a rds re g i onal these recipients. It analyzes their needs, it uses particularities, badly suited to the policy-makers their judgement criteria and it tries to understand ex p e c t a t i o ns and hardly motivating for the managers. their interaction with the policy. In this situation, As long as the writing up of the terms of reference the ev a l u a t i on produces con cl u s i ons on the has been considered an office job, it has been c om p a ra t i ve impact of diffe rent activities that difficult to make them properly adapted to the directly address the group under study. It can also needs of evaluation users. show how these different activities reinforce or contradict one another with regards to the group This poor state of practice has quickly changed under study (synergy effects). The third type of and evaluation becomes a managerial exercise. in-depth analysis concerns an expected impact or a More time is invested in drafting the terms of category of impacts (as for questions 1 and 6 reference of exercises that cost up to 0.8% of the above).The question then applies to every activity expenditures under evaluation. In the best cases, under evaluation. It focuses on their respective the com m i s s i oner of the ev a l u a t i on officially contribution to the production of the desired mandates a steering group from the outset and for impact. In this situation,the analysis concludes on the entire duration of the process. This group is no the comparative strengths or weaknesses of the lo nger a close shop that gathers funding institutions ,

7 2 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing managers and operators. It is increasingly open A standard procedure for policy evaluation consists to other partners who hold various stakes (e.g. of perfo r ming specific eva l u a t i o ns of its comp on e n t s , so mebody speaking with the voice of “env i r onm e n t ” and then of synthesizing the conclusions at the in a policy that is mainly geared at economic policy level. Evaluation by synthesis has faced development). The steering group is then given serious methodological shortcomings when it has an active role in the preparation of the terms of applied to heterogeneous activities. Since the eva l u a t i o n reference, which implies the organization of one or of each comp o nent uses its own indica t o r s ,a n a lyse s two or even three meetings before the external distinct impacts, and applies different criteria, the evaluator is chosen. This makes it possible to draft final synthesis can only be the juxtaposition of precise evaluative questions, to check that the partial evaluations. The very notion of synthesis essential (or embarrassing) questions have not therefore loses its meaning.An evaluation report been forgotten, to reject questions which are more can be produced in which a long chapter is devo t e d a matter of auditing or control, to select priority to each activity, but in which the overall conclusion questions, and to specify the judgement criteria is nothing more than a few lines with very poor or and norms . An extreme but salutary choice may be tenuous contents. the postpo nement or can c e ll a t i o n of the eva l u a t i on , when the steering group foresees that its results In these circumstances it is impossible to make will not be used. relevant recommendations for policy-makers. For ex a m p l e , it is not possible to justify rec om m e n d a t i on s This preliminary stage does not involve additional for the reallocation of budgets between activities wo rk l o a d . It is rather an alterna t i v e way of organizing because there is no argument showing that one is the evaluation process.In fact,the first meetings of mo r e effec t i v e or successful than another. To prod u c e the steering group provide an opportu n i t y to struc t u r e genuine synthetic con cl u s i on s , the ev a l u a t o r s the evaluation process in a clear, fair and useful should apply common criteria at policy level. This m a n n e r, a task that external evaluators oft e n recommendation, which is simple in theory, has undertake with great difficulties. proven to be hardly workable in the European model of policy eva l u a t i on . In most cas e s , eva l u a t o r s tend to use the crit e ri o n of budgetary cons u m p t i on , wh i c h has even been termed in Italy as the “cap a b i l i t y DISTINGUISH POLICY EVALUATION to spend public funds”. The exact opposite of an FROM PROJECT EVALUA T I O N evaluation culture!

In the framework of the European Cohesion Much attention is presently paid to measuring Policy, evaluation teams tend to view complex performance. Performance is often translated into policies as a series of components, and therefore a ratio of actual vs. expected outputs, which is also a n a lyze each of these com p onents separa t e ly. te r med “com p l e t i o n rat e” . The fact that an activity Components (or activities) tend to be evaluated in rapidly achieves its objectives may be an argument terms of their own success criteria. For example an in favor of its effectiveness or success, but this a c t i v i ty which consists of improving roads is argument is necessarily weak. Some activities have evaluated in terms of the total number of hours very cautious objectives while others have very saved annually by vehicles using the road. The ambitious one s . When such activities are comp a re d , time saved can be globally related to the public the conclusions are distorted and the synthesis at investment so that comparisons can be made with policy level is impossible. road improvement activities in other similar reg i on s . Finally, it has become clear that policy evaluations The indicator mentioned above is specific to a must address questions that are of policy interest. particular activity (improving roads), just as the Such questions are not related to the outputs of ap p ro a c h adopted is that of an eva l u a t i o n specifical ly specific activities, or to their immediate outcomes. for the road transport sector.Specific evaluations Evaluative questions of policy interest are closely have been performed for all sorts of activities: related to the ultimate policy goals, i.e. macro- tra n s p o r t infras t ru c t u re s ,t r aining of the unemployed , economic or macro-social development. These reh a b i l i t a t i o n of run - d o wn neighborh o o d s ,s u p p o rt questions are difficult ones and they require long to innovation networks, etc. Volume 2 in the and careful evaluation studies before being credibly MEANS Colle c t i o n presents numerous examples of a n s w e re d . It has proven impossible to deal ac t i v i t i e s , with a descrip t i o n of their results and specific with such difficult questions for every activity on impacts and suggestions for appropriate indicators. every site.

E V A L UATION TOOLS AND METHODOLOGIES: Case Studies 7 3 Consequently, a good practice is to distinguish two consultancy services provided, and the categories le vels of eva l u a t i on . Ev a l u a t i o ns of specific activities of businesses targeted. Individual interviews were address questions related to immediate outcomes then car r ied out with 20 operat o r s . These intervi ew s and apply judgement criteria that make sense for we r e used to describe in greater detail the strat e g i e s managers. These evaluations apply separately to developed by the operators and to establish a al l sites and activities. At the policy level , eva l u a t i on s typology of operators. Six operators were then address questions related to ultimate outcomes and selected for deeper case studies. A sample of 300 ap p l y judgement crit e r ia that make sense for policy- beneficiary firms was selected among the “clients” makers. Policy evaluation cannot reasonably apply of these six operators. Another sample of 200 to every activity and every site. On the contrary, n on - b e n e f i c i a ry firms was also con s t i t u t e d , i n they typically involve case studies and field surveys keeping with the matching pairs technique. in a limited number of sites. A second question n a i re was addressed to the beneficiary firms and to those in the matching group. A statistical comparison between the firms E V A L UATE WITHOUT MONITORING in the two groups provided quantitative indicat i on s DATA WHEN NECESSARY on the effects of the programme, in terms of turnover, productivity and investments. In order The task of evaluators is considerably easier when to avoid “biased” statements, the questionnaires they can rely upon a set of sound monitoring data c om p rised numerous and part i a lly re d u n d a n t related to inputs and outputs. In the case of questions (triangulated questionnaire).The survey the European Cohesion Policy, much effort has incorporated a search for deadweight effects. been made to develop evaluation together with monitoring. After ten years, it must be said that This example shows that evaluation can (and the state of practice is far from perfect. In most sometimes should) be implemented in the absence sites and for a majority of activities, it can be of moni t o ri n g . It shows that public authorities should considered that outputs are properly monitored, str engthen their cap a c i t y to undertake both exerc i s e s , both quantitatively and qualitatively. but that poor monitoring should not be accepted as an excuse for cancelling policy evaluation. In a near future, this may also be the case for job creation, an impact that is of strategic interest at po l i c y level . Never theless most impacts are unlikely to be subject to systematic monitoring within the next decade. Unfortunately, these impacts give rise MERGE EX A N T E , ON-GOING AND to many evaluative questions of policy interest. EX POST EVALUA T I O N Here comes the problem of evaluating without monitoring data. Examples from the European In the evaluation literature, contradictory opinions practice show this is not impossible. are expressed on the timing of evaluation. Some authors consider that eva l u a t i o n is onl y a ret ro s p e c t i v e For instance, this achi e ved in the case of the Brit i s h exercise while others also attribute a prospective TEC program (TEC for Training and Enterprise dimension to it. In the context of European Council). This success story of useful evaluation Cohesion Policy, evaluation is clearly defined as an took place in the instance of an absolute lack of monitoring data. The TEC were financed by the activity, which takes place before, during and after British Department of Trade and Industry. They the implementation period. These three phases offered SMEs management consultancy, training, are related to the policy cycle and consist of ex ante and information services. The impact of this eva l u a t i on ,i n t e r im eva l u a t i o n and ex post eva l u a t i on . programme could not be expressed in jobs created, since it was aimed at enhancing pro d u c t i v i ty Given the uninterrupted sequence of policy cycles, which, in the short term, led to the destruction of the three exer cises have been increa s i n g l y overl a p p i n g jobs. It was therefore necessary to evaluate the as the European evaluation system developed. programme in terms of other criteria such as Evaluations performed at different stages and for productivity, profitability or investments induced. different cycles tend to collide at the same time. None of these impacts had been monitored. The sequence of three eva l u a t i o n phases in successive cycles creates overlaps that have to be organized as An initial questionn a i r e was sent to the 75 operat o r s efficiently as possible to avoid any duplication in the Councils. This questionnaire concerned the of work.

7 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing The rel a t i v e cont i n u i t y of activities from one perio d an alternative technique (the Logical Diagram to the next makes it possible to use conclusions of Expected Impacts) has been successfully from the recent past to judge the relevance of the used for describing and quickly screening a n ew activities pro p o s e d . In the case of the complex policy. European Cohesion Policy, ex ante evaluations that prepare the 2000-2006 policy cycle had to be (2) Systematic evaluation on every site is not undertaken by regional authorities in 1998-1999. workable.A model of policy evaluation should By this time, the interim evaluations of the period better invo l ve all rel e vant partners in a steerin g 1994-1999 were just finished. Some of the ex post group, require them to select key issues for evaluation for the period 1989-1993 took place at in-depth ev a l u a t i on and to ask ev a l u a t i ve the same time. questions that clearly are of policy interest.

In so far as evaluation must draw conclusions from (3) Systematic rel ying upon moni t o r ing data is not the experience of preceding activities to improve alw a ys worka b l e . Pol i c y evaluators can and mus t f u t u re activities, an interesting solution is to wo r k without moni t o r ing data when necessary. establish an evaluation plan that covers several years and that more or less merges the three pha s e s . (4) Systematic following of the policy cycle is not The idea is to identify the diffe rent possible wo rk a b l e . Pol i c y eva l u a t i o n should be organized evaluations and to establish their date and content through evaluation work-plans that merge ex in relation to the agenda of policy-makers and ante, on-going and ex post exercises. to the needs of other evaluation “customers”at various levels. My personal opinion is that these lessons,learnt in the context of a large European policy, are of general value for other large policies in other TRANSFERABLE LESSONS con t i n e n t s . Ar e the corres p o nding rec om m e n d a t i on s transferable to other policy settings? This is a FROM EUROPEAN PRACTICE matter for discussion.

Les s o ns and rec om m e n d a t i o ns that emerge from the European practice can be summarized as follows: R E F E R E N C E S European Commission (1999) Ev aluating socio-economic programmes, (1) Systematic ev a l u a t i on along the logf ra m e MEANS Collection,Luxembourg: Office for Official Publications technique has proven not to be workable, but of the European Communities.

E V A L UATION TOOLS AND METHODOLOGIES: Case Studies 7 5 EVALUATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 12 Rustam Lalkaka, President,Business & Technology Development Strategies

1 . BA C K G R O U N D

Science is intended to understand the world while technology is to help change it – hopefully, for the better. From a development standpoint, distinctions between high and low technology are irrelevant. What is important is that the product or service is useful, it is ‘friendly’ to society and the environment, and it contributes to the overall priorities of economic growth with equity, social justice, basic needs and employment. Further, knowledge at the root of science may be value free, but technology is conditioned by context and culture.

A SHARED HERITAGE

Our knowledge of inner and outer space is changing our understanding of the universe and our place in it. At the same time, we are witnessing the decline of the natural environment and the bio-diversity on which we depend for life itself. While the cold war may have ended, weapons proliferate and struggles continue by the world’s poor and its ethnic minorities for a better existence. What then does the coming millennium hold for human civilization? And what can the industrializing countries in particular now do to enhance their national innovation systems, in order to apply technology to the task of accelerating economic growth with fairness and stability?

The historical perspective should be kept in view: For about 2,000 years, from say 300 BC to 1770 AD, countries which are today called ‘developing’ were at the forefront of applied science and many significant innovations moved from east to west. For the next 200 years, following the industrial revolution, countries today called ‘developed’ began to pull ahead, technologically and thereby economically and militarily. And over the last 50 years, the pace and pattern of technical change has altered sharply and the industrial countries have out-paced the rest of the world. Nevertheless, a dozen Newly Industrializing Countries now have the technical infrastructure for major innovation; and for all nations the advanced technologies – adapted, applied,and absorbed – can help improve their lives.

This paper looks at the main dimensions of the innovation system and indicates some of the special considerations involved in making evaluations of S & T programs. Two case studies illustrate appropriate techniques. Such evaluation has to become integral to the a country’s governance, as the taxpayer has the right to ensure that the large sums of money for building the S & T establishment are being spent wisely and with maximum impact.

E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 7 7 2 . MAIN COMPONENTS OF 4. Tech n i c al support services req u i r ed to enhance quality, raise productivity, lower production NATIONAL INNOVATION SYSTEM costs, and develop an integrated marketing- re s e a rch - p ro d u c t i on sys t e m , e s p e c i a lly the For the purpose of reviewing national Monitoring su p p o r t services for strengthening techn o l o gy - and Evaluation capacity, the innovation system can based small enterprises. be considered in terms of six sub-systems: 5. International cooperation through strategic 1. S & T policy com p r ising the setting of prio ri t i e s and allocation of resources for basic research, alliances,design and production sharing in the advanced science-based technologies, com- supply chain, linkages to donor agencies and petitiveness in all economic sectors, and for de vel o p ment banks, and techn i c al and financial other national priority concerns. co ll a b o ra t i o n while safeg u a r ding the intelle c t u a l property of the inventor. 2. In n ov a t ion strat e g y, both short and long - t e rm , based on the nation’s competitive advantage, Un d e r lying the above sub-systems is the overa r chi n g the management of res e a r ch and its comm e rc i a l pr oblem of mobilizing financial res o u r ces for S & T utilization, the roles of publicly-funded and (which is much more than R & D), particularly in corporate research, targeting the special needs poor countries where programs for the survival of of large rural and disadvantaged communities, its people must itself be the highest priority. the cost-effective acquisition of technologies, and the ‘deepening’ of technical competencies Ea c h country has to design its own national innova t i o n for adaptation , a p p l i ca t i on , d i s s e m i n a t i on system based on its endowments and constraints, and, progressively, local innovation. its culture, climate and historical legacy. Table 1 3. Technical human resources including science indicates some key knowledge indicators in select- education starting in primary school and its ed countries.The expenditures on R & D are a proxy progression through technical universities and of the commitment to innova t i o n but must be inter- continuous learning, vocational training, new preted carefully as they do not indicate the pro- disciplines such as the management of ductivity, the prioritization and inter-sectoral allo- te ch n o l o g y and techn o l o g i c al entrep re n e u r s h i p cations of research efforts.More relevant today are de vel o pm e n t . The computing and commu n i ca t i o n the proportions of high-tech products in total technologies have a crucial role in developing manufacturing exports and the electronic infra- new ways to eradicate illiteracy. structure in a nation.

TABLE 1: KNOWLEDGE-RELATED INDICATORS – SELECTED COUNTRIES

GNP Military TVs PCs Internet R&D R&D Hi g h - t e c h . Royalty License Patent Patent PPP Expend. per per Hosts Per s o n n e l Expend. Exports Receipts Payments Appls Appls filed US$ % GDP 1,000 1,000 per 1,000 per 1 mil. % GNP % mfg US $ mil. US $ mill. filed by by Non- 1997 1995 1997 1997 1997 1985-95 1985-95 exports 1997 1997 Re s i d e n t s Residents 1997 1996 1996 Argentina 10,100 1.7 289 39.2 15.92 671 0.4 15 6 240 n/a n/a Brazil 7,430 1.7 316 26.3 9.88 168 0.6 18 32 539 2,655 29,451 China 3,070 2.3 270 6.0 0.16 350 0.5 21 55 543 11,698 41,106 Hong Kong 24,350 n/a 412 230.8 108.20 98 0.3 29 n/a n/a 41 2,059 Egypt 3,080 5.7 127 7.3 0.33 458 0.5 7 54 365 504 706 India 1,660 2.4 69 2.1 0.11 149 0.8 11 12 150 1,660 6,632 Jordan 3,350 7.7 43 8.7 0.79 106 0.3 26 n/a n/a n/a n/a Korea 13,430 3.4 341 150.7 37.66 2,636 2.8 39 252 2,413 68,446 45,548 Malaysia 7,730 3.0 166 46.1 18.38 87 0.4 67 0 0 n/a n/a Mexico 8,110 1.0 251 37.3 8.75 213 0.4 33 130 501 389 30,305 Poland 6,510 2.3 413 74.4 45.34 1,299 0.7 12 27 175 2,414 24,902 Singapore 29,230 4.7 354 399.5 187.98 2,728 1.1 71 n/a n/a 215 38,403 S. Africa 15,690 2.2 125 41.6 34.02 938 0.7 n/a 73 258 n/a n/a Turkey 6,470 4.0 286 20.7 4.30 261 0.6 9 n/a n/a 367 19,668 Germany 21,170 n/a 570 255.5 140.58 2,843 2.4 26 3168 4694 56,757 98,338 Japan 24,400 1.0 708 202.4 107.05 6,309 2.9 38 7303 9620 340,861 60,390 USA 29,080 3.8 847 406.7 975.94 3,732 2.5 44 33676 9411 111,883 111,536

7 8 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Some major con s i d e ra t i ons in formulating an The evaluations of giant national undertakings effective evaluation for the above sub-systems are such as the Three Gorges Dam in China require outlined in Annex-1.These then form the basis for inter-disciplinary skills due to their enormous developing the needed evaluation capabilities. environmental, social and financial implications for the future. 1 . S & T POLICY 2 . I N N OVATION STRATEGIES At the national level, an appropriate strategy for developing countries starting the process of tech- For the longer term, the nation has to move from nology transformation could be a cascade of first, imitation to innovation, more so as the advanced importing equipment and know-how, acquiring proprietary technologies become difficult to buy. higher level design and operating experience, to This req u i re s : Ana l yses of the strategic implicat i on s be adapted, improved and applied, and then on of global change and competitive advantage for to indigenous innovation and exports, both of n i ches in re g i onal and intern a t i onal mark e t s ; technology-based goods and know-how itself. Realistic assessments of current capabilities and resources, to identify the gaps and take action In core technologies a nation (and a company) can on technical support services, human resource make good progress (and money) by using other development, special financing instruments; p e o p l e s ’ in n ov a t i o ns and purchasing embodied research in capital equipment, with a balance Formulation of new legislation providing fiscal between payments for intellectual property rights incentives for entrepreneurial activity and small and investments in adaptation, improvement and business prom o t i on , for protecting intell e c t u a l reverse engineering. Consider the case of Japan property and the environment; Identification of wh i c h increased the number of techn i c al colla b o ra t i on s s t ra t e g i c , g e n e ric technologies suited to loca l and fees paid by eight-fold in the period 1960–70; endowments, and ‘prospecting’ for concepts and concurrently in this period, it spent nine times re s e a rch results capable of rapid patenting, mo r e on assimilating and upgrading this techn o l o g y, prototyping and market entry; and Promotion of to build manufacturing cap a b i l i t y for the subsequent the culture of informal net-working, information export of products and technology. sharing and risk-taking. It may also involve a university linkage, some engineering, production However, with the galloping pace of technical change all countries need to move more rapidly and marketing ca p a b i l i t i e s , and almost alw ays from investment-led growth to innovation-driven require a committed entrepreneur or champion to development. This calls for investment in science mobilize resources, start the venture, survive, and and technology – one’s own or spill-offs from oth- (with luck!) thrive. ers, the efficient import of know-how and equipment, and foreign investment that brings these in. Research management The building of a knowledge society requires the continuous commitment of civil society and the Perhaps the greatest invention of this century is deployment of significant resources in scientific, the inve n t i on process at c o rp o rate industrial educational and social fields. laboratories, starting with Edison’s in New Jersey. These big establishments of GM, Xerox and IBM Te c h n o l o gy for national priority concerns were downsized, but are now being rejuvenated to again become the prime movers of innovation. The majorit y of the world ’s population still depends Even so, the R & D budgets of a single large on the traditional knowledge of its flora and fauna multinational corporation are larger than the total for food and medicine.This knowledge — and the national budgets of say China or Korea or India. related bio-diversity — have to be preserved and In d e e d , the res e a r ch expenditures of the devel o p i n g supplemented by modern technology. Rural folk countries are only around 5 percent of the whole have a shrewd sense of what helps or hurts them, world. In this situation, these countries have to and gover nments as well as interna t i o nal devel o pm e n t in c r ease budgets where possible, enter into res e a r ch agencies have not built upon their knowledge, with collaborations where appropriate, and importantly, poor results. In many situations, the traditional improve the productivity of their own research. can be upgraded by blending with the advanced, At the same time, technology for national security, For gove rnments in developing countri e s , t h e defense preparedness and national harmony has to responsibility is to restructure the publicly funded be the foremost priority. res e a r ch and techn o l o g y organ i za ti o n s , im p r ovi n g

E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 7 9 the facilities and the incentives for researchers, and TORCH program for advanced technologies. making them more accountable in their perform- Now it is investing Rmb 4.8 billion in the 1998- ance and more responsive to the needs of business. 2000 period on the ‘K n owledge Innov a t i on Institutes in industrializing countries operating for Proj e c t’ , with the objective of raising its S & T syst e m a decade or more should now recover at say, three- to rank among the world’s top ten before 2010. At q u a rters of their annual operating expenses the same time, it has muc h work ahead in revi t a l i z i n g through contract research and services. Many now its Vil lage and Township Enterpr ises and re- s t ru c t u ri n g claim to do this, some using creative accounting. its State Owned Enterprises.

In most industrializing countries, the bulk of the fo r mal R & D is done in publicly-funded laborat o ri e s , 3 . BUILDING T E C H N I CAL HUMAN RESOURCES muc h less in the techn i c al universities and corpo ra t i on s . Neve rt h e l e s s , the extent of improv i s a t i on and In countries at the leading edge such as Korea 20 manufacturing improvements taking place on the percent of tertiary enrollments are in sciences and shop floor are impressive. engineering while the Scandinavian and former soviet Union countries spend up to 8 percent of A case example is presented in Annex-2 of the their GNP on education. Countries with poor B e n ch m a rking of Research and Te ch n o l o gy technological capabilities enroll and spend less Organizations,(RTOs) towards improving their than half as much. In some cases, there is the p e rf o rm a n c e . With financial generous support m i s a ll o ca t i on of re s o u rces resulting in high from the Danish and Canadian Governments,The u n e m p l oyment among engineering gra d u a t e s World Technological and Research Organization along side high illiteracy rates. organized the ‘Best Practices Project’ to evaluate and draw lessons from 60 RTOs, 20 each in the What has changed in this decade is the function of Ame ri ca s ,E u ro p e - A f ri ca , and Asia.The methodology the university: It is no longer teaching alone. It and results can be of interest to all in the pursuit of must be restructured to take responsibility for an improved research performance. active role in the more complex field of economic de vel o pm e n t ,c o ver ing a portfolio of applied and basic Corporate innovation res e a r ch ,c onsulting and commu n i t y servi c e s ,s p e c i a l i ze d tra i n i n g ,and distance learning and tech-based ven t u r e Everywhere, managements have to be receptive to formation. While it moves towards becoming an ne w ideas, to listen. Co rp o ra t i o ns in countries such ‘en t rep re n e u r ial univer s i t y’, the corpo ra t i o n moves as Japan and US are adopting unconventional towards becoming a ‘learning enterprise’. innovative ways to stimulate creativity in their pe r s on n e l . Furt h e r , as res e a r chers better understand Within the re s e a rch unive r s i t i e s , the Media the physiological, psychological and biochemical Lab o ra t o r y at MIT,Robotics Institute at Carne g i e - roots of the creative process, the left-right-front- Mellon,and the electrical engineering department back functions of the brain, they now expect to be at Stanford have been the creators of continuous able to instill more creativity in more persons in n ov a t i on s . Some such as artificial intelli g e n c e , ex p e r t systems, fuzzy logic and biochemical computers The traditional wisdom is that the product must will become the industries of the 21st century. be based on the needs of the market, but where there was no existing market — as for the now Also being changed the world over is the educat i o n ubiquitous ‘Post-It’, paper clip, stapler and photo- curricula. New courses being introduced are on the copier — it takes extraordinary persistence to management of tech n o l o gy, w h i ch prov i d e s s u c c e e d . In promoting innov a t i on , the large kn o wledge on key issues at the interfaces of science, en t e rp r ises are mimicking the small, by out sourci n g eng i n e e ri n g , business and civil society.Young and old supplies and services and breaking out into small ar e being exposed to en t rep r eneurship devel o p m e n t , intra-preneurial groups, while the small are acting which seeks to transform the nascent entrepreneur like the big through alliances and virtual cons o rt i u m s . into a successful enterpri s e - ow n e r . And ever y person The flexibility and creativity of an entrepreneurial needs to be proficient in the English language and techno-venture may lead to more break-through co m p u ti n g ,be c ause without these you cannot work, innovations than can be generated by larger sized play, or participate in the expansion of Internet firms in many sectors. and e-commerce.

China has had success in promoting innovation Interestingly, high school students in the US fare through its SPARK program for rural enterprises poorly in mathematics and sciences,but they seem

8 0 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing to acquire the questioning attitudes and comp u t i n g effectiveness of China’s large public investment in skills, which produce good results at the university facilities such as technology parks and business l eve l . The re s e a rch universities have becom e incubators would point to interesting lessons for seedbeds for innovations, with the University of future developments in China as well as in other California system earning $ 61 million through developing countries. royalties and its 528 patents in 1997, while Sta n f o r d and M.I.T. each created 15 start-up companies. 5 . INTERNATIONAL COOPERATION

4 .T E C H N I CAL SUPPORT SERV I C E S In a world shrinking rapidly with the advent of instantaneous, low cost computing and communi- These cover the techn i c al infras t ru c t u r e of institutions cat i on s , the S & T system — and those who operat e and capability to support the innovation system. and evaluate it — have to think and act globally. They include productivity centers, metrology and total quality management, indigenous consultancy A major effort to promote such international organizations and services to strengthen small cooperation and strengthen endogenous technical en t e rp ri s e s , on which large prop o rt i o ns of employme n t , capacities was the UN Fund for Science & Tech n o l o g y incomes and exports depend. for Deve l o pm e n t . Independent ev a l u a t i ons of UN F S T D - s u p p o r ted techn i c al cooperat i o n prog ra m s Business development services in China (eg the National Remote Sensing Center and the Beijing Center for Computer Software While there is consensus that small enterprises and Research) showed that with catalytic funding will be the prime creators of employment and significant capacities were built.The main lessons: gr owth in the future, these ven t u r es need special help. the success depended on the commitment and Various financial and non-financial mechanisms discipline of the Chinese institutuions and on are converging in a synergistic system. There is a vigorous monitoring of progress. cu r r ent debate on the outrea ch , impact and financial sustainability of alternative means to strengthen I n t e rn a t i onal coopera t i on re q u i res a re n ew e d , new ventures. Such services have to be initially continuing dialogue between the poor and the su b s i d i ze d ,w h i c h the developing country govern m e n t s ri ch , without arro g a n c e, with the purpose of can ill afford and donors usually resist (although finding joint solutions and sharing the benefits of s u ch support re c e i ves major state funding in human knowledge fairly. OECD countries).

The ability of a BDS (such as an incubator) to 6 . E V A L UATION CA PA C I T Y replace the resources it consumes and become financially sustainable can be shown by an analysis In developing national capacity for science and of the flow of funds in and out of the system over technology related programs, their special charac- at least 5 years. Sustainability implies the ability to teristics have to be noted. c ontinue ach i eving positive outcomes and the durability of the benefits achieved. Effectiveness Whether it is in the speed of computing and can be expressed in terms of all the benefits communications or the unraveling of the human derived at the whole system in relation to the use genome, technologies are advancing at exponential of all resources and the overall satisfaction of those rates. While the 19th century ushered in the involved. Outreach depends on the replicability of telephone, electricity and automobile, this century the embodied concept and the means of reaching has brought us near-costless and ubiquitous larger numbers of enterprises. The metrics and information flows.The evaluators of such systems criteria of assessing BDS performance require have to be abreast of current practices and coming com m o n understandings by donors and govern m e n t s trends. They have to be sensitive to the political, as well as by the businesses they serve. cultural, ethical and environmental consequences, the possible impacts and outcomes of the prog ra m s Starting in the 1980s China has built impressive being evaluated. n e tw o rks of tech parks and incubators. Their performance has now to be assessed and enhanced. These cal l for inter-disciplinary teams of profe s s i on a l s An n e x 3 pr esents a case example on Ev a l u a t ing the with objectivity, impartiality and utmost honesty. Perf o r mance of Tech n o l o g y Business Incubators As S & T prog r ams typi ca l ly have long time horiz ons in Brazil. A similar approach to evaluating the and high investments,the judgements made in the

E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 8 1 evaluation process also have far-reaching conse- a re more complex in science and tech n o l o gy quences and potentials for high costs as well p rojects as the consequences cross national as benefits. boundaries and there is seldom a consensus on the int e rp re t a t i o n , whether it relates to global warmi n g or biologically-engineered produce. The donors have the additional problems of being responsible E V A L UATION OF WORLD BANK PROJECTS to their own taxpayers and responsive to the real needs of aid beneficiaries. The World Bank has been recently involved in loans for industrial tech n o l o gy deve l o pm e n t R E F E R E N C E S (ITD) projects in China, India, Brazil, Korea and Golden, W. T. Ed. Science and Technology Advice to the President, Turkey. As these are significant loans they justify Congress and Judiciary, Pergamon Press,1988 s e rious efforts on loan appra i s a l s , m on i t o ri n g Goldman,M. and Ergas,H., Technological Institutions and policies: p ro g re s s , evaluating results and foll ow - u p. In Their role in developing technological capability in industry, turn,the national and Bank M and E capacity has World Bank Technical Paper No. 383,19997. to be strengthened. Lalkaka, R. Applying technology for enhancing competitiveness, in Human Resource Needs for Change in R & D Institutes, WAIT- Ev a l u a t i o n lessons on policy lending for ITD proj e c t s RO-NISTADS,New Delhi,1996. in d i c ate the cha ra c t e r istics of successful projects as: La l k a k a , R. and Wu Ming Yu, Managing Science Policy and Tec h n o l o g y ■ Competitive environment and demand-led; Acquisition: Strategies for China and a Changing Wor l d , Tyc o o l y , Dublin and UN Fund for Science & Tec h n o l o g y, Ne w Yo rk ,1 9 8 4 . ■ Private sector participation and use of strong Lalkaka, Manual on Planning and Operation of Technology Business specialized institutions and financial interme- Incubators, UNESCO,Paris,1999,under publication. diaries; Need for integrated approaches; ■ Mackay, K., Evaluation capacity development, World Bank, ECD Unhindered access to international sources of Working Paper No. 6,1997. t e ch n o l o gy and pro t e c t i on of intell e c t u a l Nelson, R. R. Ed. National Innovation Systems, Oxford University property; Priority of financing for tech-based press,New York,1993. small and medium enterprises as well as for WAITRO, Best Practices for the Management of Research and non R, D & E support institutions; Technology Organizations, Special Report,1999. ■ Further, the Bank should support a broad World Bank, Knowledge for Development, Oxford University Press, range of lower-income developing countries, 1998/99. not only the advanced ones. World Bank, World Development Indicators, Washington DC,1999. UNDP. Human Development Report, 1999, Oxford University Press, Measuring performance requires agreement on New York. methods and data collection among those being UN D P - S I DA , Measuring and managing results: lessons for development measured and those wanting the results.The tasks cooperation, 1997.

8 2 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing ANNEX 1 of the research portfolios to deal with the Issues in evaluations special needs of small and large enterprises? of innovation sub-systems 4. What are the incentive systems to promote research productivity and increased research activity in private and state laboratories? CHECKLIST FOR SUB-SYSTEM-1: S & T POLICY 5. What needs to be done to introduce a program of bench-marking the performance 1. What promo t i o nal measures are being taken to of research institutes? see Annex 2. build a national consensus on the role of science- 6. Ho w does the policy and process of interna t i on a l based devel o p ment in strengthening the econom y t e ch n o l o g i cal and financial coll a b o ra t i on s and improving the lives of its people? st i m ulate (or res t r ict) the acquisition of selected 2. What are the country’s short, intermediate technologies? and long term goals to which the innovation 7. If a specific res e a r ch prog r am is under eva l u a t i on , system must be organized to contribute? how do the outputs and outcomes compare to 3. How can the processes of formulating the the original design? na t i o nal S & T strat e g y be made more effec t i v e 8. How will the research results be scaled-up and t h rough better inv o lvement of all stake- taken to market? Wil l the benefits be rep l i ca b l e holders in rural and urban communities, civil and sustainable? so c i e t y and defence establishment, pa rt i c u l a r ly 9. What are the mechanisms at the public, women, youth, ethnic minorities and other university and private research institutes for disadvantaged groups? assisting the commercialization of research? 4. How are the legislative bodies that formulate the policy instruments and regulations being CHECKLIST FOR SUB-SYSTEM-3: prepared for the content and implications of T E C H N I CAL HUMAN RESOURCES their actions? 5. What are the present arrangements for provi d i n g advice on S & T issues to the executive and 1. What is the Gove rnment pri o ri ty and legislative authorities in government? support for science and technology related 6. What methodologies are appropriate to make education and training at all levels,in schools, candid evaluations of the effectiveness of a university, continuous and distance learning, particular type of intervention? vocational training? 7. What are the experience and qualifications of 2. H ow are the new com p u t i n g, C D - ROM pe r s o ns on inter-disciplinary teams for specific and Internet technologies being applied in eva l u a t i o n tasks and how can their comp e t e n c i e s p re p a ring students, f o rmulating tra i n i n g be strengthened? materials and methods? 8. How, by whom and when will the basic 3. What are special measures to improve prof i c i e n c y information and data needed for evaluation in English language and computer literacy at purposes be collected and analyzed? all age levels, including senior citizens? 9. What measures are available for dealing with 4. How is tertiary education being prepared for co mplaints and disagreements with the findings the new millennium? and rec om m e n d a t i o ns of the eva l u a t i o n teams? 5. What programs are underway to create the ‘entrepreneurial university’ and the ‘learning enterprise’? CHECKLIST FOR SUB-SYSTEM-2: INNOVATION STRATEGY 6. What special prog r ams and methods are being used to develop new curricula specific to local 1. What is the overall allocation for research and needs and cond i t i on s , su c h as ‘Management of development in the national budget, and how te ch n o l o g y’ and Entrep r eneurship devel o pm e n t’ ? is this distributed among the industry, social 7. What are the special programs targeted at s e rv i c e s ,a g ri c u l t u re, h e a l t h ,e d u ca t i on ,d e fe n c e education of women, ethnic minorities and and other sectors? disadvantaged groups? 2. What are to be the roles of basic and 8. What structural changes are being made in applied re s e a rch , scientific bre a k - t h ro u g h s the res e a r ch establishment, se r vices and priv a t e and incremantal innovation,public,corporate, sectors to motivate techn i ca l ly competent persons university research? to work in the country rather than migrate? 3. What is the organization system in public What is being done to attract expatri a t e laboratories and the means for prioritization nationals to return to their country of origin?

E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 8 3 CHECKLIST FOR SUB-SYSTEM-4: 8. Ho w can the national metrol o g y, st a n d a rd s T E C H N I CAL SUPPORT SERV I C E S an d qu a l i t y system be strengthened for in t e rn a t i o nal competitiveness? 1. What are the main objectives and outputs 9. Ho w are the national capabilities being of the service being evaluated? Importa n t l y, de vel o p e d for preserving the environment? what are the long e r - t e r m outcomes expected? 2. What provi s i o ns have been made in the or iginal design and subsequent implemen- CHECKLIST FOR SUB-SYSTEM 5: ta t i o n for the colle c t i o n of the data needed INTERNATIONAL COOPERATION for measuring effective performance, see Annex 3. 1. As a matter of state policy, is the S & T 3. Have the financial and technical resources ac t i v i t y such as can benefit significantly by been provided to ach i eve the desire d linkages to other countries or donor agencies? results? Ample finance can be a disincentive 2. What will be the main purposes of interna - while scarce resources can mean that the ti o nal cooperation: developing local skills, p roject managers must con t i n u o u s ly ac q u i r ing special equipment? Bench- m a rk i n g and learni n g from others? mo b i l i z e additional mone y to the neglect of 3. In pursuit of better international relations, their substantive tasks. would the project experience and results 4. How do the operations of the TSS fit in to obtained be shared with other interna- the larger strategy for the development of ti o nal agencies, other developing countrie s ? the conc e r ned sector? What are the political 4. What are the benefits and costs related or other hidden agendas which the sponso r s / to forming an international association or financiers of this service seek and why? other mechanism for collaborative efforts 5. What are the levels of competence and in the future? tra i n i n g of the managers? What are the level s 5. What mechanisms are in place to attract of autono my and accountability? t e ch n i cal and financial coll a b o ra t i ons 6. If the service is in the public sector in selected S & T sectors? To better adapt, and re c e i ves public subsidy, does this ap p l y and improve the techn o l o g y acquired ? affect private sector or non-governmental 6. Ho w can the S & T system be enabled to providers of the same service? sh a r e its experiences with other countrie s , 7. Ho w are national engineering and man- d eve l o p e d and developing? Is there a ag e ment con s u l t a n cy capabilities being national technical assistance program to promoted? assist other countries?

8 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Annex 2 whereby comparisons are made on productivity, quality and good practices between a chosen Case Example: Benchmarking comparable set of RTOs. the effectiveness of Research & Be n ch m a r king req u i re s : an eva l u a ti o n of the Training Organizations (RTOs) res e a r ch process, the systematic collection of i n f o rm a t i on , t h e c o n tinuous measurement o f Scientific res e a r ch and techn o l o g i c al devel o pm e n t performance, and the long-term commitment account for the bulk (up to two-thirds) of to t a l to improve RTO effectiveness. annual S & T expenditures in many countrie s . The productivity of research and the performance of The RTO model can be conceptualized as in research institutes, largely supported by public Figure A2-1 below. funds in developing countries, are of importance to the national economy. As technological change In the WAITRO project, funded by DANIDA/ is moving at exponential rates, these RTOs can Denmark and IDRC/Canada, 60 RTOs were provide major support on training, research and examined (6 in Europe, 10 US/Canada, 20 Asia, consultancy services for small enterprises, which 10 central/south America and 14 in Africa) by an need to upgrade their total operations. international project team consisting of experts from Canada, Denmark and India. A comp re h e n s i v e study has been undertaken by t h e Wo rld Association of Industrial and The following processes were benchmarked: Tech n o l o g i c al Research Organizat i o ns (WAIT RO ) 1. RTO governance (ownership, legal structure, on benchmarking of RTOs, that is, identifying gove rning board’s con s t i t u t i on , s i ze and and measuring those practices which enabled p ow e r s ,m i s s i on and vision ,l evel of autonom y , RTOs to enhance their operative and strategic mandate on sectors or regions). efficiency. Benchmarking is a systematic process 2. Financial management (funding methods,

FIGURE A2-1: THE RTO MODEL

SOCIETAL CONTEXT

GOVERNMENTAL POLICIES RTO Ownership & INFRASTRUCTURE

Managememt

NETWORK: Partners, MARKET: Business colleagues, competitors Clients, beneficiaries, development (L o c a l ,n a t i o n a l ,i n t e rn a t i o n a l ) funders

Services

Human resources; Facilities; Finance

E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 8 5 le vel , dependence on state support, fl e x i b i l i t y, The WAITRO team has developed a consultancy retention of surplus/loss). manual of diagnostic tools for tra n s f o rm a t i on 3. Services (types such as basic/applied, testing, ef f o r ts at RTO s ,i n cluding che c k-lists and proc e d u re s tra i n i n g ,con s u l t i n g ,id e n t i f i ca t i o n of needs,qu a l i ty ) . on strategic planning, change management, self 4. Business development (activities,effectiveness assessment,technology and industrial audits. RTO in attracting business, rewards, promotion, transformation excercises have been conducted in costing, affordability, prioritization). several countries based on the WAITRO findings. 5. O r g a n i za t i onal and project management Their success depends in good measure on the (s t ru c t u r e, team selection, unit res p on s i b i l i t y to explicit administrative and financial support given meet RTO goa l s , management and follow - u p ) to the change process by all the stakeholders. 6. Ca p a b i l i t y building (staff devel o pm e n t , cap a c i ty - building methods,funding of trai n i n g / e q u i pm e n t ) . The process follows the trajectory [visualized in 7. Personnel management (Recruitment process, Figure A2-2]. below promotion, compensation, motivation, staff evaluation, internal communication). FOLLOW-UP A C T I O N S 8. Networking (relations with other technology pr oviders and with industry to better serve them) I m p l e m e n t a t i on of a bench - m a rking pro c e s s 9. Policy and programs (Role in formulating requires the candid analysis of current practices na t i o nal S & T and industrial policies, le vera g i n g and the prog re s s i v e move towa r ds an organizat i on a l government programs). transformation. This is not easy.People resist change, particularly if the change is imposed on As precise inform a t i on was not available to t h e m . Real Change happens on ly when the qu a n t i t a t i ve l y measure perf o rm a n c e s ,b road macro- researchers from the bottom-up and the leadership level indicators had to be used as proxies. These can both rec o g n i z e the need for cha n g e , can jointly in c luded aver age growth of income from cli e n t s , rat i o s implement a con s t ru c t i ve process rather than of grants to total income, expansion of the RTO in assign blame, and the leaders can motivate and services and territory served, and recognition in communicate a vision for the future in which terms of national and international patents and everyone wins. p u b l i ca t i on s . The above indicators were give n weighted ratings, for a total of 100 points. C on c u r re n t ly, the re s e a rch managers need special orientation on new techniques of busi- The main lessons learned regarding good RTO ness management under con d i t i ons of global practices are summarized as follows: ch a n g e . This is best done at special training courses focused on the needs and con d i t i ons ■ Despite differences in culture, there is a com- of RTOs. monality of problems such as poor patronage fr om industry,de c lining support from govern m e n t An RTO benchmarking network calls for sharing and difficulties in commercializing research. of informa t i o n which in turn req u i r es conf i d e n - ti a l i t y and trust among the parties conc e rn e d . A ■ Reliable management information systems are data base of best-practice indicators (with the pre r equisites for implementing good prac t i c e s . se c r et codes rather than actual names of RTOs) can then be ma d e available to those participating in ■ While RTOs from Nor th Am e ri ca ,E u rope and the network. the industrialized Asian countries were more successful ove ra ll ,m a ny effec t i v e practices were The next step would be to develop cooperative found in RTOs from developing countries. projects between small enterprises and the RTOs, ■ The best practice emerged when the RTO wh e r e enhanced res e a r ch perfo r mance can be applied was focused on the needs of their clients. to the real needs of the small industry sector.

FIGURE A2-2: THE RTO PROCESS

DIAGNOSIS PLANNING IMPLEMENTATION By local/external experts Frank diagnosis of practices Identify expertise needed Seminars & diagnostic tools Adaptation of good practices Find funding, compare results Develop report for Board Communication/consensus Develop long-term plans

FEEDBACK

8 6 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Annex 3 to greater business opportunities and promote spin-offs. Good measures of performance of an Case example: Evaluating the performance incubation system are the medium-term benefits a c c ruing to s m a ll businesses, s p on s o r s , l o cal of Te c h n o l o gy Business Incubators (TBIs) com mu n i t y, reg i o n and nation, Figure A3-1.

The overall system assessment requires that don o r s FRAMEWORK FOR PERFORMANCE A S S E S S M E N T make provi s i o n for — and pursue — the colle c ti o n of the needed information by the management The majority of incubation programs worldwide team, on firms in the facility and those leaving as can be cha ra c t e ri z ed as “pu b l i c - p r ivate partn e r s h i p s ” well as other parameters. While some of the in which initial (and often continuing) financial coefficients can be calculated readily, others req u i r e support is received from the state bodies. Many co mplex social benefit-cost estimations . governments consider them as part of the business infrastructure, and the evidence indicates that the annual taxes and other benefits from regional INSTITUTIONAL A N A LYSIS OF BIOMINAS AND PA R Q T E C e c on omic deve l o pment more than offset the capital and operating cost subsidy. Private sector The location, implementation, governance and pa r ticipates when it sees that the program will lead management factors of the Biominas and ParqTec

FIGURE A3-1: ASSESSMENT OF INCUBATOR IMPA C T S , EFFECTIVENESS AND SUSTAINABILITY

Taxes Income

State finance Go v e rn m e n t Co m m u n i t y Bank/venture capital Fed e r a l / s t a t e / c i t y Private sector

Seed Capital Credit; Equity Policy Infrastructure Support Mentoring

SPONSORS/MANAGER Entrepreneur En t e r p r i s e s INCUBATOR

Goods; Services Innovations Faculty; Graduates Change Employment Replication

Un i v e r s i t y Cu l t u r e Na t i o n a l / e x p o r t market Research Institute Skills Entrepreneursh i p Wor k force

University-market links

I. Impact/Outreach II. Effectiveness III. Sustainability

1. Enterprises created 7. Employment per net $ subsidy 13. Revenue surplus (6 years) 2. Survival rate of enterprises 8. Taxes paid per net $ subsidy 14. Services cost recovery 3. Jobs generated (6 years) 9. Income, sales & exports 15. University-business links A. in incubated/affiliated firms 10. Research commercialized 16. Stakeholder satisfaction B. in graduated firms 11. Disadvantaged groups 17. Tenant/graduate satisfac. C. indirect jobs 12. Incubator expansion 18. Changes in culture 4.Enterprises reached 19. Enhancement of skills 5. Replication of “pilot” model 20. Leveraging state policies 6. ‘Extra-curricular’ activities 21. Enhanced self-esteem

E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 8 7 incubators are reviewed below. The main charac- agencies such as FA PEMIG and SEB RA E ; teristics are summarized in Table A3-1.Incubators and a vibrant private sector with BIOBRAS as a need to be sited where there is a strong business world-class insulin producer. Biominas incubator infrastructure with availability of scientific talent, occupies a prime site of 10,000 sq m land adjacent good living conditions, and positive government to the publicly-funded, research laboratory — and community support. On these counts, both CET E C . It is a custom-built building with exce ll e n t the Biominas and ParqTec incubators have good biochemical laboratories. l o ca t i on s . Both plan major tech n o l o gy park s linked to the incubators. ParqTec: The city of Sao Paulo is Brazil’s major business hub. Some 230 km to the north-west is Biominas:Minas Gerais (MG),the second largest the city of Sao Carlos, population 200,000, which industrial state with its capital at Belo Horizonte, has the distinction of having the highest density has traditional strengths in minerals, mechanical of PhDs in science/engineering — one for every and automotive manufacture, and now in bio- 230 inhabitants. I n d u s t ry is con c e n t rated on t e ch n o l o gy. MG has an impre s s i ve tech n i cal consumption goods and mechanical appliances. It infrastructure, with universities such as UFMG, has two public universities and private learning Vi c o s a , Ou ro Preto and Ub e rl a n d i a ; re s e a rch centers, laboratories for cattle and animal protein institutes Rene Rachou, and FUNED; support development, and over 70 enterprises in aero-

TABLE A3-2: PRO FORMA INCOME AND EXPENSE AT PARQTEC AND BIOMINAS INCUBA T O R S

All figures in US$ ,000. Rate R/$ =0.84 PARQTEC BIOMINAS

1996 1997 1998 1997 1998

REVENUES Rental Income 24.4 38.6 39.4 28.9 81.7 Fees from Tenants 2.1 2.1 2.5 3.1 12.9 Utilities Cost Recovery 9.2 18.5 18.5 — — Partnership with SEBRAE 139.4 62.2 239.4 186.0 186.0

TOTAL REVENUE 175.1 121.4 299.8 218.0 280.6

EXPENSES Manager 27.7 38.6 32.9 36.0 54.0 Admin Assist/tech adviser 0.0 0.7 4.5 10.0 5.0 Lab coordinator — — — 7.0 13.4 Receptionist/Secretary 0.8 2.5 4.2 7.3 14.3 Other Professionals 22.8 26.8 33.4 37.9 48.4 Fringe Benefits 5.9 8.4 10.9 — —

SUB-TOTAL STAFF 57.3 77.0 85.9 98.2 135.1

Bldg. Maintenance/Cleaning 31.9 70.6 70.6 8.0 17.5 Utilities/Telephone (net) 18.5 24.4 23.5 30.8 48.4 Travel & Promotion 1.3 0.8 20.2 3.2 3.2 Supplies 0.7 1.3 0.7 4.5 9.3 Audit & legal 9.1 9.1 20.2 3.2 4.9 Insurance 5.0 3.8 4.7 — — Publications 56.3 — — Interest/debt repayment — — — 30.4 39.7

TOTAL EXPENSES 123.7 187.0 282.1 178.3 258.1

OPER. SURPLUS (DEFICIT) 51.4 (65.6) 17.7 39.7 22.5

8 8 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing nautics, informatics, new materials and robotics. The figures in Table A3-3 below should be Fu n d a cao Pa rq Tec de Alta Tecnologia houses considered as preliminary; as it is often difficult to the CINET and SOFTNET incubators together get data from privately held firms on sensitive with related facilities. topics such as sales, payroll and taxes. At ParqTec, the tenant firms have 69 employees while 17 (of FINANCIAL VIABILITY AND OUTREACH the 21) graduated firms have 168, making a total of 237 direct jobs. Estimates of income and expenses As ParqTec has been in operation at its present location since 1990, it has more results to evaluate Estimates of income and expenses for the two in comp a r ed to Biom i n a s ,w h i ch has been operat i n g incubators are shown in Table A3-2 below. Such onl y since July 1997 in its new permanent facilities. data is hard to collect in most countries. Annual The 1997 estimate of public capital and operating revenue from services (in relation to total) is subsidy for Parq T ec and the personal and corpo ra t e considered low. Support through the partnership taxes payable would be approximately as follows, with SEB RAE constitutes more than half of based on the limited data available: revenue, on average. Total jobs (with employment multiplier of 1.5*) 357 Both Foundations plan to achieve greater financial Capital cost subsidy per year s e l f - s u f f i c i e n cy through the deve l o pment of (20-year straight line depreciation) $ 19,150 Te ch n o l o gy Pa rk s . B i ominas has initiated the feasibility analyses and business planning for a Operating subsidy per year (average of last 3 years) $147,000 b i o t e ch related park adjacent to the existing Capital and operational incubator-CETEC complex. subsidy per year $166,150 The São Carlos Science Park is on a 172,000 Total subsidy over 7 years $1,163,050 square meter property that it owns in a prime Subsidy cost per job industrial location. Planning and design for the (excluding jobs in affiliates) $ 3,258 first 3,500 square meter building has been Estimated payroll & corporate taxes by tenants & graduated firms $1,054,320 completed along with a master plan for the balance Return on public investment of the property. The first structure will house the as taxes per year $6.34 per $ subsidy ParqTec headquarters as well as incubator modules * I n d i rect employment multiplier based on ra n ges for similar for 64 additional tenant enterprises. The master economic activities in the U.S. development plan also includes industrial sites for lease to technology-based enterprises as well as two multi-tenant buildings and a conve n t i on TABLE A3-3: JOBS AND T A X E S , 1 9 9 7 ( A P P R O X I M A T E ) , US $ center, designed to establish Cost effectiveness and stakeholder satisfaction ParqTec Biominas Jobs (tenants The evaluation methodology in this case example and graduates)* 237 92 uses a com b i n a t i on of qualitative descri p t i on , quantitative analysis, and stakeholder perceptions. 1997 payroll $1,854,000 $1,030,040 The stakeholders interviewed for this purpose 1997 sales $9,846,990 $2,558,320 were the public and private sponsors as well as the incubated enterprises themselves. 1997 payroll taxes payable $ 463,500 $ 258,510 Businesses incubated and jobs created 1997 corporate taxes payable $ 590,820 $ 153,500 An approach to evaluating the effectiveness of Total taxes $1,054,320 $ 412,010 incubation programs is to look at the number of businesses incubated, the success rate, and the Initial Investment number of jobs created by incubated firms. As in incubator $ 383,000 $1,940,000 noted, both incubators have to aggressively recruit * This includes current tenants plus the one graduate tenant at more tenants and affiliates as well as increase the Biominas and 17 gradu a t ed firms at Parq T ec for whom informa t i o n throughput of graduating businesses. is avai l a bl e .Tax es are estimated at 25% on payroll and 6% on sales.

E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 8 9 The subsidy per job should decline at ParqTec as development, and business training on site. more firms graduate and continue to expand, and as additional incubator space becomes available. For mixed-use incubators, which typically have LESSONS LEARNED much larger areas and less services for tenants, the subsidy cost per job can be much lower. A point to To summariz e,the Parq T ec and Biominas incubators note is that while the investment is made once, the studied have had positive impacts and outcomes jobs continue, and it is useful to think in terms of on their respective city and state economies in “job-years” in the stream of benefits. nurturing entrepreneurs and creating sound enter- 1997 taxes realizable from sales and payroll of pr ises with good survival rat e s . Parq T ec has generat e d ParqTec tenants and graduates could be about six employment with public subsidy of around US$ times the subsidy. 3,258 per job, without including jobs in affiliates. The estimated return in the form of taxes could be Performance evaluation by incubator tenants and graduates about $ 6 per dollar of public subsidy. Bi o m i n a s : Re p re s e n t a t i v es of the present enterpri s e s The linkages to universities and research institutes and the one graduated were asked to evaluate the ha v e resulted in comm e rc i a l i za t i o n of some techn o l o gi e s . ef fe c t i v eness of the incubator as well as the advantages and disadvantages in being tenants. All persons The sponsors and tenants at both incubators ha v e interviewed felt that the program is of value to ex p r essed satisfaction with the results achi e ved , th e m . The major benefits expressed were as follow s : part i c u l a r ly the help in marke t i n g , business planning, help in dealing with bureaucracies resulting in and securing gover nment permi t s . Both are helping faster permits; valuable assistance in marketing their gover nment sponsors in promoting techn o l o g i ca l and faster time to market for new pro d u c t s ; de vel o p ment together with other social aspects such excellent infrastructure and labs; interaction with as reinforcing the cultures of entrepreneurship and other tenants; and legal assistance. university-research-business cooperation.

ParqTec: The incubator tenants and graduates That being said, Biominas and ParqTec have the i n t e rv i ewed expressed satisfaction with their major ch a llenges ahead of enhancing their experiences. The major benefits cited were its: o p e ra t i onal effe c t i veness through innov a t i ve good location for a startup venture, access to activities and crea t i v e financing, in c r eased occupancy facilities such as labs. telephone, internet, and fax and higher fees for quality services, with more s e rv i c e, valuable marketing assistance re c e i ve d , affiliate companies and anchor tenants, in order to legal assistance for incorp o ra t i on and patent reduce dependence on state subsidies.

9 0 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing REVIEW OF DRAFT OF SCIENCE & TECHNOLOGY EVALUATION STANDARD OF CHINA 13 Chen Zhaoying,Vice President,NCSTE, China

1 . BA C K G R O U N D

The science and Technology Evaluation Standards of China (hereinafter referred to as the Standards) is developed in accordance with the following demands:

■ Demands from evaluation performers (the people who “do” evaluation) Evaluators/evaluation organisations need unified standards in order to regulate their evaluation practice, to standardise the planning and regulate their evaluation, to raise level of science and implementation of evaluation, to raise the overall level of science and technology evaluation, to ensure the evaluation quality and to reduce and avoid disputes.

■ Demands from evaluation users (the people who “use” evaluation results) Through the Standards the evaluation consignors and users can better understand science and tech n o l o gy ev a l u a t i on in order to utilise the evaluation results.

■ Demands from government agencies Mrs. Zhu Lilan, Minister of Science and Technology pointed out at the Nat i o nal Tech n i c al Innova t i o n Confe r ence that we should establish a scientific and impartial eva l u a t i o n syst e m , implement it according to law and grad u a l ly standardise our science and technology evaluation.

2 . C H A R A C T E R I S T I C S

A SELF-DISCIPLINARY DOCUMENT

The Standards is a professional self-disciplinary document. The formulation of the Standards comprises an important part of the management of science and technology evaluation. The Evaluation Standards Committee is responsible for the interpretation and revision of the Standards, but has no power to enforce the im p l e m e n t a t i on . In accordance with the Sta n d a rd s , member organisations of science and technology evaluation, such as federations and associations, may conduct self-management and self-cont r ol of the eva l u a t i o n devel o p ment and devel o pm e n t di s c i p l i n a r y actions against the violation of profe s s i o nal conduct and code of ethics.

THE ACCEPTED EVALUATION STANDARDS WITH A BINDING FORCE FOR ENFORCEMENT

According to international practice, evaluation standards can be approved by relevant government agencies as an accepted evaluation standard with a binding force for enforcement.

REVIEW OF DRAFT OF SCIENCE & TECHNOLOGY EVALUATION STANDARD OF CHINA 9 1 The initiative and development of science and by other profe s s i o nal associations for it was legislated t e ch n o l o gy ev a l u a t i on in China have been and approved by the government as an accepted supported by the government. The evaluation on appraisal standard Almost all the customers in the government investment in science and technology United States req u i r e that the appraisal assignments will occupy an important place in the overall should comply with USPAP. The authority of science and technology evaluation for a period of US P AP has been widely ackn o wledge by govern m e n t time in the future. The administrative department agencies and companies in the United Sta t e s . It has of the government should approve the standard or become the basis for formulating an international pa r t of it for use of rel e vant eva l u a t i o n organisations appraisal standard. as a reference. Since the promulgation of USPAP on April 27, 1987, it was revised 13 times. Significant changes THE STANDARDS FOR PROFESSIONAL PRACTICE have been made while the major structure remains.

As a standard of professional performance, the Standard is a condensation of the theories and INTERNATIONAL PROPERTY APPRAISAL STANDARDS (VO L . 9 4 ,9 5 ) practices of science and technology evaluation, reflecting the appra i s e r s ’u n d e r s t a n d i n g, vi e wpo i n t s International Property Appraisal Standards was and experiences of profe s s i o nal eva l u a t i on . Ev a l u a t i o n formulated and promulgated by the International wo r k req u i r es both crea t i v eness and strict comp l i a n c e Evaluation Standard Committee. The framework with the standards , and a comb i n a t i o n of profe s s i on a l is designed in accordance with the objectives of responsibilities and morality. property appraisal activities and the derived values.

S C O P E E V A L UATION STANDARDS OF OECD COUNTRIES

The St a n d a rd is a standard for science and “Evaluation Standards for Science,Technology technology evaluation profession. Any evaluation and Innovation Projects” OECD-1990 (excerpt). ac t i v i t y relating to science and techn o l o g y inclu d i n g pl a n n i n g , ach i e vem e n t s ,p ro j e c t s ,o r g a n i s a t i on s ,p o l i c y and personnel can be covered by the Standard. G OVERNMENT AUDITING STANDARDS (UNITED STATES AUDITING A D M I N I S T R A T I O N ) ■ Financial audit EFFECTIVE T I M E F R A M E ■ Audit relating to financial affairs ■ Performance audit The Standard is used not only to address the prob- ■ Comprehensive audit (contract audit) lems in present evaluation activities, but will also play roles for a long period to come. Therefore, pertinence, continuity should be adhered to during FRAMEWORK OF CHINA’S ASSET APPRAISAL STANDARD SYSTEM the formulation of the Standard. The Chinese Evaluation Association is now making research on and formulating the assets appraisal standards in the light of USPAP. 3 . MAJOR REFERENCE SYSTEMS FOR FORMULATION OF THE STANDARDS 4 . FRAMEWORK AND CONTENTS

UNIFORM STANDARDS FOR PROFESSIONAL A P P R A I S A L The differences of evaluation activities in different PRACTICE (USPAP) OF THE UNITED STATES counties are reflected in their market economy conditions, legal and management systems while USPAP is a professional standards influential not the basic evaluation methods and techniques have only in the United States, North America, but all little differences all over the world (refer to the over the world. It is a standard document with a International Assets Evaluation Standards Vol. se l f - d i s c i p l i n a r y force for the pro fe s s i on formul a t e d 94&95). The Standard takes the internationally by profe s s i o nal appraisal association and rec o g n i s e d adopted open frame struc t u r e, wh i c h can ensure the by apprai s e r s , ap p r aisal organisations and apprai s a l overa l l stability of the Sta n d a r ds and accomm o d a t e clients. The effectiveness and influences of the the changing conditions of evaluation so that the st a n d a r ds surpass the ordi n a r y standards formul a t e d Standards will be improved step by step.

9 2 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing FRAMEWORK AND CONTENTS OF SCIENCE ■ Requirements for evaluation report AND TECHNOLOGY EVALUATION STANDARDS ■ Utilisation of Evaluation results ■ Quality control of evaluation activities P a rt I. The main body ■ Archiving of evaluation activities.

The first level:basic standard The Third level:statements on standards

This part is composed of basic principles, codes of Statements on standards forms as an organic part ethic and terminology of science and technology of the St a n d a rd s ,w h i ch further identify, in t e m p e ra t e evaluation. The basic standard shows the features and explain the evaluation standards and have the of science and technology evaluation and serves as same force as the text of the standards. As the a basic and general standard for science and statements are flexible, they should be stipulated as technology evaluation profession. It is the basis req u i r ed and after being well conc e i ve d . The statements for guiding and regulating all kinds of technical evaluation activities. The basic standard sticks to can be upgraded into part of the standards accordi n g the basic rules of evaluation profession, and at the to the practical situation and maturity level during same time, has its own characteristics of technical the process of the revi s i o n of the standards . The first evaluation. The main contents include: edition of the statements on standards contains ■ Standard of professional conduct. the following: ■ Principle of independence and impartiality, ■ How to maintain independence, objective and principle of avoidance and misleading, confi- impartial? (three elements endanger the inde- dentiality, and fees. pendence). ■ Professional morals (activities violating the ■ How to justify that the appraisers capabilities professional morals). meet the re q u i rements of the ev a l u a t i on ■ Capabilities and qualificat i o n of the apprai s e r s / assignment? eva l u a t i o n organisations , the struc t u r e of apprai s e r s ’ ■ Different requirements and characteristics of com p e t e n c y and cap a b i l i t i e s ,c o ntinuing educat i o n different types of evaluation activities. and training of the appraisers. ■ ■ Major concepts and methods of classification A l t e rn a t i ves (re q u i rements for using the for science and tech n o l o gy ev a l u a t i on in alternatives). foreign countries. ■ Basic concepts and terminology of science and ■ When to use the alterna t i v es reg u l a t i o ns in the technology evaluation. process of evaluation? What are specifically The second level:technical standard required for the evaluation report when the alternative regulations are used? ■ Composed of the major evaluation procedures, and How to make use of comments from the the obligations and rights of the related parties of evaluation results caused by the limits? the evaluation, the technical standards are the basic part of the Evaluation Sta n d a rd s . It describ e s P a rt II. R e f e r e n c e s the main principles, key issues and points to be noted in the planning and implementation of the Guidelines for the implementation of typical eva l u a t i on , and stipulated in detail the req u i re m e n t s science and technology activities and contents of the evaluation report. The main contents of the technical standard include the The following is some references, cases and advis- eva l u a t i o n rep o rt . The main contents of the techn i ca l es which have no binding forces for appraisers in standard include: ■ light of some typical science and technology eval- The roles,obligations and rights of the parties uation activities. involved in the evaluation ■ ■ Ev a l u a t i o n on science and techn o l o g y prog ra m s Evaluation procedures ■ Evaluation on project relevance ■ Preparation of the evaluation ■ Evaluation on project acceptance ■ Evaluation planning ■ Quality control of individual evaluation The preconditions/presupposition of evaluation, evaluation standards, evaluation index, reliability Explanation on the background,concepts, of evaluation, rating system of evaluation methods and processes of the development of the ■ In f o rm a t i o n colle c t i on , data revi e wing and sorti n g . Science and Technology Evaluation Standards ■ Selection of evaluation methods. ■ Multi-index comprehensive evaluation model, Introduction of some representative evaluation case study, comparative study. st a n d a r ds and norms of both domestic and abroa d

REVIEW OF DRAFT OF SCIENCE & TECHNOLOGY EVALUATION STANDARD OF CHINA 9 3 5 . METHODS AND PROCESSES OF PA R T I C I PATION AND T R A N S PARENCY THE DEVELOPMENT OF THE STANDARDS OF THE STANDARDS FORMULATION Importance has been given to the participation of the appraisers and clients in the process of the STRESS ON FUNDAMENTAL W O R K formulation and revision of the Standards. It is planned to set up an eva l u a t i o n standards comm i t t e e The devel o p ment of Ame ri c an Uniform Sta n d a rd s at proper time and promulgate the draft standards for Professional Evaluation Practice (USPAP) over to the public for comments and suggestions so as the past ten-odd year has given us muc h enlightenment. to improve it to a top level of the profession. The First and foremost, USPAP is a condensation and e a rly part i c i p a t i on of people working in this su m m a r y of the theories and practice of eva l u a t i on . profession and other related fields enabled the A complex system engineerin g , US P AP is devel o p e d universality of the standards and laid a good foun- on the basis of sound theories and practice. In the dation for the understanding and implementation process of formulating the Evaluation Standards of the standards after it is formulated and revised. for Science and Tech n o l o g y of China, muc h attention is given to the fundamental work, especially the Publicity should be given to the standards after it res e a r ch on eva l u a t i o n theories and the summing-up is formulated and revised, especially the updated of experiences and cases, in order to lay a sound information. The revised Uniform Standards for basis for the development of the Standards. Profe s s i o nal Evaluation Practice should be published each year. The revised part should be notified in DEVELOP THE STANDARDS ON THE BASIS OF INTERNATIONALLY the preface in order to allow the appraisers and ACCEPTED EVALUATION PRINCIPLES; TAKE IN THE NEW CONCEPTS customers to understand the latest development AND PROGRESS OF THE INTERNATIONAL EVALUATION PROFESSION and make a better use of the standards.

The practice of international evaluation profession REVISE THE STANDARDS REGULARLY AND IMPROVE IT STEP BY STEP shows that the basic evaluation principles in all the co u n t ri e s ,m a i n l y the code of conducts and profe s s i on a l As the evaluation standards need to be revised and morals, are almost the same, although different reprinted with the deepening of the evaluation countries have their own national conditions and practice and the changing of the situation, it different types of evaluation activities. Base on the should be formulated to meet the basic demands at in t e rn a t i on a l ly accepted basic eva l u a t i o n prin c i p l e s , first rather than cover ing ever yth i n g . New cont e n t s the St a n d a rds generalised the experience and should be added to the Standards and revision research results of the evaluation practice in China, should be made step by step. In this way we can and took in the new concepts of and the latest even t u a l ly produce an eva l u a t i o n standards for science pro g r ess of the theoret i c al res e a r ch on interna t i on a l evaluation profession. and tech n o l o gy with Chinese ch a ra c t e ri s t i c s , wh i c h is influential and has a strong binding force s . Ge n e ra l ly speaking, st a n d a r ds are developed by two As evaluation theories are not perfectly established wa ys :i n d u c t i ve and deductive. The inductive way is at present, many basic definitions and principles to form specific rul e s , first according to experie n c e s are not clearly identified. We should not finalise and general practice, and then formulate general the present research results of evaluation. Instead, evaluation standards. From specific to general,the only the matured theories and experiences can be s t a n d a rds formulated this way are descri p t i ve . absorbed in the evaluation standards. Making standards in the way of deduction is to identify the generally accepted concepts first and The Evaluation Standards Committee will make then formulate the specific evaluation standards. ti m e l y revi s i o n according to the situations of prac t i c e From general to specific, standards formed this and the implementation on the basis of reviewing way are normative. Judging from the evaluation the evaluation theories and practices. The revision standards formed this way is normative. Judging will be illustrated in news media and professional from evaluation standards of different countries journals in order to obtain sufficient attention and and regions,the standards would not be consistent understanding of the public. The standards,then, if they are formulated just on the basis of experie n c e s wi l l alwa ys reflect the latest theories of the eva l u a t i o n and general prac t i c e . Based on the conceptual struc t u r e profession and remain proper consistency to serve of evaluation standards of other countries, the as the guidelines of eva l u a t i o n prac t i c e . In summary, Sta n d a r ds are formulated with methods of comb i n i n g China is still at the first stage in formulating the de d u c t i v e and inductive ways. The experiences and evaluation standards for science and technology. le s s o ns learned in the science and techn o l o g y eva l u a t i o n Therefore the standards will not be made perfect practice in China are fully related in the Standards. to cover everything at one step.

9 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PART IV ECD Opportunities for National Initiatives and International Cooperation

E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing GUIDE TO DIAGNOSING ECD NEEDS: Video Speech (Full Presentation is available in booklet form)

14 Keith Mackay, Senior Evaluation Officer, OEDWorld Bank

DEVELOPMENT OF M&E SYSTEMS WITHIN A GOVERNMENT — DIAGNOSING ECD NEEDS + READINESS

■ Danger if we take too simple an approach and assume M&E a “good thing” and its merits should be obvious to all development assistance.

■ World Bank experience (and other dev’t asst agencies) is that most forms of capacity building are difficult to achieve, and require long-term efforts if they are to be sustainable.

■ This also applies to ECD.

■ A key lesson from experience is that to be successful/sustainable we need to understand broader institutional environment.

■ Another key lesson is that there is no single, best model for ECD. • ECD has to be tailored to each country’s circumstances. • One motivation for ECD is that many of the most advanced economies in the world typically have some type of national system for M&E. • Countries such as Australia, Canada,USA, + various European countries for instance. • But when you comp a r e differ ent countrie s , you find that their approa ch e s differ considerably.

■ Increasingly, a national M&E system is not being seen as a stand-alone activity, but as part of sound governance: • pe rf o r mance measurement is seen as counterpa r t to perfo r mance management; • the multiple uses of a national M&E system can provide support to a number of different public sector reforms.

In Ghana for instance, there are a number of related and mutually- supporting reforms: • there is a medium-term expenditure framework (MTEF) a – e this sets out funding allocations for each type of activity for every ministry over next 3 years. The MTEF also focuses explicitly on planned government outputs,and this necessitates the accurate measurement of actual outputs; • there are performance improvement plans a – e prepared by each ministry and setting out particular initiatives designed to help build a client- oriented service culture in the civil service; • and there are also employment contracts for senior officials a - e these ensure a close focus on what results (what perfo r mance in terms of promi s e d ou tp u t s ) are expected from senior officials over the coming year; • and a big push to decentral i z e gover nment to the district level a – e this crea t e s

GUIDE TO DIAGNOSING ECD NEEDS: Video Speech 9 7 a greater need to be able to measure gov- do to make a national M&E system is a ernment performance at the regional an d Government decree, plus some training di s t r ict level s , to provide greater support to • but really, our experience tells us that this district government management. will not be enough • Ghanaian gove rnment (GOG) officials recognize that national + sectoral M&E ■ But another key lesson we have learned is systems can support all these reforms. that there has to be real demand æ real • actually, GOG officials increasingly under- commitment — by a Government to using stand that M&E does more than support M&E if a system is to be developed in a the reforms: they are coming to realize sustainable manner that a number of the reforms will not be •An example of this is our work with the f u lly successful unless the gove rn m e n t’s Go ver nment of Benin to develop perfo rm a n c e pe rf o r mance is measured more syst e m a t i ca l ly in d i c ators to support budget decision- m a k i n g and rigorously. — as part of an MTEF • the Finance Minister in Benin is strong l y com- ■ So in your country, helps if you identify the mitted to this use of perfo r mance indicat o r s different uses + different users of M&E info, and the different types of M&E ■ Consideration of types of M&E — and of the different users and uses of them — has big ■ Different types fill different roles: implications for how you structure M&E 1. Basic financial data on gover nment spending functions and implications for who plans or on different activities, projects, sectors commissions M&E, who conducts it, and • these data are comp l e m e n t a r y to basic socio- who uses it. ec on o mic statistics or devel o p ment indicat o r s 2. Perf o r mance moni t o ri n g / i n d i c ators on govern - The Wo rld Bank has pre p a red an ECD ment inputs, processes and outputs Diagnostic Guide1—to help Governments and 3. Cost-benefit analysis an a l ysts in addressing these issues and in devel o p i n g 4. Formal program evaluation an action plan for ECD. 5. More qualitative reviews 6. Performance audits a – e typically done by ■ There are eight key steps : a National Audit Office • the Guide provides checklists of questions and issues to help you ■ Different uses of M&E findings include: • it’s really what we call a SWOT (strengths, 1. Policy formulation and planning to help we a k n e s s e s ,o p p o r tunities and threats) analysi s clarify the pros and cons of alternative types of policy intervention; 2. A second is resource allocation in budget 1. Identify key ministries and their formal processes so M&E findings can be one relationships input to Gove rnment decision - m a k i n g, • mi n i s t r ies with major role in perfo rm a n c e along with a number of other influences management — i. e . , in res o u r ce allo c a- on government; ti o n decisions and in ongoing manage- 3. To support management via greater results ment of sectors + projects orientation — the learning function; • e.g., key central ministries and large 4. Accountability—this is the quid pro quo in line ministries res p o nse to devo l u t i o n of authorit y to managers. 2. Diagnose public sector incentives and the ■ The need to understand which of these differe n t unwritten “rules of the game” uses of M&E likely to be most important in • it is these unwritten rules + incentives, your country. not formal decrees and pro c e d u re s ,w h i ch determine the behavior of ministries, ■ It also helps to understand which differe n t line managers and other civil servants. potential users of M&E actually want this info. • so what is extent of autono my + authorit y of line managers? ■ A mistake which we evaluators make is to be • what are rewards + incentives for good supply-driven: we may think that all we need performance?

1 Keith Mackay, Evaluation Capacity Development: A Diagnostic Guide and Action Framework, ECD Working Paper Se ries No 6, January 1999,The World Bank.

9 8 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing • what are sanctions for poor perfo rm a n c e ? on key issues and dimensions: • so, this Step helps answer the question: • demand and supply “does a perf o rmance culture exist?” – where is existing demand for M&E, If not, “what are the ro a d b l o cks to and how can it be consciously built up achieving it?” by “winning hearts + minds”? – where is existing supply of M&E,and 3. Find out how budget resource allocation again, how can it be built up? and line management decisions are actually • different types of M&E; taken a – e the reality, not the rhetoric • evaluation infrastructure and systems • this Step looks at actual syst e m s ,s t ru c t u re s , – what planning systems would be roles and info flows desirable to help decide which areas of Government should be evaluated? 4. Next step is to investigate the extent of – what mechanisms are necessary to influence of M&E on budget and line ensure evaluations are completed in management decisions timely manner? • ag a i n , “what is the rea l i ty ? ” , not the rhe t o ri c – what mechanisms would be set up to pr ovide M&E findings to users of them? 5. Map out the eva l u a t i o n activities and cap a - • possible support from deve l o pm e n t bilities of ministries and other organizat i on s assistance agencies • this step focuses on supply of M&E,and – this could include techn i c al assistance and on processes for making M&E findings ad v i c e , tra i n i n g , twinning arran g e m e n t s ; available support for ECD diagnoses, etc; • think of timelines for an action plan,and 6. Map out the eva l u a t i o n activities and cont r i- sequencing and speed of implementation bution of development assistance agencies – need some flexibility in approach • development agencies often focus on – but helps to set indicative targets for building M&E for donor-funded proj e c t s , 1,3, 5 and 10 years in the future; and and this may have cont r ibuted to nationa l • sustainability a – e how sustainable do M&E capacities you think your ECD efforts might be? • and they might have been involved in – what are some of the risks and threats s u p p o rting other re l evant ca p a c i ty - to it? building work in the past, in areas such as building national statistical systems ■ This is a detailed and systematic approach; it helps ensure that your ECD work takes full 7. Understand the major public sector reforms account of opportunities and possible barriers underway or completed in recent years æ these may well provide synergies with ■ I reg r et not being with you in person to discuss ECD a – e as Ghana, for example, ill u s t ra t e s these issues —I’m sure your debate will be interesting and I wish you every success with 8. Map out an action plan for ECD, focusing your ECD efforts.

GUIDE TO DIAGNOSING ECD NEEDS: Video Speech 9 9 PART V Meeting Wr a p - U p

E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing MEETING WRAP-UP 15 Christine Roth,Evaluation Advisor, UNDP

Presentations and discussions during the 1999 Beijing Conference on Evaluation Capacity Development gave us a glimpse of the ever-evolving environment we ar e living in. At the dawn of the 21st century, in a world of increasing globalizat i on , competition, and reduced public resources, scrutiny is growing for governments and development agencies to demonstrate value for money in public services.

As a consequence, the field of evaluation is changing rapidly to deal with the new demands placed on it. First and foremost, evaluation is being recognized as an essential part of good governance and public sector reform. With the advent of public sector reforms, governments and aid agencies have begun to revisit their monitoring and evaluation (M&E) arrangements, looking for ways to develop effective learning systems and promote transparency and accountability.

One of the key elements of this reform has been the adoption of results-based approach to management (RBM). The Conference focused particularly on the in t e r face between RBM and moni t o r ing and eva l u a t i o n in conn e c t i o n with public sector reform and good governance. The shift to RBM requires a focus on development results, or outcomes, rather than inputs. In this paradigm, the continuous assessment of whether intended outcomes are being achieved is critical. M&E becomes a central function in a results oriented environment. Moving to outcomes also requires the development of strategic partnerships and the concerted efforts of many actors in society. Methodologically, this approach is raising several issues, such as attribution and accountability inter alia.

During the past two days,we have engaged in lively discussions on some of these methodological issues as well as broader topics including:

■ The Independence of the evaluation function

■ The necessary link between evaluation and decision-making

■ The balance between learning and accountability

■ The challenges facing evaluation •Political environment • Systemic problems • Internal capacities (human and financial resources)

■ The role of partnerships • Joint evaluations with governments, aid agencies and development banks • Involvement of stakeholders and civil society in evaluation exercises

Of the many constraints hindering the establishment of effective monitoring and evaluation systems at national level, we have collectively identified the following: ■ Weak demand for evaluation ■ The lack of political commitment and approp r iate institutional support for eva l u a t i o n ■ The lack of standards and criteria for evaluation ■ The lack of local expertise and resources for evaluation

ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 1 0 3 As a first step to lifting these cons t ra i n t s , the conc e p t evaluators engaged in devel o p ment related activities of networks has emerged in our discussions as a and will support the national efforts in stren g t h e n i n g powerful instrument to mainstream the culture evaluation capacity development. of evaluation in the public sector by allowing: (1) exchange of experiences, (2) access to best Last,an opportunity appeared during this meeting practices, (3) sharing databases. for the development agencies in general,UNDP in particular, to maximize the synergies between the Concretely, several countries have committed to w o rk of deve l o pment banks – the Asian establishing evaluation networks at the national Development Bank in this particular case – and of and regional level. Sri-Lanka and China are two national entities to promote the role of evaluation examples and support will be provided as needed in good governance and public sector reform. by the international community. As we move forw a rd in the field of ECD, At the international level, both UNDP and the Beijing will re p resent one of the milestones World Bank, in agreement with several other that has helped furhter refine our understanding donors are promoting the establishment of an of issues and contributed in shaping the inter- international development evaluation association. n a t i onal com mu n i ty’s game plan to meet the This association will help to form links between challenges ahead.

1 0 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PART VI An n e x e s

E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from an International Conference in Beijing ANNEX I: List of Participants

C H I N A

Mr.Sun FUNQUAN Professor NCSTE – National Center for Science and Technology Evaluation P.O. Box 3814 Beijing,People’s Republic of China100038 Tel: (86) 10-685 14841/685 15544-3805 Ext.3825 Fax: (86) 10 68514854 Email: [email protected]

Mr. Li XINNAN Professor,Vice-President NCSTE – National Center for Science and Technology Evaluation P.O. Box 3814 Beijing,People’s Republic of China100038 Tel: (86) 10-685 18087/685 15544-3805 Ext.3818 Fax: (86) 10 68514854 Email: [email protected]/[email protected]

Ms. Chen ZHAOYING Professor, Deputy Director Member of Executive Committee, International Association for Technology Assessment and Forecasting Institutions P.O.Box 3814, Beijing,P.R. China 10038 Tel: (86)10 68515544/66515544 Fax: (86)10 68514854 Email: [email protected]

Mr.Fang YAN Head, Evaluation NCSTE –National Center for Science and Technology Evaluation P.O. Box 3814 Beijing,People’s Republic of China100038 Tel: (86) 10-685 18087/685 15544-3805 Ext.3818 Fax: (86) 10 68514854 Email: [email protected]

DEMOCRATIC PEOPLE’S REPUBLIC OF KO R E A Mr. Kim SEON-BIN Director Public Welfare Technology R&D Division Ministry of Science and Technology Government Complex-Gwacheon Seoul, Democratic People’s Republic of Korea, 427-760 Tel : (82-2) 503 7606 Fax :(82-2) 749 1417 Email : [email protected]

ANNEX I: List of Part i c i p a n t s 1 0 7 F I J I Mr.Leonid Nukhimovich KOMAROVER Head, Economic Analysis and Ms.Litia MAWI Prognosis Department Director of Management Improvement Office of the Prime Minister of Public Service Commission the Kyrgyz Republic 29 Greene Street, Howell Road 205 Chuy Avenue #401 Suva, Fiji Islands Tel: 22 29 24 Tel: (679) 386 001 Fax: 66 13 82/66 25 77 Fax: (679) 301 714 Mr.Sabyrbek MOLDOKULOV Mr.Yuxue Xue Deputy Chairman, State Committee on Assistant Resident Representative UNDP Fiji Foreign Investments Private Mail Bag 58a Erkindik Bvld, #210 Suva, Fiji Tel: (996-312) 22 32 92 Tel: (679) 312 500 Fax: (996-312) 66 10 45 Fax: (679) 301 718 Email: [email protected] Ms.Elzira SAGYNBAEVA National Programme Officer UNDP Kyrgystan I N D I A 90 Chuykov Street Dr. S.P.PAL Bishkek 720000, Kyrgystan Advisor,Planning and Tel: (906-312) 666100 Evaluation Organization Fax: (996-312) 660557 Planning Commission Email: [email protected]; [email protected] Yojana Bhavan Parliament Street New Delhi - 110001 M A L AY S I A Tel: (91 11) 3710162/91 11 221 3653 Fax: (91 11) 371 7681/91 11 372 5491 Dr.Selva RAMACHANDRA Assistant Resident Representative K A Z A K H S T A N UNDP Malaysia Ms. Svetlana CHERIMISSINOVA Wisma UN Deputy Head State Programmes Division Block C, Damansara Offices Complex Regional Policy Department Damansara Heights Ministry of Economy 50490 Kuala Lumpur, Malaysia 33 Pobedy Prospect/52 Abai Str, Tel: (60 3) 253 1020/ (60 3) 255 9122 Astana, Republic of Kazakhstan Fax: (60 3) 255 2870 Tel: (7-3172) 11 81 34/29 Fax: (7-3172) 11 81 36 Mr. Ab.Alim BIN ZAKARIAH Principal Assistant Director Economic Planning Unit K Y R G Y S T A N Prime Minister’s Department Blok B6, Aras 2 Miyegombo ENKHTSETSEG Kompleks Jabatan Perdana Menteri Head, Programme Support Unit Pusat Pentadbiran Kerajaan Persukutuan UNDP, Kyrgystan 62502 Putrajaya 90 Chuykov Street Tel: (603) 88882767 Bishkek 720000, Kyrgystan Tel: (906-312) 666100 Fax (603) 88883768 Fax: (996-312) 66055 Email: [email protected] M O N G O L I A Mr.Chorobek IMASHEV Deputy Minister of Finance Mr. Rentsen BATMEND The Kyrgyz Republic Principal Officer, Cabinet Secretariat 58 Erkindik Blvd., #305 Room 374 Tel: (996-312) 22 74 55 Government House, Fax: (996-312) 66 16 45 Ulaanbaatar 12, Mongolia

1 0 8 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Dr.Dungersuren SUKHJARGALMAA Mr. Kithsiri ASOKA KASTURIARACHCHI Assistant Resident Representative Assistant Resident Representative UNDP Mongolia UNDP Sri Lanka 7 Erhuu Street 202-203 Bauddhaloka Mawatha Sukhbaatar Region Colombo, Sri Lanka Ulaanbaatar, Mongolia Tel: (94 1) 582630 Fax: (94 1) 581 116 Tel: (976 1) 327 585 Email : [email protected] Fax: (976 1) 326 221 THAILAND Email: [email protected]/[email protected] Ms. Netnarumon SIRIMONTHON Assistant Resident Representative and N E PA L Programme Management Specialist Mr. Devendra Raj Panday UNDP Thailand Chairman United Nations Building Nepal South Centre 12th Floor Tripureswar Tel: (66 2) 280 0519 Khathmandu Fax: (66 2) 280 0556 Email: [email protected] P.O. Box 8242 Tel: (977) 1 247 656 Fax: (977) 1 240 685 V I E T N A M Email: [email protected] Mr. Nguyen DINH CUNG Director, Macro Economic Policy Department Central Institute for Economic Management PA K I S T A N 68 Phan Dinh Phung Ms. Shaheen MASUD Honoi, Vietnam Joint Secretary, Economic Affairs Division Tel: (84 4) 845 8241 Ministry of Finance and Economic Affairs Fax: (84 4) 856 6795 Address: MISSING TEXT MISSING TEXT THE UNITED NATIONS DEVELOPMENT PROGRAMME (UNDP) Tel: (051) 9201 706 Fax: (051) 920 5971 / (051) 921 0734 Mr. Khalid MALIK Director, Evaluation Office UNDP, Room DC1-478 P H I L I P P I N E S One United Nations Plaza Mr. Ernesto BAUTISTA New York, NY 10017 Assistant Deputy Resident Representative United States UNDP Philipppines Tel: (212) 906 5095 NEDA Sa Makati Building Fax: (212) 906 6627 106 Amorsolo Street Email: [email protected] Legaspi Village Mr. Antonio MOLPECERES 1229 Makati City Learning Resource Advisor Metro Manila, Philipppines 304 E 45th Street Email: [email protected] New York, NY 10017 Room FF-1104 Tel: (212) 906 6724 SRI LANKA Fax: (212) 906 5120 Email: [email protected] Mrs. Nanda ALAHAKONE Director,Post Evaluation Unit Ms. Christine ROTH Ministry of Plan Implementation & Evaluation Advisor UNDP, Room 476 Parliamentary Affairs, 3rd Floor One United Nations Plaza "Sethsiripaya" New York, NY 10017 Battarumulla United States Sri Lanka Tel: (212) 906 5722 Tel: 863 954 Fax: (212) 906 6627 Fax: 862 485 Email: [email protected]

ANNEX I: List of Part i c i p a n t s 1 0 9 THE WORLD BANK OPERATIONS EVALUATION DEPA R T M E N T / ASIAN DEVELOPMENT BANK (ADB) WORLD BANK INSTITUTE Mr.Peter C. DARJES Operations Manager Mr. Keith MACKAY Asian Development Bank Senior Evaluation Officer Address: OED Room G 3212,The World Bank Tel: (63-2) 632 4111 1818 H Street, N.W.Washington, DC 20433 Fax: (63-2) 636 2161 Email:[email protected] U. S. A. Tel: (202) 473 7178 Fax: (202) 522 1655 C O N S U L T A N T S Email: [email protected] Ms.Annette BINNENDIJK Mr. Ray RIST P.O. Box 146 Evaluation Advisor Basye, Virginia 22810 USA WBI Room G 3192,The World Bank Tel: (540) 856 2624 1818 H Street, N.W.Washington, DC 20433 Email: [email protected] U. S. A. Tel: (202) 458 5625 Mr.Leonard J. EARLY Fax: (202) 522 1655 Department of Finance,Australian Public Sector Email: [email protected] 33 Bringley Cr. Fraser ACT, 2615, Australia Tel: (61 2) 6259 5390 Fax: (61 2) 6259 5382 Ms.Jody ZALL KUSEK Email: [email protected] Evaluation Advisor WBI Room G 3192,The World Bank Mr. Marc HOLZER 1818 H Street, N.W.Washington, DC 20433 Executive Director,National Center for Public U. S. A Productivity, Rutgers University, NJ, USA Email: [email protected] 7th floor Hill Hall 360 King Blvd Newark, NJ 07102 Tel: (973) 353 1351/5504 Ext.23 O E C D Fax: (973) 353 5907 Email: [email protected] Mr.Niels DABELSTEIN Head of Evaluation Secretariat Mr. Adil KHAN DANIDA/OECD/DAC WP UNDP Senior Advisor Danish International Development Assistance, Taj Samudra Hotel 25, Gale face Centre Road Ministry of Foreign Affairs Colombo 3 2, Asiatisk Plads, DK 1448 Copenhagen, Sri Lanka Denmark Tel: (00941) 889190 Tel: 45.33.92. 00.39/92.00.00 Fax: 94.1. 446348/330509 Fax: 45.33.92.16.50 Email: [email protected] Email: [email protected] Mr.Rustam LALKAKA Mr.Shoko KIMURA President, Business & Technology Evaluation Division, Development Strategies Economics Cooperation Bureau, 1441 Third Avenue,Suite 5A Ministry of Foreign Affairs, OECD/DAC, Japan New York, NY 10028, USA 2-2-1, Kasumigaseki, Chiyoda-ku Tel: (212) 452 1678 Tokyo 100, Japan Fax: (212) 535 0060 Tel:81 3 35 92 83 42 Email: [email protected] Fax:81 3 35 93 80 21 Mr. Jacques TOULEMONDE Director Ms. Fiona SHERA Center for European Evaluation Expertise Economic Advisor C3E, 13 bis Place Jules Ferry-F69006 DFID, United Kingdom Lyon, France Tel: (44) 171 917 0545 Tel: 33 (0) 4 72 83 78 80 Fax: (44) 171 917 0561 Fax: 33 (0) 4 72 83 78 81 Email: [email protected] Email: [email protected]

1 1 0 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing ANNEX II: Conference Agenda

International Conference On Evaluation Capacity Development Beijing,People’s Republic Of China,27-28 October 1999

Wednesday 27 October 1999 8:00- 9:00 Registration MORNING SESSION 09:00- 10:00 Opening Ceremony: Session Chair:Mr. Li Xinnan, Vice President, Chinese National Center for Science and Technology Evaluation, RPC (NCSTE) Opening Speech: Ms. Deng Nan, Vice Minister, Ministry of Science & Technology Address: ■ Mr. Khalid Malik, Director, UNDP Evaluation Office ■ Ms. Ma Dexiu, Director, Industry Department, State Commission for Development & Planning ■ Mr. Ray Rist, Senior Advisor,World Bank Institute ■ Mr. Zhan Jingtao, Director,Public Expenditure Department, Ministry of Finance ■ Mr.Yu Liegui, Director, Industry & Transportation Department, Ministry of Finance ■ Mr.Wang Jianzeng, Director, Science & Technology Department, State Commission for Economy & Trade 10:00 - 10:15 Launching of the Handbook “Result-Oriented Monitoring and Evaluation” Mr. Qi Rang, Director, Development & Planning Department, Ministry of Science & Technology Mr. Khalid Makik, Director, Evaluation Office, UNDP 10:15 - 10:30 Break THE EVO LVING CONTEXT FOR EVALUATION CA PACITY DEVELOPMENT Session Chair:Mr. Ray Rist,Advisor,World Bank 10:30 -12:15 Statements: The New Dimensions in ECD in Interna t ional Devel o p m e n t Mr. Khalid Malik, Director, UNDP Evaluation Office

ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 1 1 1 Background paper: The Asian Perspective in Evaluation Capacity Development: The Challenges of the New Millenium Mr.Adil Khan, Senior Advisor, M&E UNDP/UNOPS, Sri Lanka Background paper: The S&T Evaluation in China: Practices and Roles Ms. Chen Zhaoying,Vice President of NCTSE Background paper: Aid Evaluation/A donor perspective Mr.Niels Dabelstein, Head of Evaluation Secretariat DANIDA, OECD/DAC WP: Questions and Answers 12:15-12:30 Objectives of the Conference Mr.Antonio Molpeceres, Learning Resource Advisor, UNDP 12:30-14:00 Lunch AFTERNOON SESSION: PERFORMANCE MANAGEMENT IN THE PUBLIC SECTOR Session Chair:Representative of Ministry of Finance, PRC 14:00-14:45 Background papers: Results Based Management:A Donor Agency Perspective Ms. Annette Binnendijk, Consultant to the Evaluation Office, UNDP Respondents: Mr.Fang Yan, Chief of Project Evaluation, NCTSE National Delegations: Mongolia, India Questions and Answers 14:45-15:30 Background paper: Linking Evaluation to Policy Formulation and Budget Processes Mr. Ray Rist, Senior Advisor,World Bank Institute Respondents: Mr.Len Early, Department of Finance & Administration, Australian Public Sector Ms. Xu Yaoling,Vice President, NCSTE Questions and Answers 15:30-15:45 Break Announcement for reception Session Chair:Mr.Wang Jianxin,Director of Evaluation Division, Planning Department,Ministry of Science & Technology, RPC 15:45-17:30 Background paper: Evaluation Capacity Building in the People’s Republic of China Mr.Peter C. Darjes, Asian Development Bank Respondents: Mr. Zhao Lu, Director of Science Division, Public Expenditure Department, Ministry of Finance National Delegation: Sri Lanka, Fiji Background paper: Public Performance Evaluation and Improvement Mr . Ma r k Holzer , Exe c u t i v e Direc t o r ,Nat i o nal Center for Public Prod u c t i v i t y, NJ , US A Respondents: Ms. Jody Kusek, Advisor,World Bank Institute Mr. Gu Wenxing, President, Shanghai Science Research Institute Questions and Answers: Wrap up UNDP Evaluation, Fang Yan, Lalkaka 18:30 - 21:00 Reception given by UNDP Evaluation Office

1 1 2 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Thursday 28 October 1999 MORNING SESSION : A) MONITORING AND EVALUATION IN A RESULTS-BASED MANAGEMENT A P P R OA C H B) EVALUATION INITIATIVES IN SCIENCE AND T E C H N O L O G Y Session Chair:Mr. Khalid Malik,Director, UNDP Evaluation Office 09:00-9:45 Background paper: M&E and RBM: Balancing Learning and Accountability Concerns,the UNDP Experience Ms. Christine Roth, Evaluation Advisor, UNDP Evaluation Office Respondents: Mr. Dai Guoqing, Director,Finance Division, Conditions Finance Dept., Ministry of Science & Technology National Delegations: Kyrgyzstan Questions and Answers 9:45-10:30 Background paper: Evaluation Partnerships Mr.Niels Dabelstein, Chairman Working group on Aid Evaluation, OECD/DAC Respondents: Br.Yang Sai, Director, Beijing Kehuan S&T Development Center National Delegation: Vietnam, Pakistan Questions and Answers 10:30-10:45 Break Session Chair:Ms.Li Maoming,Senior Advisor, NCSTE 10:45-11:30 Background paper: Evaluation Tools and Methodologies:Case Studies Mr. Jacques Toulemonde, Director, Center for European Evaluation Expertise Re s p o n d e n t s : Mr . Du Zhanyu a n , assistant Direc t o r , De vel o p ment & Planning Departm e n t , Ministry of Development & Planning Department, Ministry of Science & Technology National delegation: Russia, Malaysia Questions and Answers 11:30-12:30 Background paper: Evaluation Initiatives in Science and Technology Programs Mr.Rustam Lalkaka, former Head of UNFSTD Respondents: Mr.Wang Jianxin, Director, Evaluation Division, Development & Planning Dept., Ministry of Science and Technology National delegation : Korea Background paper: Review of Draft of Science & Technology Evaluation Standard Ms. Chen Zhaoying,Vice President, NCSTE Respondents: Mr. Zhang Zheng, Director,Wuhan Science & Technology Evaluation Center National delegation: Kazakstan Questions & Answers 12:30-14:00 Lunch AFTERNOON SESSION: ECD OPPORTUNITIES FOR NATIONAL INITIATIVES AND INTERNATIONAL COOPERATION Session Chair:Mr. Ray Rist,Advisor,World Bank Institute 14:00-14:45 ECD opportunities for National Initiatives Video Speech: Guide to Diagnosing ECD Needs: Mr. Keith McKay, Senior Evaluation Officer,World Bank

ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 1 1 3 Questions and Answers 14:45-15:45 Group Work: National needs and planned steps for ECD National delegations, UNDP,World Bank and Asian Development Bank as facilitators 15:45-16:15 Current trends and future steps for ECD at the national level Presentations by rapporteurs of working groups 16:15-17:00 Break Session Chair:Ms. Chen Zhaoying,Vice President,NCSTE 17:00-17:15 Meeting wrap-up: Ms. Christine Roth, Evaluation Advisor, UNDP Evaluation Office 17:15-17:45 Concluding remarks: ■ Mr. Ming Tinghua, Chairman, Invention Association of China (former Director of Chinese Patent Office) ■ Mr. Ray Rist, Senior Advisor,World Bank Institute ■ Mr.Yang Qiwen, Director, Basic Research Department, Ministry of Science & Technology ■ Mr. Khalid Malik, Director, UNDP Evaluation Office ■ Mr. Li Xinnan, Vice President of NCSTE, Vice President of NRCSTD

Friday 29 October, 1999 18:30-21:00 RECEPTION HOSTED BY THE NCSTE: CLOSING CEREMONY, GREAT HALL OF THE PEOPLE Chair:Ms.Li Maoming,Senior Advisor, NCSTE Presence of senior officers Mr. Zheng Guoan, Director, Ministry of Science & Technology

1 1 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing UNITED NATIONS DEVELOPMENT PRO G R A M M E

Evaluation Office United Nations Development Programme One United Nations Plaza New York,NY 10017, USA Tel. (212) 906 5095, Fax (212) 906 6008 Internet: http://intra.undp.org.eo or http://www.undp.org/eo