Special Evaluation Study Knowledge Products and Services: Building a Stronger Knowledge Institution IndependentIInddependdent EvaluationEEvvaaluuaatiioonn This evaluation report is posted on the Independent Evaluation Department's website (www.adb.org/evaluation) upon circulation to Management and the Board. As soon as the ADB Management Response and the Chair's Summary of the Development Effectiveness Committee meeting on this report become available, they will be attached to the report which will then be re-posted on the website. Special Evaluation Study November 2012 Knowledge Products and Services: Building a Stronger Knowledge Institution Reference Number: SES:REG 2012-18 Independent Evaluation: SS-119 NOTE In this report, $ refers to US dollars. Director General V. Thomas, Independent Evaluation Department (IED) Director H. Hettige, Independent Evaluation Division 2, IED Team leaders M. Gatti, Principal Evaluation Specialist, IED B. Graham, Senior Evaluation Specialist, IED Team members F. De Guzman, Evaluation Officer, IED R. Isidro-Cajilig, Evaluation Assistant, IED Independent Evaluation Department, SS-119 The guidelines formally adopted by the Independent Evaluation Department on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of the management of the Independent Evaluation Department, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgment as to the legal or other status of any territory or area. Abbreviations ADB – Asian Development Bank ADBI – Asian Development Bank Institute ADF – Asian Development Fund ADO – Asian Development Outlook BPMSD – Budget, Personnel, and Management Systems Department CoP – community of practice COSO – Central Operations Services Office CPS – country partnership strategy DEfR – development effectiveness review DER – Department of External Relations DMC – developing member country EARD – East Asia Department ERD – Economics and Research Department ERP – external review panel ETSW – economic, thematic, and sector work ICT – information and communications technology IED – Independent Evaluation Department IMF – International Monetary Fund ISTS – information systems and technology strategy KM – knowledge management KMAP – knowledge management action plan KMF – knowledge management framework KPS – knowledge products and services KSSC – Knowledge Sharing and Services Center MAKE – Most Admired Knowledge Enterprises MDB – multilateral development bank MfDR – managing for development results NGO – nongovernment organizations OAS – Office of Administrative Services OIST – Office of Information Systems and Technology OREI – Office of Regional Economic Integration PARD – Pacific Department PRC – People’s Republic of China PSOD – Private Sector Operations Department RCS – regional cooperation strategy RKSI – Regional Knowledge Sharing Initiative RRP – report and recommendation of the President RSDD – Regional and Sustainable Development Department SERD – Southeast Asia Department SES – special evaluation study SPD – Strategy and Policy Department SWOT – strengths, weaknesses, opportunities, and threats TA – technical assistance VPKM – Vice President (Knowledge Management and Sustainable Development) WPBF – work program and budget framework Contents Acknowledgments v Foreword vii Executive Summary ix Chapter 1: Introduction 1 A. Background 1 B. Objective and Scope of the Evaluation 1 C. Approach, Methodology, and Limitations 2 D. Findings from Other Evaluations 3 E. Organization of the Report 3 Chapter 2: ADB’s Knowledge Management Agenda 5 A. Terms and Definitions 5 B. Knowledge Management Initiatives 5 C. Current Knowledge Management Institutional Structures 6 D. Stocktaking of Knowledge Products and Services 8 Chapter 3: Enabling Environment for Knowledge Management 11 A. Strategic Response 11 B. Organizational Response 16 Chapter 4: Knowledge Management Cycle 24 A. Needs Identification 24 B. Generation 27 C. Sharing 34 D. Use 37 Chapter 5: Addressing Needs of Knowledge Stakeholders 43 A. Knowledge Management Agenda and the Development Challenges 43 B. Gauging and Prioritizing Knowledge Needs 45 C. Conformity with International Good Practice and Change Management 48 Principles D. Consistency of Knowledge Products and Services with Developing 51 Member Countries’ Expectations Chapter 6: Overall Assessment 56 Chapter 7: Building a Stronger Knowledge Institution 58 A. Knowledge Institution Needs 58 B. Progress Toward Becoming a Stronger Knowledge Institution 64 C. Key Gaps and Constraints—Areas for Action 66 Chapter 8: Findings, Recommendations, Lessons, and Issues 69 A. Key Findings and Recommendations 69 B. Lessons 74 C. Key Issues 75 APPENDIXES 1 Evaluation Framework and Rating Methodology 78 2 Previous Surveys and Evaluations 84 3 Glossary of Terms and Definitions 93 4 Knowledge Management Initiatives, Institutional Structures, and 96 Processes 5 Knowledge Products and Services, 2004–2011 101 6 Assessment of Overall Performance 112 7 Summary of Achievements of 2009–2011 Knowledge Management 116 Action Plan 8 Summary of ADB Staff Perceptions Survey Results 117 9 Summary of Basic Data on Communities of Practice 126 10 Summary of Developing Member Countries' Perceptions Survey Results 131 11 Technical Assistance and Other Modalities with Significant Knowledge 137 Content 12 External Review of Selected Knowledge Publications 150 13 Statistics on Sharing and Use of Knowledge Products 160 14 Case Studies of Influential Knowledge Products 170 15 Comparator Assessment on Knowledge Management 189 16 Future Strategic Directions 203 17 Knowledge Institution Conceptual Framework 211 18 Summary of Major Achievements and Gaps under the 2004 Knowledge 219 Management Framework and 2009 Knowledge Management Action Plan SUPPLEMENTARY APPENDIXES (AVAILABLE UPON REQUEST) A Bibliography B Survey Questionnaires C Highlights of Selected Knowledge Evaluations D Communities of Practice Information Questionnaire and Profiles E External Review Panel Methodology F Comparator Profiles G Knowledge Management Center Outputs H ADB Staff Perceptions Survey Results I Developing Member Countries' Perceptions Survey Results J Basic Data on Communities of Practice K Corporate Policies and Strategies Dealing with Knowledge Management L Achievement of Knowledge Management Action Plan M Evaluation Criteria Ratings Acknowledgments This special evaluation study is a product of the Independent Evaluation Department (IED) of the Asian Development Bank (ADB). The evaluation was led by Marco Gatti as the task manager/co-team leader, assisted by Benjamin Graham as co-team leader. Franklin De Guzman provided technical support, and administrative support was provided by Rosel Isidro-Cajilig. The report is based on contributions from a team of consultants. Rolf Westling and Thierry Senechal (strategic management specialists), Francois Fortier (knowledge management specialist), and Magdalena Casuga (evaluation specialist) assisted as consultants in the preparation of the special evaluation study, while Catherine Gwin and Michael Bamberger provided advice on the evaluation design and peer reviewed an early draft of the report. Intermittent inputs were also received from Eunkyung Kwon, Agnes Anabo, Rashel Yasmin Pardo-Dimaano, and Carmencita Balbosa. The External Review Panel of ADB Knowledge Publications comprised Dr. Peter Berman, Adjunct Professor of Population and International Health Economics, Harvard University; Ms. Margaret Catley-Carlson, Board Member, United Nations Secretary General Advisory Board on Water and Sanitation; Dr. Satish Chand, Professor of Finance, University of New South Wales; Dr. Herminia Francisco, Director, Economy and Environment Program for Southeast Asia; Dr. Teresa Ho, Professorial Lecturer, School of Economics, University of Philippines; Dr. David James, Consultant, Ecoservices Pty. Ltc., Australia; Dr. Ravi Kanbur, T.H. Lee Professor of World Affairs, Cornell University; Dr. Rakesh Mohan, Professor in the Practice of International Economics of Finance, School of Management, Yale University; and Dr. Rohini Pande, Mohammed Kamal Professor of Public Policy, John F. Kennedy School of Government, Harvard University. Fatema Begum provided assistance with the External Review Panel analysis. The team wishes to thank the government officials and representatives of other development partners, who made time for interviews as part of the collection of qualitative data to inform background papers and various assessments. The team also wishes to thank those staff of ADB in Manila and in resident missions who made time to be interviewed. In particular, IED would like to thank staff of the office of the Vice President Knowledge Management and Sustainable Development and the Regional Sustainable Development Department for their proactive assistance. Their inputs all strengthened the evidence base for this report. IED retains full responsibility for this report. The evaluation was conducted under the overall direction of Vinod Thomas, Director General, IED and Hemamala Hettige, Director, Division 2, IED. Foreword The Asian Development Bank (ADB) operates in the rapidly changing environment of the Asia-Pacific region. While the region has experienced unprecedented growth in recent decades,