MEASURE RESULTS PACKAGE EVALUATION AND PRE-DESIGN STUDY MAY 2001 Judith R. Seltzer James C. Knowles Anibal Faúndes Roger W. Rochat Anne-Marie Foltz Mark H. Spohr Submitted by: LTG Associates, Inc. TvT Associates, Inc. Submitted to: The United States Agency for International Development Under USAID Contract No. HRN–C–00–00–00007–00 This document is available in printed or online versions (POPTECH Publication Number 2001–019–004). To review and/or obtain a document online, see the POPTECH web site at www.poptechproject.com. Documents are also available through the Development Experience Clearinghouse (www.dec.org). Printed copies and additional information about this and other POPTECH publications may be obtained from: The Population Technical Assistance Project 1101 Vermont Avenue, NW, Suite 900 Washington, DC 20005 Telephone: (202) 898-9040 Fax: (202) 898-9057 [email protected] MEASURE Results Package: Evaluation and Pre-Design Study was made possible through support provided by the United States Agency for International Development (USAID) under the terms of Contract Number HRN–C–00– 00–00007–00, POPTECH Assignment Number 2001–019. The opinions expressed herein are those of the authors and do not necessarily reflect the views of USAID. ACKNOWLEDGMENTS The evaluation and pre-design team would like to thank USAID staff for its work in designing the scope of work and in briefing the team. In particular, we thank Krista Stewart, CTO for the MEASURE Results Package (RP), and the technical advisors for MEASURE: Joanne Jeffers, Norma Wilson, Jacob Adetunji, Michal Avni, Sara Pacque-Margolis, and Naomi Blumberg. The team also appreciated the considerable effort made by the MEASURE partners in preparing briefings and providing documentation of the work of the five component projects under the RP. We especially thank the five project directors and their staff for careful reviews and helpful comments on the draft report. The project directors are Martin Vaessen of DHS+, Ties Boerma of MEASURE Evaluation, Nancy Yinger of MEASURE Communication, Robert Bush of the International Programs Division of the U.S. Census Bureau, and Timothy Johnson, PASA Director, Division of the Reproductive Health, the Centers for Disease Control and Prevention (CDC). The team also thanks Jennifer Bryce of the Department of Child and Adolescent Health and Development of the World Health Organization for her useful comments on the draft report. ACRONYMS AND ABBREVIATIONS AED Academy for Educational Development BASICS Basic Support for Institutionalizing Child Survival BUCEN–SCILS U.S. Census Bureau–Survey and Census Information, Leadership, and Self- Sufficiency CA Cooperating agencies CDC Centers for Disease Control and Prevention CDS Census Design System CSPro Census and Survey Processing System CTO Cognizant technical officer DFID Department for International Development (United Kingdom) DHS Demographic and Health Survey DISH Delivery of Improved Services for Health project (USAID/Uganda) DRH Division of Reproductive Health (CDC) FHI Family Health International FP Family planning FY Fiscal year GIS Geographic Information Systems G/PHN Bureau for Global Programs, Field Support and Research, Center for Population, Health and Nutrition GPS Global Positioning Systems GTZ German Technical Cooperation HIV/AIDS Human immunodeficiency virus/acquired immune deficiency syndrome HN Office of Health and Nutrition HTA High transmission area ICPD International Conference on Population and Development IDB International Data Base (U.S. Census Bureau) IMCI Integrated Management of Childhood Illness IMPS Integrated Microcomputer Processing System (U.S. Census Bureau) INTRAH Program for International Training in Health IPPF International Planned Parenthood Federation ISSA Integrated System for Survey Analysis (DHS+) JSI John Snow, Inc. KAP Knowledge, attitudes, and practice LQAS La Quality Asssessment Sampling MAB MEASURE Advisory Board MEASURE Monitoring and Evaluation to Assess and Use Results MICS Multiple Indicator Cluster Surveys MIT Management Implementation Team MMT MEASURE Management Team MSH Management Sciences for Health NAS National Academy of Sciences NGO Nongovernmental organization NORAD Norwegian Agency for Development Cooperation PAHO Pan American Health Organization PASA Participating agency service agreement PATH Program for Appropriate Technology in Health PCS Population Communication Services project PHN Population, health, and nutrition PHR Partnerships for Health Reform POP Office of Population PRB Population Reference Bureau PRIME Program for International Training in Health (technical assistance–INTRAH) QAP Quality Assurance Project QIQ Quick Investigation of Quality (instrument) R Rand (currency of South Africa) R4 Results Review and Resource Request RAMOS Reproductive Age Mortality Study RH Reproductive Health RHINO Routine Health Information Network RHIS Routine health information system RP Results Package SARA Support for Analysis and Research in Africa Sida Swedish International Development Authority SO Strategic Objective SPA Service Provision Assessment UCR University of Costa Rica UNAIDS Joint United Nations Programme on HIV/AIDS UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USAID United States Agency for International Development WHO World Health Organization CONTENTS Page Executive Summary.........................................................................................................i 1. Introduction...............................................................................................................1 1.1 Objective of the Evaluation and Pre-design..........................................................1 1.2 Methodology ........................................................................................................1 1.3 History of the MEASURE Results Package .........................................................2 2. The Current Context ................................................................................................4 2.1 Context of USAID ................................................................................................4 2.2 Current Agreements under the Results Package...................................................5 2.3 Guiding Principles for the MEASURE Results Package......................................6 2.4 Stakeholders, Decentralization, and Sustainability...............................................8 3. Technical Findings and Recommendations..........................................................11 3.1 Improved Coordination/Partnerships at International, USAID, Cooperating Agency, and Country Levels..........................................................11 3.2 Increased Host Country Institutionalization .......................................................18 3.3 Improved Tools and Methodologies ...................................................................25 3.4 Achievements in Improved Information through Appropriate Data Collection, Analysis, and Evaluation ....................................................................................34 3.5 Improved Dissemination and Utilization of Data ...............................................45 4. Management, Funding, and Design: Findings and Recommendations .............52 4.1 Management .......................................................................................................52 4.2 Funding ..............................................................................................................54 4.3 Overall Design ....................................................................................................59 4.4 Centerwide Results Package...............................................................................61 4.5 Future Procurements ...........................................................................................61 5. Conclusions..............................................................................................................68 TABLES 1. Funding of MEASURE Components by Source of USAID Funds, FY 1997–2001..........................................................................................................55 2. Funding Trends by Core G/PHN Accounts for Three MEASURE Partners, FY 1997–2001..........................................................................................................56 3. Summary of Donor Contributions to MEASURE–assisted Work, FY 1997–2001..........................................................................................................57 APPENDICES A. Scope of Work B. References C. Persons Contacted D. E-mail Questionnaire Sent to USAID Missions by the MEASURE Cognizant Technical Officer E. Host Country Capacity Building F. Summary of MEASURE Training Activities G. MEASURE Tools and Technologies List H. Child Health and Child Survival Topics in the DHS+ Core Questionnaire I. MEASURE DHS+ Surveys J. International Surveys Conducted with CDC–DRH Technical Assistance, 1997–2002 K. Cost of MEASURE DHS+ Surveys L. Summary of DHS+ Work in Nutrition M. Routine Health Information N. Summary of Basic Organizational Structures O. Yinger Memorandum, May 11, 2001, MEASURE Communication EXECUTIVE SUMMARY A 6–person team carried out a pre-design and evaluation effort of the MEASURE Results Package (RP) between April 16 and May 18, 2001. (MEASURE is the acronym for the project’s title, Monitoring and Evaluation to ASsess and Use REsults.) The objectives of this assignment were to advise staff