EVALUATION OF THE POPULATION FIELD INFORMATION SERVICES PROJECT AMERICAN PUBLIC HEALTH ASSOCIATION International Health Programs 1015 Eighteenth Street, N.W. Washington, D.C. 20036 ' " /i l. I i* EVALIIATION OF THE POPULATIONi FIELD INFOR .ITION SERVICES PROJECT A Report Prepared By: GERALD HURSH-CESAR, Ph.D. 1!ILLIAM C'DY , Ph. D. ROBER[T Cl A'.FOR[), Ph.D}. During the Period: FE LIARY 21 THROUGH APRIL 6, 1979 Under the Au:pices of: THE AENICAN PURLIC HEALIII ASSOCIATION Supported By The: U.S. AGFN CY FOR INTERNATIONAL DEVELOPMENT OFFICE OF POPULATION, AID/pha/c-1203 AUTHORIZATION: Ltr.POP/FPS: 1/26/79 Assgn. No. 1100-126 CONTENTS Page PROLOGUE iv EXECUTIVE SUMMARY 1 I. BACKGROUND 5 A. Introduction 5 B. Methodology 5 1. Early Activities 5 2. Country Visits 7 3. Post-travel Activities 8 4. Limitations of Findings 9 II. CONTRACTOR PERFORMANCE 11 A. Major Project Outputs 11 1. Resource Sets 11 2. Field Services 11 3. Media Coverage 13 4. Contract Performance B. Evaluation Perspective 13 1. AID/,Washington Perspective 14 2. G'4U Perspective 15 C. Project Implementation 16 1. Project Management 16 2. Campaign Resource Sets 18 3. Ad Hoc Field Services Teams 20 4. National and International Mass Media 21 5. Summary of Performance 22 III. LATIN AMERICA 24 A. Brazil: ,Mlarch 6-12 24 1. Activities and Respondents 24 2. Nature of Family Planning IE&C Program 24 3. PIP-II Impact 24 4. Future Directions 26 i Page B. Peru: March 13-14 29 1. Activities and Respondents 29 2. Nature of Family Planning IE&C Program 29 3. PIP-II Impact 29 4. Future Directions 30 C. Mexico: March 14-18 30 1. Activities and Respondents 30 2. Nature of Family Planning IE&C Program 30 3. PIP-II Impact 30 4. Future Directions 32 D. Summary -- Latin America 33 IV. AFRICA 37 A. Tunisia: 'larch 3-8 1. Activities and Respondents 37 2. Nature of Family Planning IE&C Program 37 3. PIP-II Impact 37 4. Future Directions 40 B. Kenya: March 9-16 41 1. Activities and Respondents 41 2. Nature of Family Planning IE&C Program 41 3. PIP-II Impact 41 4. Future Directions 48 V. ASIA 50 A. Indonesia: March 4-10 50 1. Activities and Respondents 50 2. Nature of Family Planning IE&C Program 50 3. PIP-Ii Impact 63 4. Future Directions 68 B. Thailand: March 12-13 72 1. Activities and Respondents 72 2. ~Nature of Family Planning IE&C Program 73 3. PIP-II Impact 80 4. Future Directions 81 C. Summary -- Asia 84 ii Page VI. CONCLUSIONS AND RECOMMENDATIONS 88 A. Review of Findings 88 I. Distribution of Materials 90 2. Adaptation of Miaterials 91 3. Host Country Awareness of Materials 91 4. Host Country Awareness of Services 91 5. Host Country Reaction to Materials 92 6. Host Country Use of Materials 93 7. In-country Audiences 93 8. Culture-specific Outputs 94 9. Indigenous Production & Social Marketing 95 B. Conclusions 97 1. Reasons for tio Impact 97 2. Undertaking Future Projects 98 C. Recerrrendations 99 1. Culture-soecific Materials and Production 99 2. Field Wforkers and Acceptor Audiences 101 3. Appropriate Tyoes of 1Media 104 4. Population Officers and IE(&C "Marketing" 106 5. Social Marketing as an IE9C Strategy 107 6. USAID's Role in Providing Services 109 7. M.!anagc,"ent Strategy and Performance tMeasures 113 APPENDICES I Officials and Respondents viii II Distribution of Field Information Services Materials xiv 10/1/76 - 3/31/76 to Population Officers III a. PIP - II Evaluation Plan xvi III b. Plan of Work xxv IV Contractual PIP - II Outputs xxx i PROLOGUE This report is an evaluation of the Population Field Information Services Project (known as "PIP-II") contracted to George Washington University through the AID/DS Office of Population. A number of findings reported here are critical of the contractor's per­ formance in fulfilling contractual requirements. On May 30, 1979, the evaluation team met with representatives of AID/Washingtor and GWU to air their different views. Various earlier statements that the evaluation was unfair or inaccurate were withdrawn at that meeting. Some changes in the report were made as a result of the meeting, many of which were the kinds of editorial chanqes that are made in the process of moving from draft to final version. Other changes produced more precise and less abso­ lute language. For example, "AID Officers generally were..." was changed to "of the USAID and AID officials interviewed, most were...". And, we changed some absolute statements to relative ones: from "no impact" to "little impact," but this had been done before the meeting. Another change was made to give the GWU version of one episode. In a situation in which the USAID respondents saw the G!,IU role in one light, we added to the text to indicate that G!JU saw their role in a different light. The meeting did not, however, resolve major differences amonq the nartici­ pants. Differences among them center on three issues: (1)respondents' perceptions of the project, (2) the use of project management plan, and (3) the criteria for measuring contractor performance. The first differenut cor: erns varying perceptions. For example, the GWU team felt that some of its efforts had been slighted by remarks unattributed to indi­ vidual respondents that were critical of Lhe contractor's area expertise, manage­ rial style, service flexibility, and the degree to which the team had been proactive or reactive in carrying out assignments. Our position is that respondents' identities are confidential and that the contractor's perceptions cannot substitute for respondents' perceptions. For example, if respondents perceive the contractor to have little relevant expertise for IE&C programs in Asia, the perception is a valid basis for decision-making irrespective of the number of years logged by team members in Asian countries. We did not deal only with what USAID and in-country officials knew about the project. We dealt as well with what they felt about the project. The second major difference concerns the criticism in this report that the con­ tractor lacked on paper an overall management strategy for the planning, implemen­ tation, and evaluation/monitoring of the PIP-I project. i'v The contractor and contract monitor's objection that work plans did and do indeed exist was accepted as valid. We used a phrase, "Plan of Work,!' which has the specific connotation of "work plan" in AID/Washington. We have amended the term to "management plan," because we were not talking about,work plans. Rather, we meant an overall organizational rationale and system for carrying out project activities toward specific measurable objectives, and for employing managerial criteria for decisions about all aspects of these activities: countries of work, needs to be addressed, investments required, media to use, messages employed, and audiences reached. In short, we were looking for evidence of a strategy that envisaged causal relationships: a strategy that prioritized project activities by their assumed necessity and sufficiency for changing existing conditions toward some desired -developmental goal-.. We did not find this kind of managerial strategy. If there was ambiguity about the object of our search as described in Chapter II,oaqes 16-17, it might have been clarified through cross-reference to Appendix lia and IIIb which were noted in Chapter I(pages 5-10) as listing the indicators we sought to study with respect to project management. Further, the final recommendation in Chapter VI of the report (pages 28-32) describes attributes of the kind of management plan missing in the present project. The third major difference concerns the yardstick used to measure contractor performance. The contractor felt that we should have evaluated performance against a more flexible set of criteria than the January 18, 1978 contract. He felt that measuring performance against eight different work plans approved since the original contract was a more precise measure than using the performance requirements of contract documents. Further, the contractor felt that since we did not see all work plans, we could neither know the true nature of the project nor fairly judge its performance. For reasons of depleted files and other factors, we indeed did not see all work plans. We did, however, see all contracts, amendments, and project papers as well as many of the work plans. But since we had not seen all plans, it was agreed that we should examine those now made available to us for the purpose of deciding whether various statements about contractor performance were valid. Accordingly, we re-examined all contract documents, work plans, and project papers. The re-analysis has produced no changes in our findings. For the purpose of evaluating contractor performance, three dates are significant: o October 1, 1976 -- the date of the original contract o January 18, 1978 - the date of the present contract o March 1, 1979 -- the approximate date we were in the field evaluating contractor performance. V For the purpose of evaluating contractor performance, three contractual require­ ments are significant: o The contractor was required to provide multi-media, multi-audience Popular Information Campaign Resource Sets. o The contractor was required on request to provide Field Service Teams as the means of transferring the information of the resource sets to in-country circumstances through professional guidance of adaptation, pretesting, evaluation, distribution, and utilization. o The contractor was required as of July, 1978 to adapt and process the information of the resource sets for dissemination through in-country and international mass media. The schematic below summarizes the significant dates and requirements of the contract.