Appendix a Methods and Data Set Description
Total Page:16
File Type:pdf, Size:1020Kb
Appendix A Methods and Data Set Description 1. This appendix describes multiple analytical methods used for of the evaluation. These included desk reviews of governance and anticorruption (GAC) responsiveness; compara- tive case studies; sectoral and thematic analyses; statistical analysis; surveys, structured in- terviews, and facilitated discussions with stakeholders; and process and budget reviews. Desk Review of GAC Responsiveness of Bank Country Programs and Projects 2. The evaluation included desk reviews of Bank Country Assistance Strategies (CASs), projects, and linkages to related economic and sector work (ESW) over the FY04–10 period to assess the effectiveness of the strategy in enhancing the GAC-responsiveness of Bank-country engagement during design and implementation. Using standardized questionnaires, the re- views covered both upstream design and downstream implementation processes. Therefore, they relied on CASs, CAS Completion Reports (CASCRs), Country Portfolio Performance Re- views, and related ESW, as well as Project Appraisal Documents (PADs), Implementation Sta- tus Reports (ISRs), Quality Assurance Group (QAG)analyses, Implementation Completion Reports (ICRs), and similar documentation for recipient-executed trust funds and World Bank Institute (WBI) activities. They also draw on Independent Evaluation Group (IEG) reports in- cluding Country Assistance Evaluations (CAEs), Country Program Evaluations (CPEs), CASCR reviews, ICR reviews, Project Performance Audit Reports (PPARs), as well as the In- tegrity Vice Presidency‘s Detailed Implementation Reviews (DIRs), other Integrity Vice Pres- idency analyses, and summary results of investigations. 3. The sampling methodology for the desk reviews was designed to support three levels of analysis and to ensure adequate representation of sample countries across a number of pa- rameters including income level, governance performance, geographic region, and receipt of country GAC (CGAC) and Governance Partnership Facility (GPF) window support (Figure A.1). The methodology also sought to ensure representation of sample projects across geo- graphic regions and sectors. The Level 1 analysis analyzed whether the Bank was more sys- tematic in addressing GAC issues in countries, sectors, and projects since the launch of the strategy. Level 2 assessed whether CGAC/GPF window efforts contributed to improve- ments in GAC responsiveness. Level 3 involved field-based assessments of the results of GAC efforts. 105 APPENDIX A METHODS AND DATA SET DESCRIPTION Figure A.1. Three Levels of Analysis Level 3: What was the impact of GAC-responsive Bank efforts in client countries? What worked, what did not? • 12 countries • 70 projects Level 2: What difference did CGACs and related efforts make in improving GAC responsiveness? Level 2a. • 33 countries What worked, what • 160 projects didn’t with CGAC and Window 1 countries? Level 1: Was the Bank more systematic in addressing GAC at the country, sector, and project levels? • 50 countries • 200 projects 4. Level 1 Sampling. Level 1 analysis assessed whether the Bank has been more systematic and strategic in addressing GAC issues at the country, sector, and project levels since the launch of the strategy. To support this broad review of GAC-responsiveness of Bank opera- tions during design and implementation, countries were selected randomly from clusters representing governance performance (as measured by their governance scores in the Coun- try Policy and Institutional Assessment, or CPIA), per capita income levels (according to the World Bank country classification), and geographic region.42 A sample of 50 out of a possi- ble 143 borrowing countries—comprising 17 CGAC and Window One countries and 33 non- CGAC, non-Window One countries—was selected for desk review of GAC-responsiveness (Table A2). From these selected countries, a random sample of projects was chosen to ensure a pre-identified distribution by region and sector for desk review.43 The resulting sample of 200 closed and active projects approved during FY04–10 was selected out of a total popula- tion of 806 International Bank for Reconstruction and Development (IBRD) or International Development Association (IDA)-financed projects and 452 trust fund projects.44 Projects were selected randomly based on regional and sectoral distribution (Table A3). 5. Some inconsistencies with project lists were discovered during the project desk reviews. This led to re-sampling of part of the Level 1 project sample. First, eight trust fund projects were found to have approval dates outside the period of study (FY04–10); these were re- placed with the projects that had the next highest random number in the population of 452 trust fund projects, keeping the regional and sectoral distribution intact. Second, nine projects were found to have both a trust fund and an IBRD or IDA component and so were considered duplicates; to avoid overlaps, these were replaced by projects with the next highest random number in the population , keeping the same regional and sector distribu- tion of 122 IBRD/IDA lending projects and 78 trust fund projects. 6. The representativeness of the countries and projects relative to the CPIA governance score and gross national income (GNI) per capita of the whole population was tested and confirmed (Figure A.2). 45 Average GNI per capita and CPIA scores of each sample country do not deviate more than +/- 10 percent from regional averages within a cluster. For exam- 106 APPENDIX A METHODS AND DATA SET DESCRIPTION ple, Mali has an average CPIA governance score of 3.50 and GNI per capita of $460 where Sub-Saharan Africa Regional averages are 3.37 and $488 for CPIA scores and GNI per capita, respectively. Mali‘s deviation is +6 percent for CPIA score and -4 percent for GNI per capita. This holds for all Level 1 countries. A list of selected countries and projects is located in Tables A.2 and A.3. Figure A.2. Level 1 Country Sample Representativeness by CPIA and GNI Per Capita 7. Level 2 Sampling. The Level 2 analysis assessed whether CGACs and similar efforts made a difference by contributing to improvements in GAC-responsiveness of Bank opera- tions. Using the results of the Level 1 desk review, the Level 2 analysis involved a compara- tive analysis of Bank-country engagement in CGAC and Window 1 countries and matched pairs of non-CGAC, non-Window 1 countries. It also sought to assess the contribution of CGACs and Window 1-financed efforts to GAC responsiveness over time. To arrive at a Level 2 country sample, 33 low and lower middle income countries from the lowest three CPIA governance quintiles (consisting of CPIA governance scores between 1 and 3)— comprised of 17 CGAC and Window 1 countries, and 16 non-CGAC, non-Window 1 matched pairs—were drawn from the Level 1 country sample (Table A.2).46 Representative- ness of Level 2 countries was tested and confirmed and 160 projects in these 33 countries were drawn from the Level 1 project sample (Figure A.3). Figure A.3. Level 2 Country Sample Representativeness by CPIA and GNI Per Capita 107 APPENDIX A METHODS AND DATA SET DESCRIPTION Table A.1. Sampling of CGAC and GPF Window Countries47 CGAC Countries (In italics) GPF WINDOW 1a/ GPF WINDOW 2b/ GPF WINDOW 3c/ AFRICA Burkina Faso Burkina Faso Angola Africa Burundi Cameroon Cameroon Cameroon DRC Cape Verde Congo, Democratic Republic of (DRC) Ghana Cote d'Ivoire Ethiopia Kenya Djibouti Kenya Liberia DRC Mali Nigeria The Gambia Swaziland Sierra Leone Ghana Zambia Uganda Guinea Zambia Guinea-Bissau Kenya Mali Mauritania Mozambique Niger Nigeria 5/ Rwanda Senegal Sierra Leone Sudan 4/ Uganda Zambia EAST ASIA AND PACIFIC Cambodia Cambodia Cambodia Asia Indonesia Mongolia Indonesia d/ Mongolia Philippines Lao PDR Mongolia Papua New Guinea Philippines Thailand Timor-Leste EUROPE AND CENTRAL ASIA Albania Albania Bosnia-Herzegovina Moldova Tajikistan Kosovo Ukraine Kazakhstan Uzbekistan Kyrgyz Republic Russian Federation Tajikistan Turkey Uzbekistan LATIN AMERICA AND CARIBBEAN Guatemala Haiti Argentina e/ Latin America Honduras Brazil Paraguay Chile Peru Colombia (Continued on next page) 108 APPENDIX A METHODS AND DATA SET DESCRIPTION CGAC Countries (In italics) GPF WINDOW 1a/ GPF WINDOW 2b/ GPF WINDOW 3c/ Dominican Republic e/ Honduras e/ Jamaica Mexico Paraguay Peru d/ MIDDLE EAST AND NORTH AFRICA Iraq Morocco Lebanon Egypt Yemen Jordan West Bank and Gaza Yemen SOUTH ASIA Bangladesh Afghanistan Afghanistan Pakistan Nepal Nepal Bangladesh Sri Lanka India Nepal Notes: CGAC countries in italics; Country = Country selected in IEG sample a. Includes Grant Funding Request (GFR)2079 allocated to peer reviewers for Window 1. b. Includes GFRs allocated regionally for country studies: GFR 3098 for Angola, Cameroon, Mongolia, Lao PDR, the DRC, Ghana, Niger and Nigeria; GFR 2976 (2998) allocated to Timor-Leste, Democratic Republic of Congo, Ghana, Guinea, Mauritania, Niger, Nigeria; GFR 1739 allocated to India, Indonesia, Mexico, Jordan, and Nepal; GFR 2174 to Kenya, Zambia, Democratic Republic of Congo (and eastern central Africa), Bangladesh, India (two states), Mexico, Chile, the Philippines and South Korea; GFRs 1698 and 1728 allocated to Europe and Central Asia; GFRs 1749, 2615 and 2716 allocated to Latin America and the Caribbean; GFR 2659 allocated to the Middle East and North Africa; GFR 1712 allocated to South Asia; and GFR 2702 allocated to Southern Africa. c. Non-country specific. Nine GFRs coded ―Global‖ for training and knowledge purposes or for regions. d. Projects in these countries have two separate GFRs for the same project, a Bank-executed portion and a Recipient-executed GFR. They are GFRs 3579 and 2678 for South Sudan; GFRs 1804 and 216 for Indonesia; and GFRs 5052 and 2716 for Peru. e. Countries with two separate GFRs for different projects. For Nigeria GFR 4027 (W1) and 1695 (W2); for Sierra Leone GFR 4001 (W1) GFR 2548 (W2); for Uganda GFRs 4028 (W1) and 2653 (W1); for the Dominican Republic GFR 2720 (W1) and 4129 (W2); for Argentina GFRs 1784 (W2) and 4235 (W2); and for Honduras GFRs 1749 (W2) and 2615 (W2).