Guidelines for the Economic Analysis of Investment Projects

ECA Chief Economist Office

Sectors:

 Agriculture  Education  Water And Sanitation  Energy And Power  Investment Funds (Social Or Otherwise)  Lines Of Credit  Population, Health And Nutrition  Transportation  Technical Assistance  Urban

Operational Manual: OP/OD/BP

For Peer Reviewers: Form to Evaluate the Economic Analysis in a Project

AGRICULTURE

Link to CAS and ESW

How does the operation link with the CAS? Is intervention in the sector a priority? Is the sequencing with other operations (SAL, ASAL) appropriate? Have sector studies been undertaken? Does the project fit with their conclusions? Were proposed activities identified as priorities? Do project activities integrate the lessons from previous experience in the sector (country, region or Bankwide)? Adequacy of Policy Environment Are policies in the sector generally appropriate? What are the remaining macro- or micro- economic distortions? How do they affect the project choice or its design? How does the present status of and progress in land ownership and the legal framework affect the project? Is there evidence of gender discrimination in access to services or assets? If so, what actions are planned to eliminate such discrimination? Are there any distortions that need to be eliminated (or reduced) first for the project to be fully effective? Can or should the project be used to induce appropriate policy changes? What would be the impact on the project of such policy changes? Public/Private justification Since agriculture is essentially a private activity, the public nature of the intervention should be clearly defined and, in particular, very careful scrutiny should be given to any on-farm intervention. Cost-benefit and analysis of alternatives Have alternative means of attaining the project objectives been identified and evaluated? Services: what cost effectiveness indicators (e.g., cost per farmer trained) or quantitative benchmarks (e.g., increased number of farmers served per extension agent, etc.), or other benefit indicators (e.g. from any comparable services provided by private sector) have been used to evaluate the value and cost effectiveness of the services provides? Rehabilitation: have financial and economic rates of return been estimated (e.g. based on increased output and/or changes in cost from pilot studies or experience in other localities)? Did these calculations pay careful consideration to the opportunity cost of water and other inputs? Forestry: have Cost-Benefit analysis, or cost effectiveness indicators (e.g., cost per hectare reforested), being used? Have critical externalities been included in the analysis?

LOC: see lines of credit All analysis should incorporate environmental effects either quantitatively or qualitatively Sensitivity and Risk Analysis Is there an explicit discussion of the impact of project and nonproject risks on financial and economic viability? What is the impact on NPV/ROR of varying the most critical underlying parameters (such as inputs, world prices, yields)? Is the sensitivity analysis based on alternative scenarios associated with specific risks, rather than a mechanical xx% change in costs? Is there any reasonably objective way of assigning probabilities to the main risk events? Fiscal Impact/ Cost Recovery What is the share of cost recovery for investments and for Operation and Maintenance? What is the structure of tariffs/prices for services and/or O&M? Is there an assessment of the fiscal impact of non-recovered investment and associated recurrent costs? Should there be mid-term reviews to determine if the project activities continue to merit sustaining the fiscal effort? Institutional Capacity Do the relevant Ministry and any executing agencies have adequate implementation capacity? Is institutional reform needed to achieve sustainable implementation? Is it planned? If there is a separate Project Implementation Unit, are there provision for institutionalization, or other measures to insure sustainability? Poverty analysis Is there an assessment of the impact of the project on the main poverty groups? This is required if the project is intended to benefit the poor (e.g., if the rationale for government intervention is built on equity considerations), or if it entails any actions that could significantly affect the poor (e.g., increases in water charges). Environmental Analysis Environmental analysis is generally of critical importance. Is there a comprehensive analysis of potential impacts on the environment, be they negative (e.g. from increased chemical use, additional irrigation, etc.) or positive (protective structures against soil erosion, reduced deforestation, etc.), and mitigation plans, if relevant? Full quantification of these effects may be impossible, but it may be necessary to develop some approximation the their magnitude, especially if environmental effects are critical for the project justification ((See OD 4.01 for requirements in the preparation of Environmental Data Sheet Performance Indicators Does the project define measurable performance criteria which indicate not only successful project implementation, but also its final development impact? Do they give any information on the counterfactual without the project (e.g. by monitoring control groups)? For suggestions on specific Indicators see Note from AGR (Task Manager: Cornelis de Haan) [Back to List of Sectors] Send comments or questions to ECA Project Evaluation EDUCATION

Links to CAS and ESW How does the operation link with the CAS? Is action in the sector a priority? Is the sequencing with other operations appropriate? Have sector studies been undertaken? Does the project fit with their conclusions? Were proposed activities identified as priorities? Do project activities integrate the lessons from previous experience in the sector (country, region or Bankwide)? Adequacy of Policy Environment What is the policy framework that guides resource allocation within the sector? Is there overstaffing? How are non-teacher inputs in primary education being protected? How demand responsive is post primary education? How are public resources beig allocated to make the system more demand responsive? Is there competition among schools? Are there any restrictions on entry of the private sector? Are there impediments to charging tuition? Is there equitable access to education by girls? Is there any other gender bias in the system? Can or should the project be used to induce policy changes? Public/ private justification Is there adequate discussion of the role each sector should play and why? Is the private sector allowed to participate? Cost-benefit Analysis; Alternative Designs Have alternative ways of achieving the project objectives been evaluated? E.g., should budget resources in the sector be reallocated? Are capacity increases really necessary? If so, should they focus on improving quality or expanding coverage? Is there evidence of the value of the services to be provided by the project? An effort should be made to quantify those benefits -- e.g. by using projected wage differentials or increased productivity in labor markets where there is a reasonable amount of full entry, particularly in countries where labor markets are developing in the emerging private sector (Central Europe). Additional evidence would be indicators of excess demand for the type of schooling being expanded by the project. Other non-market benefits -- such as impact on fertility, ability to cope with diseases -- should be discussed. Estimation of NPV/ERR is not required for education projects, but analysis of private and social returns is encouraged for best practice. Does the evaluation include adequate cost effectiveness analysis? This will normally involve using quantitative indicators of inputs and results (e.g., number of teachers trained, training days per staff worker, etc.). Are project activities the least cost way achieving these results? Sensitivity/Risk Analysis Is there an explicit discussion of the impact of project and nonproject risks on financial and economic viability? What is the impact on NPV/ROR of varying the most critical underlying parameters (such as inputs, demand, demographic factors)? Is the sensitivity analysis based on alternative scenarios associated with specific risks, rather than a mechanical xx% change in costs? Is there any reasonably objective way of assigning probabilities to the main risk events? Fiscal Impact/ Cost Recovery What is the fiscal impact of the project, including future recurrent costs? Are the main impacts at the local, regional or national level? Are measures for cost recovery included in the project? Would financial/fiscal impact be improved by changing some elements of the project design (e.g. loan and/or project horizon)? This is a critical element to assess the overall sustainability of the project Institutional capacity Do the relevant Ministry and any executing agencies have adequate implementation capacity? Is institutional reform needed to achieve sustainable implementation? Is it planned? If there is a separate Project Implementation Unit, will its expertise be lost when the project is completed? In addition to implementation capacity, are the relevant institutions reasonably effective in their analytical, planning and management functions? Poverty Analysis Who loses (pays) and gains as a result of the project, including any proposed cost recovery mechanisms? Are there measures to protect access by poor students? Generally each education subsector should be treated differently. A discussion of the links with poverty alleviation strategy is especially important for basic education Environmental Analysis Generally there will not be environmental implications unless new infrastructure to be built Performance Indicators Does the project define measurable performance criteria which indicate not only project implementation, but also its final outputs, i.e. learning outcomes? Is there an adequate education assessment system in the country (if no, how is this being addressed)? Do indicators measure labor market outcomes in addition to learning (e.g. in vocational training and higher education)? Do they give any information on the counterfactual without the project (e.g. by monitoring control groups)? For suggestions on specific Indicators, see Note from HDD (Sverrir Sigurdsson, TM) [Back to List of Sectors] Send comments or questions to ECA Project Evaluation

WATER AND SANITATION

Links to CAS and ESW How does the operation link with the CAS? Is action in the sector a priority? Is the sequencing with other operations appropriate? Have sector studies been undertaken? Does the project fit with their conclusions? Were proposed activities identified as priorities? Do project activities integrate the lessons from previous experience in the sector (country, region or Bankwide)? Adequacy of Policy Environment Are policies in the sector (including tariffs and other cost recovery measures) generally appropriate? What are the remaining macro- or micro-economic distortions, and how do they affect the project choice or its design? How does the project link to progress in ownership status of housing? How does it link to progress in land sales by the public sector and the possibilities of entry by private developers? Are there any distortions that need to be eliminated (or reduced) first for the project to be fully effective? Can or should the project be used to induce appropriate policy changes? What would be the impact on the project of such policy changes? Public/private justification Is there a role for the private sector -- e.g. should consideration be given to private management of utilities? Is the regulatory framework adequate? Cost-benefit or cost effectiveness; Alternative Designs What project options (alternative engineering designs, project components, size, etc.) have been identified and evaluated? Is the estimation of benefits based on careful analysis of price/demand with realistic assumptions? Are environmental and health benefits incorporated? Are there other differences between financial and economic returns to the project? Is there adequate evidence of the cost effectiveness of the project and its design (this is especially important where benefits are hard to measure, e.g., sanitation)? Sensitivity/Risk Analysis Is there an explicit discussion of the impact of project and nonproject risks on its financial and economic viability? What is the impact on NPV/ROR of varying the most critical underlying parameters (such as inputs, demand, technical effectiveness)? Is the sensitivity analysis based on alternative scenarios associated with specific risks, rather than a mechanical xx% change in costs? Is there any reasonably objective way of assigning probabilities to the main risk events? Fiscal Impact/ Cost Recovery What is the structure of tariffs/prices for services and/or O&M? What is the share of cost recovery on investments and recurrent costs? What is the net fiscal impact of the project, in its construction and operating phases? This analysis -- plus the determination of the financial viability of the public utility, if relevant -- is a critical element to assess the overall sustainability of the project. Institutional capacity Do the relevant Ministry and any executing agencies have adequate implementation capacity? Is institutional reform needed to achieve sustainable implementation? Is it planned? Is there an assessment of related institutions that have a critical bearing on the project performance (e.g. regulatory agency, sectoral ministries)? Is the structure of intergovernmental relations/incentives adequate? Poverty analysis Is there an assessment of the impact of the project on the main poverty groups? This is required if the project is intended to benefit the poor (e.g., if the rationale for government intervention is built on equity considerations), or if it entails any actions that could significantly affect the poor (e.g., increases in water charges). Any targeting should be justified with an assessment of its incidence and of the cost effectiveness of the targeting mechanism. Environmental Analysis Is there a systematic qualitative analysis of expected impacts on the environment with and without the project? Does the anaylisis assess the likely impact of the project on key variables, such as water and natural resources, or recreational facilities as the result of increased (or reduced) degradation? Where possible, the environmental impact should be quantified and incorporated into the CBA. Is resettlement an issue? Performance Indicators Does the project define measurable performance criteria which indicate not only project implementation, but also its final development impact? For suggestions on specific Indicators, see Note from TWU (TM: Guillermo Yepes) [Back to List of Sectors] Send comments or questions to ECA Project Evaluation

ENERGY AND POWER

Links to CAS and ESW How does the operation link with the CAS? Is intervention in the sector a priority? Is the sequencing with other Bank (or IFC, MIGA) operations appropriate? Have sector studies been undertaken? Does the project fit with their conclusions? Were proposed activities identified as priorities? Do project activities integrate the lessons from previous experience in the sector (country, region or Bankwide)? Adequacy of Policy Environment Are policies in the sector generally appropriate? What are the remaining macro- or micro- economic distortions, and how do they affect the project choice or its design? Are there any distortions that need to be eliminated (or reduced) first for the project to be fully effective? Are there regulatory issues, especially for natural monopolies, that need to be addressed? Can or should the project be used to induce appropriate policy changes? What would be the impact on the project of such policy changes? Public/private justification. Does the policy framework allow for some private sector participation, e.g. management of utilities, power generation? Is there a clear rationale for a public sector project? What would be the impact on private operators, if any? Cost-benefit; analysis of alternatives Have other reasonable project options (e.g., different fuel supplies, sizes, technologies, etc.) been considered and evaluated? Is the estimation of benefits based on careful analysis of price/demand with realistic assumptions? Are environmental and health benefits incorporated? Sensitivity and Risk Analysis Is there an explicit discussion of the impact of project and nonproject risks on financial and economic viability? What is the impact on NPV/ROR of varying the most critical underlying parameters (such as inputs, demand, world prices, technical efficiency)? Is the sensitivity analysis based on alternative scenarios associated with specific risks, rather than a mechanical xx% change in costs? Is there any reasonably objective way of assigning probabilities to the main risk events? Fiscal Impact/ Cost Recovery What is the structure of tariffs/prices in the sector? Are there realistic plans to move tariffs to cover full costs? Is there cross-subsidization between different users? What is the net fiscal impact of the project, in its construction and operating phases? Institutional capacity Does the relevant utility or executing agency have adequate implementation capacity? Is institutional reform needed to achieve sustainable implementation? Is it planned? Is there an assessment of related institutions that have a critical bearing on the project performance (e.g. regulatory agency, sectoral ministries)? Poverty analysis Is there an assessment of the impact of the project on the main poverty groups? This is required if the project is intended to benefit the poor (e.g., if the rationale for government intervention is built on equity considerations), or if it entails any actions that could significantly affect the poor (e.g., large increases in tariffs). Any targeting should be justified with an assessment of its incidence and of the cost effectiveness of the targeting mechanism. Environmental Analysis Since environmental effects can be a critical element of the project, have they been systematically considered, and accounted for, in the project design? E.g., does the analysis assess the likely impact of the project on key variables, such as human health, forest and crops, visibility, etc? Has the full cost of mitigation measures and compliance with environmental standards been included in the CBA? Is resettlement an issue? Performance Indicators Does the project define a set of measurable performance indicators, selected in consultation with the client, and to be reviewed consistently during project implementation? For suggestions on specific Indicators, see Notes from IEN (TMs: Jean-Pierre Charpentier (Power), and William Porter (Oil and Gas)).) [Back to List of Sectors] Send comments or questions to ECA Project Evaluation INVESTMENT FUNDS (SOCIAL OR OTHERWISE)

Links to CAS and ESW How does the operation link with the CAS? Is intervention in the sector a priority? Is the sequencing with other operations appropriate? Has there been a poverty assessment? Has the safety net been assessed? How does the project fit with the analysis and recommendations from these assessments? Do they integrate the lessons from previous experience in the sector (country, region or Bankwide)? Adequacy of Policy Environment Are policies in the sectors where investments are likely to be financed (health, education, water or electricity, etc.) generally appropriate? What are the remaining macro- or micro-economic distortions, and how do they affect the project choice or its design? Are there any distortions that need to be eliminated (or reduced) first for the project to be fully effective? Can or should the project be used to induce appropriate policy changes? What would be the impact on the project of such policy changes? Public/ private justification If no subsidy element, relatively unimportant. Cost-benefit or cost-effectiveness; analysis of alternatives Have different ways of achieving the project objectives been considered and evaluated-- e.g., traditional public works programs, food for work programs, cash transfer programs, food stamps, feeding programs, Grameen Bank-type credit programs? Are there adequate selection criteria for activities and projects to be financed by the Fund-- cut-off ROR, or cost effectiveness standards? Sensitivity and Risk Analysis Is there an explicit discussion of the impact of project and nonproject risks on financial and economic viability? Is there an assessment of the risk of failure on the activities to be financed by the Fund (e.g. from experience in other countries)? Do selection criteria allow for consideration of the tradeoff between risk and returns in these activities? How is gender bias in project selection being avoided? Fiscal Impact/ Cost Recovery What is the fiscal cost of running the program, including potential defaults? Institutional capacity Does the executing agency have adequate implementation capacity? Is there an assessment of related institutions that have a critical bearing on the project performance (e.g. regulatory agency, Central Bank, local government )? Is institutional reform needed to achieve the project objectives? Is it planned? Is there an analysis of strengths and weaknesses of the local institutions which will made the actual decisions on activities to be financed by the project? Poverty analysis Typically this is very important: there should be a determination of the incidence of the activities financed by the Fund, as well as an assessment of the cost effectiveness of any targeting mechanism. What is the "leakage" to non target groups? Environmental Analysis Are the criteria for project selection adequate in their inclusion of environmental concerns (e.g. are there safeguards to penalize activities damaging the environment)? Are the managing institutions, or other environmental organizations, capable of evaluating and monitoring the environmental performance of the Fund? Performance Indicators Does the project define measurable performance criteria which indicate not only project implementation, but also its final development impact? Are they reasonably straightforward to calculate and interpret? For suggestions on specific Indicators, see Note from FSD (TMs: David Scott and Monika Queisser) [Back to List of Sectors] Send comments or questions to ECA Project Evaluation

LINES OF CREDIT / FILS

Links to CAS and ESW Is this operation a priority for Bank intervention? Is the sequencing with other IFC and Bank operations (SAL, FESAL, SECAL, etc.) appropriate? Is lack of credit (rather than low demand due to macro instability, policy uncertainty, implicit taxation of the sector, or lack of a sufficient legal framework) the constraint on investment? Do project activities integrate the lessons from previous experience in the sector (country, region or Bankwide)? Adequacy of Policy Environment Macroeconomic: Is the exchange rate overvalued? Is there significant capital flight or is a substantial share of savings going to finance the fiscal deficit (in other words, because of fungibility, will a large part of the credit line go to other uses?) How conducive is the macroeconomic environment for financial decisions, especially with high and/or rising inflation? Sector: Are there major price distortions? Are we convinced that our funds will end up in sectors where the country has a true comparative advantage? Is it likely that future price liberalization could undermine credit decisions made under the current price structure? Financial: Are interest rates market determined? If some are not, what proportion of credit is subject to non-market rates? Is there a program in place to address the non-market determined interest rates? (see OD 8.30 for pre-conditions on interest rates). Are policies regarding laws, financial restructuring, capital market development etc. adequate? Is there credit rationing between sectors or among beneficiaries, either generally or through the proposed operation (this could introduce subjective allocation measures)? Does the cost of capital under the proposed credit line reflect the price the market will bear (i.e., competition among beneficiaries)? Are on-lending arrangements satisfactory? Is credit allocation gender-neutral? If there is evidence of discrimination, how is it being addressed? Can or should the project be used to induce appropriate policy changes? What would be the impact on the project of such policy changes? Public/ private sector What is the justification for government financing in credit markets (e.g., subsidies, guarantees, etc.)? Are banks independent enough from the government to intermediate funds purely according to commercial criteria? Are the regulatory framework, supervisory capacity, bankruptcy law, collateral law, etc. adequate? Cost-benefit or cost effectiveness; Analysis of alternatives Have other options to increase flow of credit to intended beneficiaries been considered e.g., measures to reduce risk (insurance, price hedging, guarantee operations), improve collateral (land titling, collateral laws), reduce transactions costs (technology), TA to beneficiaries or bankers, measures to increase profitability of subprojects (elimination of price/trade distortions, etc.), transactions subsidies, to financial intermediaries, etc.? Is it clear who will bear the foreign exchange risk and why? Is the proposed mechanism for allocating this risk logical? Is it likely to be accepted? Where projects involve large external economies or diseconomies, or where major price distortions affect the inputs and outputs of potential investments, ERRs should be estimated to identify if financing for certain investments should be excluded (i.e., where ERRs are expected to be well below FRRs). Is there an analysis of the cost effectiveness of any implicit or explicit subsidy (including guarantees, interest rate and loan default coverage, technical assistance)? Demand for funds should be estimated. Supply of funds from other sources should be assessed to ensure that LOC is not displacing private capital. Is the LOC expected to increase investment by intended beneficiaries? Will the project reduce the marginal cost of credit, i.e., can beneficiaries borrow as much as they want at a lower cost? If only limited amounts are available at the lower rate, there will simply be an income transfer with no effect on investment levels. Sensitivity and Risk Analysis Is there an explicit discussion of the impact of project and nonproject risks on financial and economic viability? Although conventional sensitivity analysis is not likely to be relevant, there should be an assessment of the impact on the project of potential risk factors (e.g., variations on the demand for loans, foreign exchange risk, financial sector crisis, regulatory or policy failures). Fiscal Impact/ Cost Recovery What is the net fiscal impact of the project? The assessment should include not only the direct fiscal cost of any TA and institutional development component, but also any subsidy element on interest or default coverage, or indirect subsidies through concessional financing sources or preferential rediscounting, etc.; as well as the budget impact of financial sector reforms associated with the project (e.g., reduction of subsidies, increase in cost of servicing public debt, cost of recapitalizations, etc.). Institutional capacity Is there an adequate assessment of the financial capacity of participating lending institutions (financial performance, loan collections, capital adequacy, resource mobilization) and the institutional capacity of lending institutions (managerial autonomy, data processing, staff skills, etc.) other relevant institutional arrangements (e.g., land titles).? Are any corrective measures being taken? Poverty analysis Is there an assessment of the impact of the project on the main poverty groups? This is required if the project is intended to benefit the poor (e.g., if the rationale for government intervention is built on equity considerations), or if it entails any actions that could significantly affect the poor (e.g., subsidies, guarantees). Are any poverty-oriented measures in the project grounded on a poverty assessment and adequately targeted? Is there a need for special actions to achieve gender-equity? Environmental Analysis Although a specific environmental analysis generally is not needed, is there an assessment of the quality of environmental regulations in the country and their consistency with the Bank's environmental guidelines, as well as the extent to which they are adhered to by financial intermediaries in appraising subprojects? Performance Indicators Does the project define measurable performance criteria which indicate not only project implementation, but also its final development impact? Are they reasonably straightforward to calculate and interpret? For suggestions on specific Indicators, see Note from FSD (TMs: David Scott and Monika Queisser) [Back to List of Sectors] Send comments or questions to ECA Project Evaluation

POPULATION, HEALTH AND NUTRITION

Links to CAS and ESW How does the operation link with the CAS? Is intervention in the sector a priority? Is the sequencing with other operations appropriate? Have sector studies been undertaken? Does the project fit with their conclusions? Were proposed activities identified as priorities? Do project activities integrate the lessons from previous experience in the sector (country, region or Bankwide)? Adequacy of Policy Environment Are policies in the sector generally appropriate? What are the remaining macro- or micro- economic distortions, and how do they affect the project choice or its design? Is the structure of intergovernmental relations/incentives appropriate? Are there any distortions that need to be eliminated (or reduced) first for the project to be fully effective? Can or should the project be used to induce appropriate policy changes? What would be the impact on the project of such policy changes? Public/private justification What role should each sector play and why? Indicate plans for expanding private sector participation, if available. Cost-effectiveness; analysis of alternatives Have a full range of alternative approaches for improving health status been considered and evaluated? Have design choices related to process, time and size been fully explored -- such as "by whom", "when", "how well" and "big" versus "small"? How much of the desired expansion of services can be achieved by reallocating resources rather than by expanding capacity? Should the emphasis be on changing the quality of the service or on increasing its quantity? Have all relevant costs, effects and benefits been quantified? Is there an adequate assessment of cost effectiveness (or cost-utility analysis), e.g. by comparing options in terms of their impact on DALYs (Disability Adjusted Life Years)? Or is the design based on international 'best practice' information (and cited in project documents)? Have gender dimensions been taken into account in the analysis? E.g., in ECA countries abortion rates are particularly high. Is there a project subcomponent to specifically improve safe family planning? Sensitivity and Risk Analysis Is there an explicit discussion of the impact of project and nonproject risks on financial and economic viability? What is the impact on NPV/ROR of varying the most critical underlying parameters (such as inputs, demand, effectiveness of treatments)? Is the sensitivity analysis based on alternative scenarios associated with specific risks, rather than a mechanical xx% change in costs? Is there any reasonably objective way of assigning probabilities to the main risk events? Fiscal Impact/ Cost Recovery What is the net fiscal impact of the project, in its initial implementation and regular operating phases? Are recurrent cost implications properly included? Is there an assessment of the budgets (e.g., appropriate sector and subsectoral composition; capital versus operation), including the project costs, and at the relevant level (local, region, country)? Is there an analysis of private affordability? Are proposed direct charges and cost recovery mechanisms consistent with the public/private justification, affordability discussion and poverty implications? This analysis is a critical element to assess the overall sustainability of the project Institutional capacity Do the relevant Ministry and executing agencies have adequate implementation capacity? Is institutional reform needed to achieve sustainable implementation? Is it planned? In addition to implementation capacity, are the relevant institutions reasonably effective in analytical, planning and management functions critical to the success of the project? Poverty analysis Is there an assessment of the impact of the project on the main poverty groups? This is required if the project is intended to benefit the poor (e.g., if the rationale for government intervention is built on equity considerations), or if it entails any actions that could significantly affect the poor (e.g., significant increase in user charges). Is there a clear link with the poverty alleviation strategy, especially for primary health? Has there been a full analysis of the tradeoffs between access, cost and quality? Are poverty-oriented measures adequately targeted? Is the targeting justified with an assessment of its incidence and of the cost effectiveness of the targeting mechanism? Environmental Analysis Generally there will not be environmental implications, unless new infrastructure to be built. Performance Indicators Does the project define measurable performance criteria which indicate not only project implementation, but also its final development impact? Do they give any information on the counterfactual without the project (e.g. by monitoring control groups)? For suggestions on specific Indicators, see Note from HDD (TM: Jorge Barrientos) For additional discusssion see: Economic Analysis of Population, Health and Nutrition Sectors, by Alexander S. Preker.

[Back to List of Sectors]

Send comments or questions to ECA Project Evaluation

TRANSPORTATION

Links to CAS and ESW How does the operation link with the CAS? Is intervention in the sector a priority? Is the sequencing with other Bank (and IFC and MIGA) operations appropriate? Have sector studies been undertaken? Does the project fit with their conclusions? Were proposed activities identified as priorities? Do project activities integrate the lessons from previous experience in the sector (country, region or Bankwide)? Adequacy of Policy Environment Are policies in the sector generally appropriate? How will any existing distortions in markets relevant to the project -- e.g. underpriced energy -- have an impact on project activities? Are there other macro- or micro-economic distortions, and if so how do they affect the project choice or its design? Are there any distortions that need to be eliminated (or reduced) first for the project to be fully effective? Can or should the project be used to induce appropriate policy changes? What would be the impact on the project of such policy changes? Public/private justification Are sector activities in which an expanded private sector participation should be encouraged? Is there a clear justification for any proposed government intervention? Cost-benefit or cost effectiveness; analysis of alternatives Have alternative design options -- e.g. intermodal options; high vs. low traffic, alternative technologies -- been examined and evaluated? Has the project been properly separated into subcomponents? Is the estimation of benefits based on careful analysis of price/demand with realistic assumptions? Are the assumptions and data used (e.g. patronage or traffic created or diverted, value of travel time saved, operating costs, etc.) clearly stated and justified? Are there significant distortions (e.g. in fuel prices) creating a gap between financial and economic returns? Are project activities expected to have a significant effect on the level or seriousness of road accidents? On congestion? On air quality? If it is not possible to calculate NPV and rates of return, is there at least an analysis of cost effectiveness using such quantitative measures as cost per vehicle mile, etc.? Has the HDM been used to estimate costs savings? Sensitivity and Risk Analysis Is there an explicit discussion of the impact of project and nonproject risks on financial and economic viability? What is the impact on NPV/ROR of varying the most critical underlying parameters (such as traffic/patronage estimates, investment and operating costs, )? Is the sensitivity analysis based on alternative scenarios associated with specific risks, rather than a mechanical xx% change in costs? Is there any reasonably objective way of assigning probabilities to the main risk events? Fiscal Impact/ Cost Recovery What is the structure of tariffs/prices for services? What is the share of cost recovery on investments and on Operation and Maintenance? What is the net fiscal impact of non- recovered investment and associated recurrent costs? Institutional capacity Do the relevant Ministry and any executing agencies have adequate implementation capacity? Is institutional reform needed to achieve sustainable implementation? Is it planned? If there is a separate Project Implementation Unit,will its expertise be lost when the project is completed? Is there an assessment of related institutions that have a critical bearing on the project performance (e.g. regulatory agencies)? Poverty analysis Is there an assessment of the impact of the project on main poverty groups? This is required if the project is intended to benefit the poor (e.g., if the rationale for government intervention is built on equity considerations), or if it entails any actions that could significantly affect the poor (e.g., large increase in tariffs). Any targeting should be justified with an assessment of its incidence and of the cost effectiveness of the targeting mechanism. Environmental Analysis Is there a systematic qualitative analysis of expected impacts on the environment with and without the project? Where possible, the environmental impact should be quantified and incorporated into the CBA. Is resettlement an issue? Performance Indicators Does the project define measurable performance criteria which indicate not only project implementation, but also its final development impact? Are they reasonably straightforward to calculate and interpret? For suggestions on specific Indicators, see Note from TWU (TM: Colin Gannon) [Back to List of Sectors] Send comments or questions to ECA Project Evaluation

TECHNICAL ASSISTANCE

Links to CAS and ESW How does the operation link with the CAS? Is intervention in the sector a priority? Is the sequencing with other operations appropriate? Is there adequate coordination with other providers of TA? What is the background work suggesting the need for this specific TA? Does the project integrate the lessons from previous experience in the sector (country, region or Bankwide)? Adequacy of Policy Environment Link between policy reform and TA should be discussed in both directions depending on the specific intervention: to what extent are specific policy changes required for that TA to have an effect? to what extent will the domestic capability to identify policy options be enhanced by TA? Is there evidence of gender discrimination in government-supported training programs? How is it prevented under Bank projects? Public/private justification Why should government be involved? What is the justification for public provision of information and skills to private sector? Would changes in incentive framework inspire proposed beneficiaries to obtain information/training themselves? What is the private sector counterfactual? Cost-benefit or cost effectiveness; analysis of alternatives Is the final output of the TA clearly identified? Is possible to estimate its value? E.g. a common objective would be to allow the recipient of the TA to provide a better product or service; thus, the final output of the TA would be measured by indicators of the increase in quality (improvements on tax collection by TA to tax authorities, lower intermediation costs by TA to banks, better and faster privatization and land reform by TA to privatization agencies, etc.). Is there an adequate description of activities under the TA (e.g. number of people to be trained, training time, providers), as well as a discussion of their cost effectiveness (e.g. compared with other ways of providing similar assistance)? Sensitivity and Risk Analysis Is there an explicit discussion of the impact of project and nonproject risks on financial and economic viability? Fiscal Impact/ Cost Recovery Should TA be provided on a fee for service basis? What is degree of cost recovery anticipated ? What is fiscal impact of non- recoverable portion? Institutional capacity Do the relevant Ministry, any executing agencies and service providers have adequate implementation capacity? Is institutional reform needed to achieve sustainable implementation? Is it planned? Is there an appropriate use of local vs. expatriate staff? Poverty analysis Not relevant unless TA directed toward services to be targeted to the poor. Environmental Analysis Not likely to be relevant unless the project is directed at environmentally-related activities Performance Indicators Does the project define measurable performance criteria which indicate not only project implementation, but also its final development impact? Do they give any information on the counterfactual without the project (e.g. by monitoring control groups)? For suggestions on specific Indicators, see Note from OPR (TM: Nimrod Raphaeli) Additional Guidelines: World Bank, Operations Policy Department, (1993), Handbook on Technical Assistance

Send comments or questions to ECA Project Evaluation

Links to CAS and ESW How does the operation link with the CAS? Is intervention in the sector a priority? Is the sequencing with other operations appropriate? Have sector studies been undertaken? Does the project fit with their conclusions? Were proposed activities identified as priorities? Do project activities integrate the lessons from previous experience in the sector (country, region or Bankwide)? Adequacy of Policy Environment Are policies in the sector generally appropriate? What are the remaining macro- or micro- economic distortions, and how do they affect the project choice or its design? What is the pace of privatization in the sector (retail and housing)? Is public land being auctioned? We need a minimum progress in this area prior to the project Are there other distortions that need to be eliminated (or reduced) first for the project to be fully effective? Can or should the project be used to induce appropriate policy changes? What would be the impact on the project of such policy changes? Public/ private sector Very important to articulate why there is a case for a public project, particularly for municipal services and housing. Is the regulatory framework for private operators adequate? Cost-benefit or cost effectiveness; alternatives What alternative designs have been considered and evaluated? This is more important for urban infrastructure; less relevant for municipal finance or urban management. Is the estimation of benefits based on careful analysis of price/demand with realistic assumptions? Are environmental and health benefits incorporated? Is there adequate evidence of the cost effectiveness of the project and its design (this is especially important where benefits are hard to measure)? Sensitivity and Risk Analysis Is there an explicit discussion of the impact of project and nonproject risks on financial and economic viability? What is the impact on NPV/ROR of varying the most critical underlying parameters (such as demand, construction/operating costs, etc.)? Is the sensitivity analysis based on alternative scenarios associated with specific risks, rather than a mechanical xx% change in costs? Is there any reasonably objective way of assigning probabilities to the main risk events? Fiscal Impact/ Cost Recovery What is the structure of tariffs for municipal services? What is the share of cost recovery for investments and for Operation and Maintenance? What is the fiscal impact of non- recovered investment and associated recurrent costs? This is a critical element to assess the overall sustainability of the project Institutional capacity Do the relevant Ministry, executing agencies and service providers have adequate implementation capacity? Is institutional reform needed to achieve sustainable implementation? Is it planned? Is there an assessment of related institutions that have a critical bearing on the project performance (e.g. regulatory agency, sectoral ministries)? Is the structure of relations and incentives between local, regional and central governments an issue? Poverty analysis Is there an assessment of the impact of the project on the main poverty groups? This is required if the project is intended to benefit the poor (e.g., if the rationale for government intervention is built on equity considerations), or if it entails any actions that could significantly affect the poor (e.g., large increases in user charges). Any targeting should be justified with an assessment of its incidence and of the cost effectiveness of the targeting mechanism. Environmental Analysis Is there a systematic qualitative analysis of expected impacts on the environment with and without the project? Where possible, the environmental impact should be quantified and incorporated into the CBA. Is resettlement an issue? Performance Indicators Does the project define measurable performance criteria which indicate not only project implementation, but also its final development impact? Do they give any information on the counterfactual without the project (e.g. by monitoring control groups)? For suggestions on specific Indicators, see Note from TWU (TM: Patricia Annez)) Additional references: for Urban Transport projects, seeTransport Sector.

Send comments or questions to ECA Project Evaluation or to R. Martin

May 1996