Asian Development Bank. 6 ADB Avenue, Mandaluyong City, 1550 Metro Manila, Philippines Tel +63 2 632 4444; Fax +63 2 636 2163; [email protected]; www.adb.org/evaluation

Evaluation Approach Paper Project Performance Evaluation Report for Loan 2526-PRC: Urban Transport and Environmental Improvement Project October 2018

Team Leader: Sung S. Shin, Senior Evaluation Specialist (email: [email protected]) Contact: [email protected]

A. Rationale

1. The evaluation will assess the Xinjiang Urban Transport and Environmental Improvement Project of the Asian Development Bank (ADB) in the People's Republic of (PRC) which intended to help improve and expand urban transport and sanitation infrastructure and the environment in the five fast-growing cities of Xinjiang Province.1 ADB provided a loan of $100 million to improve urban road infrastructure, traffic management and safety, and environmental sanitation in the cities of Altay, , , , and in Xinjiang Uygur Autonomous Region. The loan was approved on 26 August 2009 and became effective on 27 November 2009. ADB closed the loan account on 26 June 2015. The project was intended to complement other ADB-financed projects in Xinjiang, including the Xinjiang Municipal Infrastructure and Environmental Improvement Project, which aimed to improve road infrastructure and utility services in Alashankou land port, in the Kanas scenic region, and in City of Xinjiang.

2. The evaluation will build on the initial findings and recommendations of the ADB self- assessment and the validation of the Independent Evaluation Department (IED). In its project completion report (PCR), ADB assessed the project highly successful, based on the achieved outputs and outcomes.2 IED’s validation of the PCR assessed the project successful, in view of lower ratings on relevance and effectiveness (para. 12).

3. The findings of this evaluation will inform future ADB operations and will feed into a broader evaluation of ADB transport operations.

B. Country Context and Project Description

4. Xinjiang is in the northwest of the PRC, bordering seven countries and covering about 1.7 million square kilometers, or about one-sixth of the PRC’s total area. At the time of project preparation, Xinjiang was one of the poorest provinces in the PRC and lagged other regions in terms of economic growth. To stimulate economic growth and raise living standards in the western region including Xinjiang, the government launched the National Strategy for Development of the Western Region in 2000. In November 2008, the government further launched a fiscal stimulus package in Xinjiang that largely focused on infrastructure development. As a result, Xinjiang underwent rapid urbanization. Increasing urban population, industrialization, and economic

1 ADB. 2009. Report and Recommendation of the President to the Board of Directors: Proposed Loan to the People’s Republic of China for the Xinjiang Urban Transport and Environmental Improvement Project. Manila. 2 ADB. 2016. Completion Report: Xinjiang Urban Transport and Environmental Improvement Project in the People’s Republic of China. Manila. 2 development led to greater demand for urban infrastructure services such as roads and solid waste management.3

5. The project aimed to catalyze sustainable economic growth and improve the living standards in Xinjiang’s five fast-growing cities. It was designed to improve urban road infrastructure, traffic management and safety, and environmental sanitation in these cities. It was also designed to achieve the environmental objectives approved in the city master plans and Eleventh Five-Year Plan, 2006–2010.

6. The project had five physical components: (i) Altay Roads and environmental improvement, (ii) Changji Roads and environmental improvement, (iii) Hami Roads and environmental improvement, (iv) Kuytun Roads and environmental improvement, and (v) Turpan Roads and environmental improvement. It also had an institutional development and capacity building component to ensure effective project implementation and sustainable operation and maintenance of the project facilities.

7. The Xinjiang Uygur Autonomous Region government was the executing agency. A project leading group was established to provide overall guidance and support to project preparation and implementation. In each project city, the city government, through its construction bureau, was the implementing agency. The program was implemented over 5 years from 2009 to 2014.

8. The expected outputs were (i) Altay—16 existing roads upgraded, three new roads constructed, and five new bridges constructed; (ii) Changji—nine new roads and one new bridge constructed; (iii) Hami—three existing roads upgraded; (iv) Kuytun—13 existing roads upgraded and two new bridges constructed; and (v) Turpan—eight existing roads upgraded and six new roads constructed. New environmental sanitation facilities were also expected to be constructed, equipment provided, and institutional management capacity improved in all five cities.

9. The project was estimated to cost $187.2 million, of which $100.0 million was proposed for ADB financing and $87.2 million was for counterpart funding. At project completion, actual cost was $149.0 million, with ADB financing $97.8 million and government funding $51.2 million. The PCR reported that the project cost reduction was primarily due to savings from unused contingencies, reduced resettlement cost, and financial charges during implementation. Actual loan proceeds disbursed was $97.8 million, and the undisbursed loan proceeds of $2.2 million were cancelled.

C. Evaluation Findings

10. The PCR rated the project highly successful. It considered the project highly relevant as it was closely aligned with the government’s development strategies, ADB’s country partnership strategy, and the regional cooperation and integration strategy. It assessed the project highly effective in achieving outcomes, efficient, and highly likely to be sustainable. It noted that the project properly implemented mitigation measures as proposed in the environmental management plan, thereby minimizing potential adverse environmental impacts. The project was credited with directly benefiting about 1.65 million local people through upgraded urban road infrastructure, improved access to environmental facilities, employment opportunities, and increased income.

3 Footnote 1; and IED. 2017. Validation Report: Xinjiang Urban Transport and Environmental Improvement Project in the People’s Republic of China. Manila: ADB. 3

11. IED’s validation viewed the project successful overall. The project was rated relevant instead of highly relevant due to overestimated costs at appraisal and the absence of transformative or innovative effects in the project design or significant demonstration value for other projects. For capacity development, appointment of the international consultants charged with planning, designing, quality control of construction, training, and safeguards monitoring was delayed (43% of time elapsed), although about 200 participants were trained in 15 domestic training courses and domestic and overseas study tours on financial management, audit, procurement, contract management, traffic management, and social safeguards policy. No targets for institutional strengthening were provided in the design and monitoring framework except for a general target of improved traffic management, city planning, financial management, landscaping, and road maintenance. Effectiveness of the institutional strengthening component of the project needs further review.

12. IED’s validation confirmed most of the lessons highlighted in the PCR. The PCR recommended that the Xinjiang project management office and implementing agencies should continue to monitor the outcome indicators for 2016–2017 and report to ADB. Future projects could better articulate outcome and impact of urban development to overall inclusive economic development and environmental sustainability of the subregion. The validation also observed that more technical input should have been included in the terms of reference for the project management consultants to ensure that latest developments in highway design and traffic control technology are included in the highway components and traffic control systems provided. Improving job creation opportunities for the ethnic minority women should also have been emphasized more by the project.

D. Evaluation Objectives and Methodology

13. The PPER will assess the performance and the outcomes of the project to inform future ADB operations and feed into a broader IED evaluation of ADB transport operations. It will address key issues related to program design and theory of change (Table); and achievement of intended outcome, outputs, and their sustainability. The assessment will be based on IED’s guidelines for evaluating public sector operations using the standard criteria of relevance, effectiveness, efficiency, and sustainability.4 It will also assess the non-core evaluation criteria such as development impacts and performance of ADB and the borrower and/or executing agency. It will present the findings and identify key issues and lessons for future ADB operations.

4 ADB. 2016. Guidelines for the Evaluation of Public Sector Operations. Manila. 4

Theory of Change ADB Inputs Outputs Outcome Impact A loan of $100 million Altay Roads and Environmental Improved Sustainable from ADB’s OCR to Improvement—16 existing roads urban road socioeconomic cover civil works, upgraded, 3 new roads infrastructure, growth and equipment and constructed, and 5 new bridges traffic improved supplies, and other constructed management, living related costs; and and standards in $87.2 million Changji Roads and Environmental environmental five cities of counterpart funds from Improvement—9 new roads and 1 sanitation in Xinjiang government to cover new bridge constructed the cities of other costs, including Altay, Changji, land acquisition and Hami Roads and Environmental Hami, Kuytun, resettlement Improvement— and Turpan in 3 existing roads upgraded Xinjiang Uygur A grant of $150,000 Autonomous from Gender and Kuytun Roads and Environmental Region Development Improvement—13 existing roads Cooperation Fund to upgraded and 2 new bridges Improved improve and constructed management strengthen ethnic and marketing minority women’s Turpan Roads and Environmental ability of local economic development Improvement—8 existing roads cultural under the Turpan upgraded and 6 new roads tourismb Women’s Ethnic constructed Minority Cultural Improved Tourism Development All five cities—new environmental tourism and subprojecta sanitation facilities constructed, income of and equipment provided, and femalesb institutional management capacity improved

Skills development training, participation in social activities, and employment opportunities for womenb

Gender support and development support for local minority females and development of small enterprises based on the unique cultural heritageb ADB = Asian Development Bank, OCR = ordinary capital resources. a This is under the auspices of technical assistance for Promoting Gender Equality and Women Empowerment (TA 6143-REG). b Not included in the design and monitoring frameworks of the project’s report and recommendation of the President and project completion report, but found in Assurances and Covenants, and in the technical assistance consultant’s report of TA 6143-REG: Promoting Gender Equality and Women’s Empowerment subproject for Turpan Women’s Ethnic Minority Cultural Tourism Development. Source: Asian Development Bank. 5

14. Overarching question. To what extent has the project contributed to sustainable economic growth and improvements to the living standards in the project urban areas?

15. Sub-questions. The overarching question will be supported with the following questions:

(i) To what extent has the project contributed to the improved urban roads, traffic management, and safety? (ii) To what extent has the project empowered and strengthened the institutional capacity of the Xinjiang Uygur Autonomous Region government, city governments, and implementing agencies of project cities to ensure effective project implementation and sustainable operation and maintenance of project facilities? (iii) In what ways has the project demonstrated in encouraging women and the ethnic minorities in the area to participate in the project planning, implementation, evaluation, public awareness and education program, and on health, safety, and labor issues, including employment of women in connection with the project? (iv) To what extent has the economic benefits from the project been passed on to local people?

16. The evaluation will involve (i) review of project documents; (ii) consultations in ADB, primarily with staff at the East Asia Department; (iii) visit to the project roads; (iv) survey on traffic and road conditions; (v) data collection for evaluation through internet and interviews; (vi) discussion of the project’s relevance effectiveness, efficiency, sustainability, and impact, according to the proposed evaluation framework (Attachment), with staff from government, implementing agencies, project implementation units, and project management offices, the PRC Resident Mission, local governments, and other stakeholders including business associations and local communities. The project evaluation will rely heavily on document reviews on the components for ethnic minority women’s involvement, land acquisition and resettlement, as well as safeguards compliance.

E. Tentative Schedule and Resources

17. The evaluation will be undertaken by an evaluation team comprising Sung S. Shin (Senior Evaluation Specialist/Team Leader), Caren Joy Mongcopa (Associate Evaluation Officer), and Elizabeth Li-Mancenido (Evaluation Analyst), supported by consultants. The report will be reviewed by Tomoo Ueda (Principal Evaluation Specialist, IESP) and an external peer reviewer, Fang Xu (Senior Evaluation Officer, World Bank Independent Evaluation Group).

18. The evaluation team will review the project files and conduct consultations with the East Asia Department on the main issues associated with the formulation and implementation of the project. It is envisaged that the consulting services of one international consultant (Transport Economist/Specialist) and one national consultant (Institutional/Transport Specialist) will be needed.

19. It is proposed that an independent evaluation mission (IEM) be undertaken in the PRC around January 2019. The IEM will visit two of the five cities (Altay, Changji, Hami, Kuytun, and Turpan). The IEM will have meetings with the Xinjiang Housing and Urban-Rural Development Department; Xinjiang Finance Department; Xinjiang Development Reform Commission; staff at PRC Resident Mission; the city governments; and other relevant agencies and stakeholders.

20. The following proposed schedule is subject to government clearance of the mission and availability of suitable consultants: 6

Activity Timing Evaluation approach paper approval IV October 2018 Fielding of independent evaluation mission I–II January 2019 Circulate draft report to commenter and peer reviewer III February 2019 Submission of draft report to Director IV February 2019 Interdepartmental review of draft PPER II March 2019 Circulation of draft PPER to the government for review I–II April 2019 OSEC editing III–IV April 2019 Submission of draft final PPER to Director II May 2019 Submission to Director General, IED for approval II May 2019 Circulation of final PPER III May 2019

F. Dissemination Plan

21. The PPER will be available to the public after the approval by the Director General, IED. The report will be uploaded on ADB’s external and internal websites and will provide inputs to ADB’s evaluation information system.

Attachment: Evaluation Framework Attachment 7

EVALUATION FRAMEWORK Evaluation Indicators/ Source of Methods/Analysis Criteria Evaluation Questions Information Required Information to Be Used Relevance Was the project relevant Provincial socio- RRP, PCR, PVR, Desk review to national and/or economic background and BTORs provincial priorities? Key informant Government Minutes of Board interviews Was the project aligned development policies meetings, Staff with ADB’s policies and Review Committee Analysis of program strategies at the sector ADB’s policy, country meeting, design indicators and ADB-financed project operational strategy, Management review level? and sector strategy meetings

Was the project and Project design and Relevant program design monitoring framework government appropriate in addressing documents the envisaged impact, outcome, and outputs? Discussions with project staff, Did the project inputs, government officials, outputs, and outcomes and other key follow the logical results stakeholders chain to achieve the project’s objective? Effectiveness To what extent were Realized project RRP, PCR, PVR, Desk review project outputs, and ADB- outputs and outcomes and BTORs financed project and in relation to the Key informant sector outcomes achieved targets set Discussions with interviews as indicated in the design project staff, and monitoring Project monitoring government officials, Analysis of program framework? framework and other key design indicators Implementation stakeholders What are the factors that process contributed to the Monitoring reports of achievement or non- Issues and challenges outputs and achievement of expected related to achieving outcomes outputs and outcomes? outputs and outcomes Efficiency How well were the Loan disbursement Project documents Desk review project’s resources used and fund utilization in achieving the expected data Discussions with Key informant outcomes? project staff, interviews Implementation and government officials What were reasons for procurement and ADB Cost-benefit overestimated costs? arrangements procurement staff analysis

Monitoring data on inputs and outputs

Economic rate of return Sustainability What is the likelihood that Assessment of Discussions with Desk review project benefits would be revenue generating project staff, sustained beyond the life capacity and government officials Key informant of the project? activities interviews

Are there provisions for generating adequate revenue or funding for maintenance? 8 Attachment

Evaluation Indicators/ Source of Methods/Analysis Criteria Evaluation Questions Information Required Information to Be Used Are there any institutional issues to affect the performance of the project?

Are there any risks that may degrade the environmental protection and social benefits?

Are there any issues on resettlement after the operation?

Are there any risks that affect the sustainability of the project? Impact What are the Information on Government reports Direct observation socioeconomic impacts of beneficiaries and statistics through field visits the ADB-financed (e.g., extent of access projects? To what extent to services, livelihoods Discussions with Key informant income increased and and income earning project staff, interviews poverty reduced in the opportunities) government officials, project affected areas? stakeholders

Did the ADB-financed projects have any unintended or adverse impact on the society and economy?

Did the project comply with safeguard requirements? Are there any issues left or new issues after the completion?

Are there any positive impact on gender equality? ADB = Asian development Bank, BTOR = back-to-office report, PCR = project completion report, PVR = project completion report validation report, RRP = report and recommendation of the President. Source: Asian Development Bank Independent Evaluation Department.