Validation Report August 2020

Azerbaijan: Water Supply and Sanitation Investment Program (Multitranche Financing Facility and Tranche 3)

Reference Number: PVR-691 Project Number: 42408-013 and 42408-043 MFF Number: 0032 Loan Number: 3079

ABBREVIATIONS

ADB – Asian Development Bank Azersu – Azersu Joint Stock Company DMF – design and monitoring framework EIRR – economic internal rate of return EMP – environmental management plan FIRR – financial internal rate of return GAP – gender action plan IEE – initial environmental examination IPMC – international project management consultant MFF – multitranche financing facility PCR – project completion report PMO – project management office PPER – project performance evaluation report RRP – report and recommendations of the President SAWMC – State Amelioration and Water Management Committee TWUA – town water users’ association WSS – water supply and sanitation WWTP – wastewater treatment plant

NOTE

In this report, “$” refers to United States dollars.

Director General Marvin Taylor-Dormond, Independent Evaluation Department (IED) Deputy Director General Veronique Salze-Lozac’h, IED Director Nathan Subramaniam, Sector and Project Division (IESP) Team Leader Srinivasan Palle Venkata, Senior Evaluation Specialist, IESP

The guidelines formally adopted by the Independent Evaluation Department (IED) on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of IED management, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. The final ratings are the ratings of IED and may or may not coincide with those originally proposed by the consultants engaged for this report.

In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, IED does not intend to make any judgments as to the legal or other status of any territory or area.

PROJECT BASIC DATA Project number 42408-033 PCR circulation date 31 October 2019 Loan number 3079 PCR validation date Aug 2020 Program name Water Supply and Sanitation Investment Program (Multitranche Financing Facility and Tranche 3) Sector and subsector Water and other Urban policy, institutional and capacity development urban infrastructure Urban sewerage and services Urban water supply Strategic agenda Environmentally sustainable growth Inclusive economic growth Safeguard categories Environment B Involuntary resettlement B Indigenous peoples C Country Republic of Approved Actual ($ million) ($ million) ADB financing ADF: 0.00 Total project costs 813.39 663.56 ($ million) OCR:600.00 Loan2571 75.00 74.20 2842 300.00 299.71 3079 150.00 149.02 Project 4 75.00 0.00 Borrower 2571 25.00 21.37 2842 110.00 78.51 3079 53.39 40.79 Project 4 25.00 0.00 Beneficiaries 0.00 0.00 Others 0.00 0.00 Cofinancier Total cofinancing 0.00 0.00 Approval Date Effectiveness 2571 14 Oct 2009 date2571 16 Jan 2010 17 Feb 2010 2842 22 Dec 2011 2842 6 Jul 2012 20 Sep 2012 3079 5 Dec 2013 3079 26 Feb 2015 28 May 2015 Signing Date Closing date 2571 17 Dec 2009 2571 30 Jun 2013 8 Nov 2017 2842 6 Jun 2012 2842 30 Jun 2017 4 May 2018 3079 28 Nov 2014 3079 31 May 2018 10 Dec 2018 Project Officers Location From To L. Zheng ADB HQ Oct 2009 Mar 2010 V. Padmanabhan ADB HQ Apr 2010 Feb 2013 S. Joshi ADB HQ Feb 2013 Mar 2014 L. Shen ADB HQ Apr 2014 Jan 2017 F. Huseynbeyov AZRM Jan 2015 Nov 2017 S. Jafarova AZRM Jan 2017 May 2018 IED review Director N. Subramaniam, IESP Team Leader Srinivasan Palle Venkata, Senior Evaluation Specialist, IESP* ADB = Asian Development Bank, ADF = Asian Development Fund, AZRM = Azerbaijan Resident Mission, HQ = headquarters, IED = Independent Evaluation Department, IESP = Sector and Project Division, OCR = ordinary capital resources, PCR = project completion report. *Team members: H. Hettige (Quality Reviewer), F. De Guzman (Senior Evaluation Officer), W. Zhou and D. Gibson (Consultants).

2

I. PROJECT DESCRIPTION A. Rationale

1. Azerbaijan’s water supply and sanitation (WSS) system was old and generally in a poor state of repair. Secondary towns faced particularly critical situations characterized by missing or dysfunctional water treatment facilities and inappropriately sized and dilapidated distribution network. As a result, most households had no access to safe drinking water, despite a piped- network coverage rate of 83%. The sewerage system in secondary towns was also in poor condition, with a mere 32% coverage in urban areas other than . Poor sanitation and leaking sewers created serious health risks and environmental hazards.

2. With improving access to basic services identified as a strategic priority,1 the government set targets of 100% coverage by 2015 and 24-hour water supply in Baku, and 80%–85% coverage in secondary towns and villages. In view of the magnitude of required investments in WSS improvements and the scale of related policy reforms, the government recognized the need for sequenced interventions linked with reforms for improving WSS service delivery in secondary towns and requested ADB to finance part of its investment program.

3. ADB approved a multitranche financing facility (MFF) in September 2009 for the WSS Investment Program up to $600 million from ADB's ordinary capital resources.2 The program aimed to improve public health and environment in the participating towns with about 500,000 residents, through enhanced coverage, continuity and quality of water supply and wastewater disposal. It proposed five tranches, and adopted a long-term and holistic approach to improve facilities, institutions, and capacities, which would collectively contribute to enhanced service delivery. The use of MFF modality was justified as it allowed an 8-year partnership between ADB and the government. A combination of interrelated physical and non-physical investments in a phased manner was backed by a clear strategy and policy framework. An MFF also had lower transaction costs and enhanced the possibility for a long-term partnership more than in a stand- alone project, and for relatively large investments as compared to a sector loan. More so, it had broader coverage than a program loan, which could only focus on policy reforms.3

4. Following the approval of tranche 1 in 2009 and tranche 2 in 2011, tranche 3 under the investment program was approved in December 2013 for $150 million. It aimed to improve the WSS infrastructure in Agjabedi and the peri-urban areas of ,4 benefiting over 70,000 residents. ADB prepared three project completion reports (PCRs), two tranche-specific PCRs and one PCR for the facility and tranche 3.5 The latter is the subject of this validation. Two PCRs for tranches 1 and 2 were validated separately.6

1 Government of Azerbaijan. 2008. State Program on Poverty Reduction and Sustainable Development in the Republic of Azerbaijan for 2008–2015. Baku; and Government of Azerbaijan. 2009. State Program on Socio-Economic Development of Regions of the Republic of Azerbaijan for 2009–2013 Years. Baku. 2 ADB. 2009. Report and Recommendation of the President to the Board of Directors: Proposed Multitranche Financing Facility to the Republic of Azerbaijan for the Water Supply and Sanitation Investment Program. Manila. 3 ADB. 2019. ADB's Multitranche Financing Facility, 2005–2018: Performance and Results Delivered. Manila. 4 The peri-urban areas covered by this tranche included Qaracuq, Qarahanbeyli, Bulgan, Haciniyyet and Tumbul, located south and southwest of Nakhchivan city. 5 ADB. 2019. Completion Report. Azerbaijan: Water Supply and Sanitation Investment Program (Tranche 1). Manila; ADB. 2019. Completion Report. Azerbaijan: Water Supply and Sanitation Investment Program (Tranche 2). Manila; and ADB. 2019. Completion Report. Azerbaijan: Water Supply and Sanitation Investment Program (Multitranche Financing Facility and Tranche 3). Manila 6 Independent Evaluation Department. 2020. Validation Report: Water Supply and Sanitation Investment Program in Azerbaijan (Tranche 1). Manila: ADB; and Independent Evaluation Department. 2020. Validation Report: Water Supply and Sanitation Investment Program in Azerbaijan (Tranche 2). Manila: ADB. 3

B. Expected Impacts, Outcomes, and Outputs

5. As stated in its design and monitoring framework (DMF) and the PCR, the investment program’s expected impact was improved public health and environment in participating towns. Its intended outcome was improved coverage, continuity and quality of water supply and wastewater disposal in participating towns. The outputs underpinning its outcome were in three components: WSS infrastructure development, institutional effectiveness, and establishment of a program management facility. Component 1 consisted of four parts involving rehabilitation, replacement and expansion. The first two parts dealt with piped-water supply systems, water works, and treatment facilities and sewerage networks. The third was concerned with developing and implementing a water quality monitoring system, while the fourth with constructing new wastewater treatment plants (WWTPs). Component 2 had six parts: (i) improving institutions for service delivery, (ii) developing capacity for better system planning, operation and maintenance, (iii) enhancing cost recovery through metering, billing and collecting tariffs, (iv) increasing efficiency for timely processing of effective WSS projects, (v) finalizing twinning arrangement with an external WSS utility to support capacity development, and (vi) mainstreaming consumers on program implementation. The expected impact and outcome of tranche 3 was fully aligned with the investment program, with planned outputs focusing on Agjabedi and Nakhchivan's peri-urban areas.

C. Provision of Inputs

6. The MFF was approved in September 2009 with an expected availability period until the end of May 2018. The actual completion was in May 2018, with no extension. The indicative financing plan in the report and recommendations of the President (RRP) originally listed five tranches. However, tranches 2 and 3 in the list were combined as one tranche (tranche 2) due to a 1.5-year delay in implementation (footnote 5). Tranche 1 was approved in October 2009 and closed in June 2017, 4 years later than expected, due to design change and delay of civil works in . Tranche 2 was approved in December 2011 with expected closing in June 2017. The actual closing was 6 months later due to delays in equipment delivery and construction of Goychay WWTP. Tranche 3 was approved in December 2013, signed in November 2014, made effective in May 2015 and closed in May 2018 as scheduled. The delay in loan signing was due to delays in getting authorization from government ministries for loan agreement signatories. Tranche 4 was approved in August 2016 but remained unsigned until the expiry of the approval validity in August 2017. The appraised cost of the program was $600.00 million, or close to 75% of the total cost. 7 At completion, the actual amount of ADB loan disbursement was $522.93 million, or 79% of the actual total project cost. The considerable difference between actual disbursement and the approved amount was mainly due to the cancellation of tranche 4.

7. Tranche 1. At completion, the civil works contracts increased from the estimate of $58.08 million to $67.86 million due to substantial cost overruns in Goychay's WSS networks. The actual expenditure for equipment and materials was $0.22 million, lower than the planned $2.05 million to cover WSS network maintenance for Nakhchivan. The consulting service contracts for design and construction supervision increased from $7.65 million at appraisal to $10.61 million. These increases were covered by reallocation of loan funds between categories, including contingencies and interest during construction. At completion, the disbursement of ADB loan proceeds was $74.20 million against the planned $75.00 million.

7 The total project cost at appraisal was supposed to be $800.00 million, of which $600.00 million will be covered by ADB and $200.00 million by the government (footnote 2, p. 11, tables 2 and 3). The approved total project cost was $813.39 million, $600.00 million from ADB and $213.39 million from government’s own contribution. 4

8. Tranche 2. At completion, the civil works contracts increased to $358.56 million against estimated $323.33 million as a result of cost overruns in the Agdash, and Goychay WSS construction contracts. These were a result of changes in technical parameters, time delay in civil works implementation, and significant expansion of Agdash’s administrative area that necessitated a redesign of WSS network and water intake facilities. The consulting service contracts at completion was $9.30 million, showing an increase of 90% over planned cost due to the redesigned payments for the water intake facilities in Agdash and the increased scope of construction supervision for the expanded WSS network. At completion, the disbursement of ADB loan proceeds was $299.71 million, lower than the planned $300.00 million.

9. Tranche 3. At completion, the civil works totaled $179.90 million, higher than the $170.52 million estimated at appraisal due to the change in scope. The Agjabedi WWTP was excluded due to protracted discussion between ADB and Azersu Joint Stock Company (Azersu) about the necessity of biological treatment (para. 12). The WSS network construction in Shakarabad and sewerage collectors in Agdash were included in tranche 3. The consulting service contracts for design and construction supervision was $5.71 million, which was higher than the $4.92 million planned at appraisal and reflecting the additional supervision for the expanded works. At completion, the disbursement of ADB loan proceeds was $149.02 million lower than the planned $150.00 million.

10. International project management consultants (IPMCs) were recruited using quality- and cost-based selection. Azersu’s IPMC provided design and supervision consulting services for all tranches, which led to contract cost escalation and left tranche 1 open despite the completion of all major works. The IPMC for State Amelioration and Water Management Committee (SAWMC) was recruited independently for each tranche. A total of 13 civil works contracts for WSS networks and WWTPs were procured, through international competitive bidding following ADB's single- stage, one-envelope procedure with post-qualification.

11. The program and tranche 3 were classified as category B for both environment and involuntary resettlement, and category C for indigenous people. The program was categorized as effective gender mainstreaming. Its gender action plan (GAP) included doing a gender analysis, consultations, and surveys to determine the specific needs of women and men in relation to WSS, as well as the development of tranche-specific GAPs. Also, it envisaged the conduct of gender awareness workshops for executing agencies and the participation of women in town water users’ associations (TWUAs) and in public outreach programs to provide feedback. The GAP for tranche 3 consisted of ten activities and eight targets.

D. Implementation Arrangements

12. As planned, Azersu was the executing agency for Agdash, Agjabedi, Beylagan and Goychay subprojects, and SAWMC was the executing agency for Nakhchivan subproject. The project management offices (PMOs) established at Azersu and SAWMC with assistance from the IPMCs were responsible for a range of duties.8 Azersu abolished its PMO in January 2015 as part of its efforts to streamline the implementation of all investment projects by forming a new corporate unit for the administration of facilities under construction. The PCRs for tranches 1, 2 and 3 noted that this arrangement adversely affected civil works implementation with frequent and significant changes in the construction plan, substantial cost overruns and delays. The PCR

8 The duties were: submitting periodic financing requests, undertaking due diligence for subsequent projects, preparing detailed design for future investments, supervising procurement and construction, overseeing safeguards and gender compliance, monitoring and evaluating program progress and preparing reports, among other things. 5

for tranche 3 also noted that the discussion between ADB and Azersu on the need for biological treatment for the WWTPs and compliance with ADB’s Safeguard Policy Statement were significantly hindered by Azersu, leading to the exclusion of Agjabedi WWTP from the procurement plan of tranche 3. The requirement for the establishment of (i) a program steering committee to provide policy direction and oversee project progress and (ii) project implementation review committees to review monthly progress and resolve operational issues was waived at the government’s request in early 2012.9

13. The executing agencies complied with all framework financing agreement covenants, except for partial compliance with tariff levels and payment recovery, gender compliance, training for TWUAs, and submission of annual progress reports. The WSS tariffs were not revised annually, as the Tariff Council revised the tariffs last in 2016. The GAP was not effectively implemented due partly to the inadequate formulation of some indicators at the design stage and the lack of ownership by the executing agencies (para. 25). Lack of funding led to failure to meet TWUA-related covenants. The executing agencies did not meet the requirements for the submission of annual reports on the performance monitoring indictors. For tranche 3, the executing agencies complied with all covenants, except for partial compliance with one covenant in the project agreement with Azersu—the substantial cost overruns had adverse implications for subproject implementation efficiency.

II. EVALUATION OF PERFORMANCE AND RATINGS

A. Relevance of Design and Formulation

14. The PCR rated the program and tranche 3 highly relevant. The program and its intended outcome were strategically aligned with the government's program on poverty reduction and sustainable development for 2008–2015. The program was also aligned with the state program for 2009–2013 on socio-economic development of regions (footnote 1), where improving access to basic services was identified as a strategic priority. The government set the targets of 100% WSS coverage and 24-hour water supply in Baku and 80%–85% WSS coverage in secondary towns and villages. The relevance of WSS in secondary towns was also accentuated in the government's 2016 strategic road map10 for the development of utilities, which set the objectives of providing countrywide coverage through high quality WSS infrastructure and services and increased WSS system efficiency. The program was fully aligned with ADB strategies as stated in the country strategy and program update,11 country partnership strategy,12 ADB's Strategy 2020,13 and ADB’s Water Operational Plan 2011–2020.14

15. Using the MFF as the financing modality was appropriate. The preference for the MFF to other modalities was well justified in the RRP. Preconditions including road map, policy framework, strategic context, investment program, financing plan and undertakings were all satisfied. The MFF modality facilitated continuity and linkages between tranches during implementation. It allowed the government to decide the financing for each project based on its

9 ADB. 2012. Loan 2571-AZE: Water Supply and Sanitation Investment Program (Project 1) Amendment to Loan Agreement. Letter to the Ministry of Finance. Manila. 11 January (internal). 10 Government of Azerbaijan. 2016. Presidential Decree of the Azerbaijan Republic No. 1138 on 6 December 2016. About Approval of Strategic Road Maps on National Economy and the Main Sectors of Economy. Baku. 11 ADB. 2006. Country Strategy and Program Update: Azerbaijan. Manila showed that WSS was identified as one of the core areas of operations. 12 ADB. 2014. Country Partnership Strategy: Azerbaijan, 2014-2018. Manila 13 ADB. 2008. Strategy 2020. Manila. 14 ADB. 2017. Water Operational Plan 2011-2020. Manila. 6

readiness and defer some outputs to subsequent tranches. The experience and lessons from each tranche were fed into the design and implementation of subsequent tranches, improving readiness and facilitating procurement. The program's design was appropriate. However, implementation deviated somewhat from the original scope, materially affecting the relevance ratings of individual tranches and the program.

16. Tranche 1. The subproject in Goychay had design, costing and implementation deficiencies. Even though the detailed design was completed at appraisal to prepare the ground for advance contracting, the design and costing administered by Azersu were inaccurate—the lowest bid was $96 million lower than the cost estimate of $36 million for WSS network contract at appraisal. This significant cost overrun led to the cancellation of the planned advance contracting in mid-2010. The scope of civil works for Goychay was then reduced, with the remaining portion of the WSS network transferred to tranche 2. The re-tendering and procurement of reduced works were finalized in late 2012, causing a delay of over 2 years. This affected the attainment of the original outcome and outputs of tranche 1.

17. Tranche 2. The original plan of achieving full WSS coverage in Agdash with this tranche's funds was not possible due to the government's expansion of Agdash's administrative area, which necessitated a change in the design leading to cost overruns. This tranche covered a large part of Agdash's expansion area, and the remainder was financed by the state. The WWTPs in Agdash and Beylagan were excluded from the tranche as a result of cost overruns incurred by a fundamental design change. Azersu changed the daily per capita water consumption from 120 to 180 liters for all service areas. This led to substantial changes to key design parameters, i.e., pipe diameters, number of wells, and size of reservoirs among others. These changes delayed civil works and increased the cost of raw materials and equipment. The PCR of tranche 2 and its associated documents did not substantiate that the expansion of Agdash area and the design change by Azersu were unforeseeable external events.15

18. Tranche 3. In late 2016, Azersu proposed to change the wastewater treatment technology from secondary to primary. Multiple discussions between ADB and Azersu on the importance of biological treatment did not reach a timely consensus, resulting in the exclusion of Agjabedi WWTP from the procurement plan of tranche 3 (para. 12). The Agjabedi WWTP was financed using government funds as an alternative arrangement. The construction started in February 2019 and has been completed in 2020. 16 This ensured the achievement of tranche 3’s intended outcome relating to the coverage of the sewerage services. In addition, the inclusion of Shakarabad WSS and Agdash wastewater collector was an appropriate response to the requests from the executing agencies, contributing to the extension of WSS services.

19. The program featured an innovation in water treatment technology. In Nakhchivan, SAWMC researched various technological options and chose to replace the standard chlorination-based water treatment with the more advanced nanofiltration technology. This technology reduced mineral content and water hardness and used gravity as a standard design feature to reduce electricity consumption, which enhanced its sustainability. In the event of power shortage, water could still be filtered with ozone and sodium hypochlorite. Being a good

15 However, communications with the Central and West Asia Department (CWRD) revealed that the project team couldn’t have foreseen the changes made to the assumption on water demand by Azersu nor the expansion of administrative area by Agdash. 16 The PCR of the program and tranche 3 dated October 2019 indicated that the construction of Agjabedi WWTP started in February 2019 and was planned for completion in December 2019. As Azerbaijan Resident Mission (AZRM) in July 2020 reported, the construction has been completed. 7

practice, the technology could have demonstration and transformational effects in the WSS sector in Azerbaijan.17

20. Tranche 3 and the entire program were aligned with the government's and ADB's strategies. The MFF modality was appropriate and relevant. The nanofiltration technology was adopted by multiple water treatment plants in the region. This validation assesses tranche 3 highly relevant. The whole program, however, did not qualify for a highly relevant rating due to the aforementioned issues associated with tranches 1 and 2. This validation therefore assesses the whole program relevant.

B. Effectiveness in Achieving Project Outcomes and Outputs

21. The PCR rated the program and tranche 3 effective. The program largely achieved its intended outcome of improved coverage, continuity and quality of water supply and wastewater disposal in program towns. The outcome indicator of 100% WSS services coverage was achieved by SAWMC in Nakhchivan but partially achieved by Azersu in the four participating towns. A complete WSS services network was developed in Goychay. However, the WWTPs in Agdash, Agjabedi and Beylagan were still being constructed using state budget when the program was closed. Thus, households would be connected to the sewerage networks constructed under the program only after these WWTPs were commissioned and operational. The outcome indicator of residents being served by fully pressurized 24-hour water supply was achieved in all towns. The water loss rate was reported by Azersu and SAWMC to be reduced to 16% and 5% respectively, exceeding the target of 25%. The indicator of 100% wastewater collection and treatment was fully achieved in Nakhchivan but partially achieved in Azersu subprojects. Specifically, Goychay reached 95%, whereas Agdash, Agjabedi and Beylagan would only meet this target after the completion of WWTPs construction.

22. Underpinning the program's largely achieved outcome indicators were the physical outputs that were delivered via its three tranches. With the exception of wastewater treatment, where target indicators were partially achieved, the program achieved or exceeded target indicators for all outputs directly or indirectly relating to physical investment, including water distribution network, sewer lines, storage reservoirs, system leaks, water meters, tariff collection efficiency, and water quality monitoring. The program's institutional effectiveness outputs comprised 15 indicators associated with six specific outputs that covered WSS service institutions improvement, capacity development for system planning and operation, cost recovery enhancement, project processing efficiency increase, twinning arrangement, and consumers mainstreaming. Out of the 15 indicators, seven were partially achieved, unachieved, or not implemented. A key one was the annual tariff review and adjustment, which was not realized as tariff adjustments are centralized nationally through the Tariff Council. Others included digitized maps for WSS networks, TWUA, communication strategy, customer care unit, performance benchmarking system, and gender awareness workshops.

23. For tranche 3, two outcome indicators were achieved, and one was partially achieved. At completion, 100% of residents in Agjabedi and the peri-urban towns of Nakhchivan had access to 24-hour potable water supply. Water loss rate was reduced to less than 20% in Agjabedi and

17 The PCR of the program and tranche 3 dated October 2019 did not adequately discuss the technology's potential transformational effects. However, additional information has been provided by AZRM in July 2020 to substantiate the transformational effects. The technology has been scaled up for multiple water treatment plants in the region, and water reuse has been systematized for irrigation. In addition, the projects are characterized by their intrinsic value of contributing significantly to improving the beneficiaries' well-being. Water treatment technology plays a fundamental role in this context. 8

less than 5% in Nakhchivan peri-urban towns. As for access to sanitation facilities, 100% coverage was achieved at Nakhchivan peri-urban towns, but not in Agjabedi at program completion since WWTP construction was removed from the program's scope.18 Physical outputs relating to WSS development were largely achieved. In Agjabedi, 210 (planned 204) kilometers (km) of water distribution network, 200 (planned 197) km of sewer lines, and 7,000 (planned 8,000) cubic meter (m3) of storage reservoirs were constructed. In Nakhchivan peri-urban towns, 65 (planned 62) km of water distribution network and 49 (planned 48) km of sewer lines were constructed. As unplanned output, 16km of water distribution network and 15km of sewer lines were constructed in Shakarabad using contract savings and unused loan balance. As for the outputs relating to institutional effectiveness, five out of the seven indicators were achieved. The target of women comprising 15% of the recruited staff was not achieved. The twinning program with an external WSS utility was not implemented in view of of a 5-year training program arrangement between Azersu and Suez Environment.

24. The program and tranche 3 were correctly classified for safeguard categories (para. 11). An environmental assessment and review framework and a resettlement framework were prepared at the time of program preparation. For each tranche, the environmental assessment comprising initial environmental examinations (IEEs) on the subprojects included in the tranche met ADB’s requirements. The IEEs did not identify any significant environmental impact that would lead to a detailed environmental impact study.19 The Land Acquisition and Resettlement Framework and Land Acquisition and Resettlement Plan (LARP) were prepared in line with ADB's Safeguard Policy Statement (2009) and relevant legal requirements. The executing agencies conducted monitoring and reporting and prepared a series of Environmental Monitoring Reports (EMRs) to update project implementation and mitigation measures for the corresponding periods. On the whole, the safeguard performance of the program and tranche 3 was satisfactory.

25. There were 20 activities and 2 targets in the program's GAP (para. 11). The non- implementation of the five activities relating to TWUAs significantly affected the implementation of the program's GAP. Consequently, nine activities were implemented, but none of the targets was achieved. The successful conduct of consultations was critical for the preparation of tranche- specific GAPs. For tranche 3's GAP, its implementation was not successful, given 50% completion of activities and 38% achievement of targets. This was in view of some GAP indicators that were not applicable to the local context and the executing agencies' lack of ownership of the GAP.

26. The program and tranche 3 were generally successful in achieving intended outcomes by substantially delivering on the WSS infrastructure. There were shortfalls in outputs relating to institutional effectiveness. However, considering the importance of physical outputs relative to the soft components, this validation assesses the program and tranche 3 effective.

C. Efficiency of Resource Use

27. The PCR rated the program and tranche 3 efficient as the re-evaluated economic internal rate of return (EIRR) of the whole investment program and that of the Agjabedi subproject under tranche 3 were both higher than the benchmark of 12%. The PCR re-evaluation of EIRRs was consistent with the approach taken at appraisal of the program and also aligned with ADB

18 The WWTP in Agjabedi was financed by government funds and its construction has been completed in 2020. See footnote 16. 19 The Environmental Management Plans (EMPs) prepared as part of the IEEs adequately described the anticipated impacts, stages of monitoring and mitigation measures and established institutional requirements and responsibilities for implementing the subprojects’ safeguards. 9

guidelines.20 The economic capital investment was updated based on the actual financial costs characterized by actual scope and implementation delay. The estimate of annual operations and maintenance (O&M) costs was adjusted accordingly. The economic benefits of water supply were generated by both incremental and non-incremental outputs. The incremental benefits were estimated using the average willingness to pay obtained from surveys in local areas and the long- run marginal cost proxied by the average incremental economic cost. The non-incremental benefits were estimated based on resource cost savings associated with existing supplies under the without-project scenario, including the cost of time used for collecting water from point sources, cost of in-house water storage (purchase, O&M, water boiling and filtration), cost of tankered water, and cost of bottled water. The economic benefits of sewerage subprojects monetized in the calculation were estimated based on costs that would have been incurred in the without-project scenario, including septic tank emptying cost, drainage overflow cost and loss of work and leisure due to poor hygiene and health. The benefit of wastewater treatment was included for Goychay and Nakhchivan, but was not supposed to be for Agdash, Agjabedi and Beylagan due to the removal of the WWTPs from the program scope (paras. 22-23). For all subprojects, benefits relating to long-run marginal cost savings, increase in property values, health impact and environmental impact were not quantified due to lack of information.

28. The EIRRs recalculated by the PCR were 14.6% for Agdash, 24.7% for Agjabedi, 14.9% for Beylagan, 12.9% for Goychay, 15.7% for Nakhchivan and 15.3% for the whole investment program, all exceeding the 12% benchmark. This validation noted some inconsistencies in the PCR. Paragraph 13 of the PCR's Appendix 10 on economic reevaluation stated that WWTPs in Agdash, Agjabedi and Beylagan were excluded in its analysis. However, the WWTP benefit was still included in the calculation, as shown in Table A10.5 and the note to Table A10.10. The WWTP benefits were also included in the supporting Excel spreadsheet. This validation recalculated the EIRRs by removing WWTP benefits from the total economic benefits of Agdash, Agjabedi and Beylagan subprojects and found them to be 13.7%, 23.6% and 14.3% respectively. The recalculated EIRR for the whole investment program was 14.9%.

29. Table A1.1 of the PCR's DMF for the whole investment program stated that the water loss rate was reduced to 16% for Azersu subprojects (Agdash, Agjabedi, Beylagan and Goychay) and 5% for SAWMC subproject (peri-urban towns of Nakhchivan). However, Table A1.2 of the PCR's DMF for tranche 3 indicated that the water loss rate was 20% for Agjabedi subproject. Meanwhile, the water loss rates for Agdash, Beylagan, Goychay and Nakhchivan reported in the PCR for Project 2 were 11%, 10.5%, 9.5%, 10%, respectively.21 To test the sensitivity of the EIRRs to the water loss rate, this validation recalculated the EIRRs by setting the water loss rate to 20% for all Azersu subprojects and 15% for SAWMC subproject. Under this adverse scenario, the recalculated EIRRs continued to be higher than the 12% benchmark. 22 This validation thus assesses the program and tranche 3 efficient.

D. Preliminary Assessment of Sustainability

30. The PCR rated the program and tranche 3 likely sustainable. The PCR's financial analysis was consistent with the approach taken at appraisal and aligned with the general principles and sector-specific considerations outlined in ADB's guidance note.23 It updated the financial analysis to reflect the actual situation at project completion. Apart from project cost and implementation

20 ADB. 2017. Guidelines for the Economic Analysis of Projects. Manila 21 ADB. 2019. Completion Report. Azerbaijan: Water Supply and Sanitation Investment Program (Tranche 2). Manila 22 13.6% for Agdash, 21.7% for Agjabedi, 13.5% for Beylagan, 12.3% for Goychay, 15.2% for Nakhchivan and 14.5% for the whole investment program. 23 ADB. 2019. Financial Analysis and Evaluation: Technical Guidance Note. Manila 10

period, the updated analysis used the actual water and wastewater tariff revisions from 2011 to 2016. The combined tariff for water and wastewater was increased from Azerbaijan manat (AZN) 0.17 per m3 to AZN 0.31 per m3 in 2011 and further to AZN 0.45 per m3 in 2016. Accordingly, the PCR updated the tariff revision assumption in the model to an increase of 25% in every two years, applied to all five towns. Also, the AZN was devalued during the implementation period—dollar to AZN rate went from 0.85 in 2009 to 1.69 in 2017. Since financial analysis is in dollars, this devaluation affected the re-evaluated FIRRs considerably.

31. The recalculated FIRRs were –5.3% for Agdash, –3.8% for Agjabedi, –15.4% for Beylagan, –9.1% for Goychay and –5.2% for Nakhchivan, whereas the weighted average cost of capital (WACC) was 2.1%. Higher FIRRs exceeding the WACC would be possible with much stronger tariff revision and/or with an operational subsidy if consumer affordability was a major factor. Importantly, the government on-granted ADB financing to Azersu and SAWMC, which implied they were not subject to loan repayment. Hence, the PCR evaluated the subproject's operational sustainability from the perspective of O&M cost recovery with possible partial capital cost recovery to meet periodic asset replacement requirements. It was found that Nakhchivan had been operationally sustainable since 2017 and Agdash, Agjabedi, Beylagan and Goychay would achieve operational sustainability by 2022. Meanwhile, with the same assumption on tariff revisions, the percentages of household income spent for water supply and sewerage in all towns were well below the generally accepted limit throughout the project period, thereby confirming consumer affordability.

32. The above operational sustainability was conditional upon periodic tariff increase and collection efficiency assumed in the PCR re-evaluation. This was a very strong assumption characterized by uncertainties as evidenced by the fact that only two tariff increases were done from 2009 to 2019, rather than the expected annual tariff review and increase. The PCR pointed out that periodic tariff increase was a challenging policy action, which required a complex approach through a well-designed communication strategy, effective coordination and capacity building involving all relevant government authorities, such as the Tariff Council, Ministry of Economy, Ministry of Finance, and the concerned departments of the Cabinet of Ministers. This validation notes that the PCR of tranche 1 gave a rating of less than likely sustainable to tranche 1, which like tranches 2 and 3, was financially unsustainable due to negative FIRRs but operationally sustainable on the condition of periodic tariff increase and collection efficiency. The program and tranches made improvements in institutional effectiveness and project management. However, these were not adequate to mitigate the material risks so much so that the tariff would not be adjusted as expected. 24 Given these considerations, this validation assesses the program and tranche 3 less than likely sustainable.25

24 The CWRD argues that sustainability in the Central and West Asia context cannot be measured based on the FIRR calculations and tariffs alone, as no country in the region has achieved the level of tariff adjustments needed for full cost recovery, or even operational sustainability without government subsidies. It added that the fact that the government committed to invest using their own funds to complete the WWTPs in the project towns before MFF closure and that the WWTPs in Agdash, Aghjebedi, and Beylagan are already constructed and will all be operational by end of 2020 are a clear indication that there is political will and continued financial support. The IED acknowledges that full cost recovery is difficult to attain for any country. However, per the Evaluation Guidelines, if the project FIRR is less than the WACC, the rating should be “less than likely sustainable.” Moreover, covering O&M costs needed strong assumptions to be satisfied. The IED notes that there is no evidence to suggest that the risks of not being able to revise tariffs periodically and not being able to improve collection efficiency have been mitigated (tariff revisions occurred only twice during 2009 to 2019). In addition, the use of the government’s own funds to complete the WWTPs is not an evidence of risk mitigation. 25 CWRD disagrees with this validation’s assessment of sustainability. 11

III. OTHER PERFORMANCE ASSESSMENTS

A. Preliminary Assessment of Development Impact

33. The PCR rated the program and tranche 3 satisfactory on development impact. With increased network coverage, provision of 24-hour supply of pressurized water, reduction in water loss rate, and the availability, quality, quantity and reliability of WSS services were significantly improved in all five towns of the program. Residual chlorine at the tail end of water supply system was maintained at appropriate level. The well-being of residents in these towns improved significantly. Women's workloads were lightened considerably as they did not have to collect, filter, boil and chemically treat water. The operationalization of the new sewerage network and WWTPs eliminated the negative public health and environmental impact from discharging wastewater into streets and nearby watercourses due to blocked and collapsed sewers. As for health, the percentage of children aged five or younger suffering from diarrhea was found to have been reduced to less than 10% in Goychay and 2% in Nakhchivan based on information provided by the EAs. The program gained public satisfaction—informal interviews with selected beneficiaries in three towns conducted during ADB's final review mission and PCR mission26 confirmed the beneficiaries' satisfaction with the program outcomes. Institutionally, the capacity of Azersu and SAWMC was developed during the program implementation. Azersu established an in-house design team that was able to prepare and manage projects independently. The SAWMC achieved operational sustainability for WSS in Nakhchivan and its peri-urban areas, with good practices replicable throughout the secondary towns in Nakhchivan Autonomous Republic (NAR). Considering the above observations and the development impact’s congruence with the program outcomes and outputs, this validation assesses the development impact of the program and tranche 3 satisfactory.

B. Performance of the Borrower and Executing Agency

34. The PCR rated the performance of the borrower and SAWMC satisfactory and the performance of Azersu less than satisfactory. The borrower performed its functions, including timely submission of the periodic financing request, effectively facilitating loan negotiations and agreement signing, providing adequate counterpart financing and supporting efficient oversights and actions during project implementation. In general, the executing agencies facilitated and supported ADB's review missions during implementation and at completion. The PMOs comprising technical, financial and safeguards professionals, managed project activities relating to planning, implementation, progress monitoring and reporting. Audited reports of project accounts and financial statements were submitted to ADB in compliance with the loan agreement. However, the specific performance of Azersu and SAWMC differed substantially.

35. The SAWMC demonstrated strong commitment, ownership and accountability. It took initiative to conduct proper due diligence and technical designs, engage in technical discussions, and closely monitor implementation through a fully staffed PMO. In contrast, Azersu was much less proactive and accountable. The lack of a comprehensive approach to project formulation, implementation, and monitoring reflected Azersu’s weak management and lack of checks and balances to ensure proper due diligence, technical verification, and contract management. The abolishment of Azersu's PMO (para. 12) adversely impacted the civil works implementation and also led to the GAP and institutional capacity building activities being largely ignored.

26 Based on the PCR (footnote 14, para. 52), informal interviews were conducted with project beneficiaries in Agdash, Beylagan and Goychay. The PCR and mission reports did not indicate the coverage of these interviews. 12

This validation assesses the performance of the borrower and the executing agencies satisfactory, although the performance of Azersu was less than satisfactory.

C. Performance of the Asian Development Bank and Cofinanciers

36. The PCR rated ADB's performance satisfactory. ADB worked closely with the borrower, executing agencies and other stakeholders to ensure timely project preparation, approval and implementation. The advance procurement led to the expeditious contract awarding. Despite frequent staff turnover during the early stage of program implementation, ADB established and maintained a close working relationship with the executing agencies. Project-related issues were identified in time and effectively resolved through loan review missions. Close coordination between ADB headquarters and the Azerbaijan Resident Mission ensured the timely provision of proactive, comprehensive and continuous support to the executing agencies. Timely inputs, advice and document approvals and prompt payments of withdrawal applications were ensured consistently.

37. ADB's safeguard work quality at appraisal and supervision was satisfactory. The program and tranche 3 were correctly classified on safeguards (para. 11). The translation of environmental and social requirements to RRP, loan agreements and facility administration manual was adequate. For tranche 3, the quality of the four IEEs for all subprojects (Agdash, Agjabedi, Nakhchivan and Shakarabad) was satisfactory and the environmental management plans (EMPs) gave adequate details on potential impacts, stages of monitoring, and mitigation measures. ADB's supervision of EMPs and LARP implementation was effective. The seven EMRs between July 2015 and June 2018 and several ADB's site visits met ADB's requirements and showed that the PMOs had the capacity to implement the EMPs. This validation assesses ADB's performance satisfactory for the program and tranche 3.

IV. OVERALL ASSESSMENT, LESSONS, AND RECOMMENDATIONS

A. Overall Assessment and Ratings

38. The PCR rated the program and tranche 3 successful. This validation assesses tranche 3 highly relevant due to its alignment with the government's and ADB's strategies and the scaling up of its innovative design elements. It could not assess the program highly relevant as the changes made to the project scope and design could not be justified for unforeseeable reasons. The validation assesses the program and tranche 3 effective, as they were successful in achieving the intended physical outcomes. It assesses them efficient since the EIRRs of all subprojects are above the benchmark rate of 12%, but it assesses the program and tranche 3 outcomes less than likely sustainable due to the negative FIRRs of all subprojects and the unrealistic assumption on operational sustainability. Overall, this validation assesses tranche 3 successful. It assesses the entire program successful given that tranches 2 and 3 were assessed successful, and tranche 1 issues were addressed in the follow up tranches.

13

Overall Ratings Reason for Disagreement Validation Criteria PCR IED Review and/or Comments Relevance L2571 Highly relevant Relevant The design had deficiencies L2842 Highly relevant Relevant under tranche 1 and tranche 2 L3079 Highly relevant Highly relevant (paras. 16-17). MFF0032 Highly relevant Relevant Effectiveness L2571 Less than effective Less than effective L2842 Effective Effective L3079 Effective Effective MFF0032 Effective Effective Efficiency L2571 Efficient Efficient L2842 Efficient Efficient L3079 Efficient Efficient MFF0032 Efficient Efficient Sustainability L2571 Less than likely Less than likely Negative FIRRs; not able to sustainable sustainable address the preconditions of L2842 Likely sustainable Less than likely periodic tariff increases. sustainable L3079 Likely sustainable Less than likely sustainable MFF0032 Likely sustainable Less than likely sustainable Overall Assessment L2571 Successful Less than successful L2842 Successful Successful L3079 Successful Successful MFF0032 Successful Successful Preliminary Assessment of Impact L2571 Satisfactory Satisfactory L2842 Satisfactory Satisfactory L3079 Satisfactory Satisfactory MFF0032 Satisfactory Satisfactory Borrower and executing agency L2571 Satisfactory Satisfactory L2842 Satisfactory Satisfactory L3079 Satisfactory Satisfactory MFF0032 Satisfactory Satisfactory Performance of ADB L2571 Satisfactory Satisfactory L2842 Satisfactory Satisfactory L3079 Satisfactory Satisfactory MFF0032 Satisfactory Satisfactory Quality of PCR Satisfactory Para. 43. ADB = Asian Development Bank, IED = Independent Evaluation Department, PCR = project completion report. Source: ADB (IED).

14

B. Lessons

39. The PCR identified three lessons, which this validation endorses. First, executing agency engagement and accountability are critical to the success of program and projects. Second, weak assessment of local conditions and unclear definitions of administrative areas adversely affect technical designs and project feasibility. Third, tariff adjustments to ensure the sustainability of WSS services need to be holistically pursued in Azerbaijan.

40. This validation identifies four sector-level lessons. First, effectively engaging key government agencies can provide the much-needed oversight to monitor progress and facilitate the resolution of design and implementation issues. Second, adequate stakeholder consultations can ensure the appropriateness and feasibility of the design of soft components such as gender and capacity building in various local contexts. Third, a gender specialist should join reconnaissance and fact-finding missions to discuss GAP details with executing agencies and agree on implementation arrangements. Fourth, ADB's project readiness financing modality may be utilized to support project cost estimates, detailed design and bidding document preparation.

C. Recommendations for Follow-Up

41. The PCR recommended three specific follow-up actions by ADB, which are related to continuous monitoring of the subprojects' performance and future ADB intervention in support of Azerbaijan's WSS sector. First, the government-financed WWTP constructions in Agdash, Agjabedi and Beylagan should follow ADB's environmental safeguard requirements. Azersu should be required by ADB to submit semiannual environmental reports to ensure compliance. Second, future ADB intervention in Azerbaijan's WSS sector should take a sector development program approach to focus on institutional reforms rather than infrastructure to ensure continuity of WSS services and long-term financial sustainability of Azersu. Third, ADB should support SAWMC through additional investments in the sector and promote the NAR as a regional center to demonstrate best practices. This validation considers these recommendations useful to future WSS programs/projects in Azerbaijan.

V. OTHER CONSIDERATIONS AND FOLLOW-UP

A. Monitoring and Reporting

42. The PMOs developed project performance, monitoring and evaluation indicators, which were approved by ADB. Quarterly progress reports and semiannual environmental monitoring reports were prepared and submitted to ADB in a timely manner. Financial management systems were established, with internal financial and accounting controls mostly meeting ADB’s requirements. Beneficiaries' feedback to program outcomes was collected through informal interviews (footnote 26.

B. Comments on Project Completion Report Quality

43. The PCR quality is rated satisfactory as it is well-prepared and generally consistent with the guidelines. Coherent with the tranche-specific PCRs, it comprehensively described the program and tranche 3 designs and implementation. It presented a convincing assessment of the achievements, except for the rating of highly relevant. Major design and implementation issues and scope changes are well-explained. The safeguards and gender aspects are succinctly summarized. The economic and financial re-evaluations are good and clear, with the supporting spreadsheet presenting an elaborate and transparent model. The extent and the coverage of the 15

interviews upon which the impact assessment was made are not adequately described (footnote 26. The lessons and recommendations are relevant and useful.

C. Data Sources for Validation

44. Data sources for this validation include the PCRs for tranche 1, tranche 2 and tranche 3 and the program, the RRP, the periodic financing request reports for tranche 2 and tranche 3, loan mission reports, safeguard assessment, national and ADB strategies, and ADB guidelines.

D. Recommendation for Independent Evaluation Department Follow-Up

45. The PCR did not explicitly make a recommendation for a project/program performance evaluation report (PPER). However, the tranche 1 PCR suggested that a PPER be prepared by the end of 2021 when the facilities of tranche 1 would have been in operation for 4 years. The tranche 2 PCR suggested that a PPER in the second half of 2021 for all three tranches was more suitable. This validation though suggests preparing an all-in-one PPER after the actual commissioning of the Agdash, Agjabedi and Beylagan WWTPs. The time required for the WWTPs to reach stable operational conditions should be taken into account. The operation of Goychay and Nakhchivan WWTPs since commissioning could serve as a useful reference.

16

APPENDIX: LINKED DOCUMENTS

PCR Validation Report: Azerbaijan: Water Supply and Sanitation Investment Program (Multitranche Financing Facility and Tranche 3) a. CWRD Comments on Final PVR Document available upon request.

b. IED Response to CWRD Comments

https://www.adb.org/sites/default/files/evaluation-document/630081/files/ied-

response-cwrd-comments-pvr-aze-3079_0.pdf