Report No: ICR00003928
Document of The World Bank
Report No: ICR00003928
Public Disclosure Authorized
IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-79630)
ON A
LOAN
IN THE AMOUNT OF US$24 MILLION Public Disclosure Authorized TO THE
REPUBLIC OF ARMENIA
FOR A
E-SOCIETY AND INNOVATION FOR COMPETITIVENESS (EIC) PROJECT
December 27, 2016
Public Disclosure Authorized
Public Disclosure Authorized Transport and ICT Global Practice Trade and Competiveness Global Practice Europe and Central Asia
CURRENCY EQUIVALENTS (Exchange Rate Effective June 30, 2016)
Currency Unit = AMD US$1.00 = AMD 476.24 AMD 1 = US$0.0021
FISCAL YEAR January 1–December 31
ABBREVIATIONS AND ACRONYMS CA Certification Authority CPS Country Partnership Strategy EIC E-Society and Innovation for Competitiveness e-ID Electronic Identification Document EIF Enterprise Incubator Foundation EKENG E-Governance Infrastructure Project Implementation Unit FDI Foreign Direct Investment FFPMC Foreign Financed Projects Management Center FM Financial Management GDP Gross Domestic Product GoA Government of Armenia GTC Gyumri Technology Center IBM International Business Machines ICT Information and Communication Technology ISR Implementation Status and Results Report ISTC Innovation Solutions and Technologies Center IT Information Technology ITES Information Technology-Enabled Services LIL Learning and Innovation Loan M&E Monitoring and Evaluation MTR Midterm Review NCFA National Competitiveness Foundation of Armenia PAD Project Appraisal Document PDO Project Development Objectives PHRD Program for Human Resources Development PIU Project Implementation Unit PPP Public Private Partnership PR Public Relations R&D Research and Development SME Small and Medium Enterprise TA Technical Assistance TTL Task Team Leader VC Venture Capital VTC Vanadzor Technology Center
Senior Global Practice Director: Jose Luis Irigoyen Country Director: Mercy Miyang Tembon Country Manager: Laura E. Bailey Practice Manager: Jane Lesley Treadwell Project Team Leader: Sandra Sargent, Feyi Boroffice ICR Team Leader: Junko Narimatsu, Arsen Nazaryan ICR Primary Author: Junko Narimatsu
ARMENIA E-Society and Innovation for Competitiveness (EIC) Project
CONTENTS
Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph
1. Project Context, Development Objectives and Design ...... 1 2. Key Factors Affecting Implementation and Outcomes ...... 7 3. Assessment of Outcomes ...... 13 4. Assessment of Risk to Development Outcome ...... 20 5. Assessment of Bank and Borrower Performance ...... 21 6. Lessons Learned...... 23 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ...... 24 Annex 1. Project Costs and Financing ...... 25 Annex 2. Outputs by Component...... 26 Annex 3. Economic and Financial Analysis ...... 29 Annex 4. Bank Lending and Implementation Support/Supervision Processes ...... 32 Annex 5. Beneficiary Survey Results ...... 34 Annex 6. Stakeholder Workshop Report and Results ...... 36 Annex 7. Borrower's ICR and/or Comments on Draft ICR ...... 37 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ...... 39 Annex 9. List of Supporting Documents ...... 40 MAP IBRD 33364 ...... 41
A. Basic Information E-Society and Innovation for Country: Armenia Project Name: Competitiveness (EIC) Project Project ID: P115647 L/C/TF Number(s): IBRD-79630 ICR Date: 12/27/2016 ICR Type: Core ICR REPUBLIC OF Lending Instrument: SIL Borrower: ARMENIA Original Total USD 24.00M Disbursed Amount: USD 23.30M Commitment: Revised Amount: USD 23.30M Environmental Category: B Implementing Agencies: Enterprise Incubation Foundation e-Government Infrastructure Implementation Unit OJSC (EKENG) Foreign Financing Projects Management Center Cofinanciers and Other External Partners:
B. Key Dates
Revised / Actual Process Date Process Original Date Date(s) Concept Review: 06/25/2009 Effectiveness: 05/18/2011 09/08/2011 Appraisal: 09/07/2010 Restructuring(s): 07/09/2014 08/07/2015 Approval: 11/30/2010 Mid-term Review: 10/25/2013 05/03/2013 Closing: 12/31/2014 06/30/2016
C. Ratings Summary
C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Low Bank Performance: Satisfactory Borrower Performance: Satisfactory
i C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Highly Satisfactory Highly Satisfactory Agency/Agencies: Overall Bank Overall Borrower Satisfactory Satisfactory Performance: Performance:
C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Quality at Entry Project at any time No None (QEA): (Yes/No): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately
Closing/Inactive status: Satisfactory
D. Sector and Theme Codes
Original Actual Sector Code (as % of total Bank financing) Public administration - Information and communications 24 24 Telecommunications 20 20 Information technology 30 30 Banking Institutions 7 7 Other Industry, Trade and Services 19 19
Theme Code (as % of total Bank financing) Administrative and civil service reform 7 7 Infrastructure services for private sector development 55 55 Micro, Small and Medium Enterprise support 19 19 Technology diffusion 19 19
E. Bank Staff
Positions At ICR At Approval Vice President: Cyril E Muller Philippe H. Le Houerou Country Director: Mercy Miyang Tembon Asad Alam
ii Practice Jane Lesley Treadwell Philippe Dongier Manager/Manager: Project Team Leader: Sandra Sargent Juan Navas-Sabater ICR Team Leader: Junko Narimatsu, Arsen Nazaryan ICR Primary Author: Junko Narimatsu
F. Results Framework Analysis
Project Development Objectives (from Project Appraisal Document) To address constraints to competitive e-Society and enterprise innovation in Armenia by strengthening the underlying infrastructure and enabling environment
Revised Project Development Objectives (as approved by original approving authority) The PDO did not change. A PDO indicator 'Vanadzor Technology Center (VTC) facility is ready for occupancy' was added as a PDO indicator in the 2014 level 2 restructuring. However, it was obvious that this output indicator did not belong as a PDO indicator and should have been added as an intermediate outcome indicator. The project team cited to the ICR authors that the team considers this an inadvertent error that occurred during the comprehensive restructuring efforts, and was not meant to be added as a PDO indicator (see section 2.3 of ICR)
(a) PDO Indicator(s)
Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Access to Internet services (% of population using the Internet) Indicator 1 :
Value 15 50 73.36 quantitative or
Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Comments Target surpassed, achieving 146.7% of original target value. (incl. % achievement) Access to computers (% of population using a computer) Indicator 2 :
Value 15 50 57.4 quantitative or
Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016
iii Comments Target surpassed, achieving 114.8% of original target value. (incl. % achievement) Development of knowledge and technology-intensive industries (ratio of Indicator 3 : IT/ITES sector revenue in GDP)
Value 1.5 2.2 5.29 quantitative or
Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Comments Target surpassed, achieving 240.4% of original target value. (incl. % achievement) IT/ITES sector employment (number of people) Indicator 4 :
Value 5,190 6,900 12,685 quantitative or
Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Comments Target surpassed, achieving 183.8% or original target value. (incl. % achievement) The construction of Vanadzor Technology Center (VTC) building is Indicator 5 : complete
Value No Yes Yes quantitative or
Qualitative) Date achieved 06/10/2014 06/30/2016 06/30/2016 Comments
(b) Intermediate Outcome Indicator(s)
Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Residential broadband household penetration Indicator 1 :
Value 8.3 30 54.4 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/10/2014 Comments
iv
Business broadband penetration Indicator 2 :
Value n.a. n.a. 54,000 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/10/2014 Comments
Value n.a. n.a. n.a. (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/10/2014 Comments
Value 14 5 1.43 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Comments Target surpassed, % decreased by 89.7%. Value from last ISR corrected (incl. % by EIF. achievement) Number of electronic certificates issued Indicator 5 :
Value 0 23,000 1,022,262 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Target greatly surpassed, achieving 4,445% of original target value. The Comments end-target indicated in ISR (5,000) was incorrect, and did not show (incl. % achievement) accumulative target throughout the project.
Percentage of households owning a computer Indicator 6 :
Value 10 40 57 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Comments Target surpassed, achieving 142.5% of original target value. (incl. % achievement)
v Number of households owning a computer purchased under the Computer Indicator 7 : for All Program
Value 3,500 15,000 37,171 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Comments Target surpassed, achieving 247.8% of revised target value. (incl. % achievement) Number of ideas/matching grants approved Indicator 8 :
Value 0 27 31 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Target surpassed, achieving 114.8% of original target value. The end- Comments target tracked in ISR (8) was incorrect, not indicating accumulative end- (incl. % achievement) value.
Number of projects financed through the Venture Fund Indicator 9 :
Value 0 12 11 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Comments Nearly achieved, reaching 91.7 percent of target value. The end target (incl. % tracked in ISR (3) was incorrect, not indicating accumulative end value. achievement) Number of projects prepared and presented to potential investors Indicator 10 :
Value 0 36 210 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Target greatly surpassed, achieving 583% of original target value. The Comments end-target tracked in ISR (3) was incorrect, not indicating accumulative (incl. % achievement) end-value.
Number of enterprises benefited from Gyumri Technology Center (GTC) Indicator 11 :
Value 0 50 102 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Comments Target surpassed, achieving 204% of original target value. Latest value (incl. % confirmed with EIF, hence minor difference with last ISR.
vi achievement) Occupancy rate of Gyumri Technology Center (GTC) Indicator 12 :
Value 0 80 99.2 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Comments Target surpassed, achieving 124% of original target value. (incl. % achievement) Export of IT/ITES services (US$, millions) Indicator 13 :
Value 71.5 165 133.4 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/04/2014 Comments
Value 0 1.15 3.092 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016 Comments Target surpassed, achieving 268.9% of original target value. (incl. % achievement) Labor productivity in IT/ITES sector (US$, thousands per employee) Indicator 15 :
Value 25 43 37 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/04/2014 Comments
Value 0 650 16,572 (quantitative or Qualitative) Date achieved 11/30/2010 12/31/2014 06/30/2016
vii Target surpassed, achieving 2549% of original target value. The end- Comments target tracked in ISR (150) was incorrect, not indicating accumulative (incl. % achievement) end-value.
Residential broadband household penetration in villages under the project Indicator 17 : scope
Value 8.3 30 41.6 (quantitative or Qualitative) Date achieved 06/10/2014 06/30/2016 06/30/2016 Comments
G. Ratings of Project Performance in ISRs
Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 02/27/2011 Satisfactory Satisfactory 0.00 2 10/19/2011 Satisfactory Satisfactory 0.32 3 06/18/2012 Satisfactory Moderately Satisfactory 1.21 4 12/26/2012 Satisfactory Moderately Satisfactory 2.44 5 02/10/2013 Satisfactory Moderately Satisfactory 2.44 Moderately Moderately 6 11/02/2013 6.10 Unsatisfactory Unsatisfactory 7 04/19/2014 Moderately Satisfactory Moderately Satisfactory 9.28 8 09/10/2014 Satisfactory Satisfactory 10.19 9 03/23/2015 Moderately Satisfactory Moderately Satisfactory 11.39 10 09/23/2015 Moderately Satisfactory Moderately Satisfactory 13.08 11 05/05/2016 Moderately Satisfactory Moderately Satisfactory 19.22 12 06/30/2016 Satisfactory Moderately Satisfactory 22.86
H. Restructuring (if any)
ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Reallocation of funds 09/08/2011 N S S 0.32
viii ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions Revised project activities; reallocation of funds; 07/09/2014 N MS MS 9.28 18months closing date extension
Amendment to disbursement 08/07/2015 N MS MS 13.08 categories and description
I. Disbursement Profile
ix
1. Project Context, Development Objectives and Design
1.1 Context at Appraisal
1. Country context. Armenia had gained independence from the Soviet Union in 1991. It is a landlocked country with limited natural resources and comprises 10 regions (marzes) and the capital city of Yerevan. In 2010, it had a population of 2,963,496 and a per capita gross national income of US$3,370 (gross national income per capita, Atlas method). About 36 percent of the population lived below the national poverty line, of which many have been concentrated in rural areas. The services and agricultural sectors generated about 36 percent and 20 percent of the gross domestic product (GDP), respectively.1 After experiencing a decade-long economic expansion, the economy was hit strongly by the global financial crisis, and contracted by 14.1 percent in 2009.2 The World Economic Forum’s Global Competitiveness Index 2009–2010 ranked Armenia 97 out of 133 countries.3 Armenia’s prospects for inclusive and long-term growth required it to diversify into knowledge-driven sectors and move into higher-value products and services.
2. Sector context. Substantial growth in mobile penetration, exceeding over 100 percent, placed Armenia ahead of its subregional neighbors. However, at the time of project preparation, Internet services were still costly for average citizens.4 Only 15 percent of the population had access to computers and the mobile and fixed broadband markets were still at its infancy and concentrated in urban centers.5 As a result, the level of Internet penetration in 2010 remained at about 25 percent of the population. This was significantly lower than the Europe and Central Asia (ECA) average of 56.1 percent. 67 Access to broadband communications remained virtually nonexistent for the rural population.
3. On the other hand, Armenia’s information technology (IT) software and services industry was young but growing at a rapid pace, particularly in Yerevan. Between 2008 and 2010, the industry showed a compound annual growth of 16 percent, and the industry’s contribution to total exports had risen from 3.6 percent in 2003 to 8.5 percent in 2010. There were about 200 firms (foreign and domestic) that provided employment to 4,960 people (4,110 technical).8 The industry revenue totaled to about US$149 million. Despite rapid growth of the industry, local technology entrepreneurship remained weak, largely because of the lack of sufficient skills and know-how, limited access to finance, and the absence of public support instruments that supported innovation. The absence of industry development outside of Yerevan was even more striking, even in Gyumri, the second-largest city in the country that has a population of about 170,000. A city previously known as the ‘city of trade and arts’, Gyumri suffered major damage during the 1988 Spitak
1 National Statistical Service of Armenia. 2 http://documents.worldbank.org/curated/en/967701468003049908/Fiscal-consolidation-and-recovery-in-Armenia- impact-of-the-global-crisis-on-a-small-open-economy. 3 Global Competitiveness Report 2009–2010, World Economic Forum. 4 For example, monthly fee for unlimited Internet traffic of Asymmetric digital subscriber line (ADSL) technology at the speed of 1 Mbps in 2008 stood at US$80 in Armenia, US$10 in Georgia, US$40 in Azerbaijan, and US$28 in Turkey. 5 In 2009, VivaCell MTS introduced 3G mobile Internet, but the mobile broadband market was still at its infancy. 6 World Bank data. 7 World Bank Group. 2010. World Development Indicators. 8 http://mineconomy.am/uploades/EIF-ArmenianITIndustryReport-2010-ENG.pdf.
1
earthquake, and had not been able to restore its entrepreneurial know-how. These limitations hampered technology absorption and knowledge-based enterprise growth in Armenia.
4. Government strategy. In 2001, the Government of Armenia (GoA), with support from the World Bank and U.S. Agency for International Development, among others, developed the information and communication technology (ICT) Master Strategy, which included visions to increase global competitiveness through entrepreneurship and increase access to capital and markets. In 2002, the Enterprise Incubator Foundation (EIF) was established within the framework of a US$5 million World Bank Learning and Innovation Loan (LIL), with an objective to (a) provide a demonstration effect of company formation in an inhospitable business environment (employment creation/productivity objective); (b) provide demonstration effect of foreign direct investment (FDI) in IT-related manufacturing (FDI objective); and (c) pilot demand-driven mechanisms of continuous upgrading of IT skills of students, professors, enterprise managers, and employees (continuous skill upgrading objective). The project closed in 2006, noting satisfactory outcomes as verified in Independent Evaluation Group reviews. Despite the project closing, the EIF maintained its financial sustainability and continued to grow from renting out workspace, collecting user fees, and providing consulting services.
5. In 2008, the Information Technology Sector Development Concept Paper (2008) was adopted, which envisioned achieving ‘extended use of e-services systems’, ‘high-level computer literacy’, and ‘existence of a large local IT market and progressive knowledge-based industry’, over the next five years. To help meet these goals, a concept paper on the Development of e- Society in the Republic of Armenia (2010–2012) was approved in early 2010, which laid out plans to implement a new backbone network infrastructure and to create a secure, efficient, and affordable electronic identification document (e-ID) system. The e-ID system was envisioned to be used for provision of multiple government services, including the GoA’s pension reform program and e-health services, among others. Around the same time, a comprehensive strategic framework for transforming Gyumri into a ‘TechnoCity’ was developed to achieve regional economic development outside of Yerevan.
6. Rationale for World Bank involvement. Despite the solid foundations, the Government faced a number of challenges toward achieving its development objectives for the sector. Internet and computer penetration was low particularly in the regions, and the firms’ start-up and growth were constrained by low access to credit and commercial investments, lack of a qualified workforce, low productivity, and the lack of formal standards for the IT industry. The EIF’s support for innovation and entrepreneurship was also limited because of its size and available financing.
7. The World Bank was well positioned to play a crucial role in the area of information society and innovation based on the World Bank’s experience worldwide in financing similar projects. Examples included broadband development and e-Government projects in Africa (for example, Regional Communications Infrastructure Program), Latin America (for example, Nicaragua Rural Telecommunications Project), and South Asia (for example, e-Lanka Project), among others. In the science, technology, and innovation space, the World Bank was involved in multiple projects worldwide, from the Republic of Korea and India in the 1990s to Turkey, Mexico, and Croatia.
2
8. The World Bank had also been advising successive governments on telecommunications and ICT issues in Armenia since the early 2000s. These included (a) liberalization efforts in the telecommunications sector from 2001 through a series of nonlending technical assistance (TA); (b) e-Government development through the Public Sector Modernization Project; and (c) implementation of the Enterprise Incubator LIL, which established the EIF and laid the foundation for the project. The World Bank’s continued engagement in the sector was deemed essential for consolidating the important gains and for taking the sector’s performance to the next level.
9. Higher-level objectives to which the project contributed. The World Bank’s Country Partnership Strategy (CPS) for Armenia for 2009–2012, grounded in Armenia’s Sustainable Development Program, identified the following two key pillars: (a) addressing vulnerability, with the aim of mitigating the impact of the global economic crisis on Armenia and the most vulnerable population groups and (b) promoting competitiveness and growth, with the objective of helping Armenia to position itself to take advantage of the recovery of the global economy and maintain high and sustained growth over the medium term. The second pillar explicitly prioritized support to Armenia’s transformation into a knowledge economy. Within this context, the Armenia E- Society and Innovation for Competitiveness (EIC) Project was well positioned at supporting the second CPS pillar, while at the same time providing indirect support to the first pillar. The project was also well aligned with the objectives of a series of Development Policy Operations that were under implementation, as a number of policy actions involved reforms in the telecommunications sector. The project’s implementation remained highly relevant to the current World Bank Group Country Partnership Strategy (CPS) for Armenia 2014–17 (see section 3.1)
1.2 Original Project Development Objectives (PDO) and Key Indicators
10. The Project Development Objectives (PDO) as described in the Project Appraisal Document (PAD) was “to address constraints to competitive e-Society and enterprise innovation in Armenia by strengthening the underlying infrastructure and enabling environment.”
11. Key outcome indicators linked to the PDO were as follows:
(a) Access to Internet services (% population using Internet)
(b) Access to computers (% population using a computer)
(c) Development of knowledge and technology-intensive industries (ratio of IT/information technology-enabled service [ITES] sector revenue in GDP)
(d) IT/ITES employment (number of people)
1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification
12. The PDO was not revised during the project.
13. Minor revisions were made to the PDO indicators in the 2014 restructuring, which were (a) introduction of an indicator ‘Vanadzor Technology Center (VTC) facility is ready for occupancy,’ when the establishment of VTC was added to Component 2 (refer to section 2.3. This
3
is an output that should have been added as an intermediate outcome indicator); (b) the updating frequency of Access to Computers was changed from biannual to annual given limitations of available data; and (c) responsibility of Access to Internet Services was moved from the National Competitiveness Foundation of Armenia (NCFA) to the EIF when the NCFA was dropped as an implementing agency because of the change in scope of the broadband component.
1.4 Main Beneficiaries
14. Project beneficiaries were not described explicitly in the PAD but by inference the primary target groups were citizens, trainees, entrepreneurs, firms, and relevant stakeholders that benefited from the Computer for All Program, Innovation Matching Grants, Gyumri Technology Center (GTC), VTC, Venture Fund, and the tech labs (for example, Microsoft, mLAB ECA); recipients and users of the e-ID/e-Certificates; and the village residents that were targeted under the rural broadband (Wi-Fi) subcomponent.
15. Secondary target groups included citizens that benefited from increased employment opportunities generated by the project and consumers of the applications and services that were developed as a result of the project activities.
1.5 Original Components
16. The project components as described in the PAD’s project description were as follows.9
Component 1: E-Society Infrastructure Development (US$12.7 million)
17. The objective of this component was to increase access to affordable broadband services for citizens, businesses, and public institutions, to equip citizens and businesses with a tool for identification and authentication for electronic transactions, and to increase access to affordable computers, content, and e-services for citizens. The subcomponents were as follows:
(a) Subcomponent 1.1: Nationwide Broadband Backbone and Government Network (US$7.9 million). This subcomponent was to support deployment on a public private partnership (PPP) basis of a nationwide broadband backbone network, which would facilitate broadband access throughout the country including in rural areas and with adequate international linkages, as well as with the necessary connections for public institutions throughout the country (Government virtual private network). The subcomponent was to also provide necessary TA for regulatory framework and backbone implementation.
(b) Subcomponent 1.2: Digital Citizen Program (US$1.3 million). This subcomponent was to support the implementation of the Digital Citizen Program aimed at the development and distribution of e-IDs to equip citizens with a tool for identification, authentication, and electronic signatures. As part of this subcomponent, a Certification Authority (CA) for electronic signatures was also to be established.
9 There were minor deviations of the categorization of subcomponents in the Loan Agreement. The PAD is referenced for the purpose of the ICR.
4
(c) Subcomponent 1.3: Computer for All Program (US$3.5 million). This subcomponent was to promote increased computer penetration and digital literacy through a Computer for All Program, which would be financed through a Line of Credit for the Computer for All Program.
Component 2: Fostering Enterprise Innovation (US$10.2 million)
18. The objective of this component was to promote the creation, growth, and competitiveness of knowledge and technology-driven enterprises,10 while at the same time encouraging traditional sectors to adopt new technologies. The subcomponents were:
(a) Subcomponent 2.1: Financial Support for Innovation in Knowledge and Technology-intensive Firms (US$4.7 million). This subcomponent was to finance: (i) Ideas Generation Mini Grants (US$2,000–US$10,000) for business plan and proof of concepts for innovative ideas, and Innovation Matching Grants for product or process adaptation, improvement, and development (US$10,000–US$150,000); (ii) public contribution toward establishing a seed and early-stage Venture Fund, including management team remuneration; and (iii) Innovation Brokerage Team remuneration.
(b) Subcomponent 2.2: Establishment of Gyumri Technology Center (US$2.9 million). This subcomponent was to finance works, goods, and consulting services for the establishment of a technology center in Gyumri to provide technology support and incubation services to local firms and entrepreneurs as an important piece in the Government’s strategy for regional economic development in the Gyumri marz.
(c) Subcomponent 2.3: Support to IT/Knowledge-intensive Industry Development (US$2.6 million). This subcomponent was to finance activities aimed at enhancing the competitiveness of Armenian IT and knowledge-intensive firms by (i) promoting international sales and FDI, through a Silicon Valley Sales Force Representation Office and Managerial Secondments and (ii) facilitating access to know-how and new technologies and promoting productivity improvements, through TA and training for skills development and certification.
Component 3: Project Management (US$0.5 million)
19. This component was to finance necessary activities for adequate project management, including monitoring and evaluation (M&E), and public relations (PR) communications campaigns. Specifically, it was to support TA, capacity building, and incremental operating costs, including travel, as needed for project management, for the e-Governance Infrastructure Project Implementation Unit (EKENG), EIF, NCFA, and Foreign Financed Projects Management Cent (FFPMC). This component was to also support the establishment of a regularly updated M&E
10 Knowledge and technology-driven enterprises include technology-based firms (for example, IT firms) as well as firms that compete on the basis of product, process, or service innovation. Innovation can be either incremental or radical, and either new to the Armenian market or new to the world.
5
system (preferably on a web-based platform) and finance project audits, as well as any necessary beneficiary audits as per the Project Implementation Plan.
1.6 Revised Components
20. The June 2014 Level 2 restructuring revised several subcomponents (Report No. RES13152).
(a) Component 1. Subcomponent 1.1: Nationwide Broadband Backbone and Government Network was scaled down to a Rural Broadband subcomponent (‘digital village’), aiming to install public Wi-Fi access points in 344 rural villages across all 10 regions (excluding Yerevan). The component cost was subsequently reduced to US$400,000 (see paragraph 22).
(b) Component 2. Several modifications were made: (i) subactivity Ideas Generation Mini Grants was removed from activity 2.1.(i) because of duplication with other activities within the project; (ii) establishment of VTC, and associated Innovation Matching Grants were added to Subcomponent 2.2 given the overwhelming reception of the GTC that showed early results of regional IT hub development; (iii) activity 2.1.(iii) ‘Innovation Brokerage Team remuneration’ was dropped as similar services were offered as part of the GTC and VTC; and (iv) subactivity ‘Managerial Secondments’ was dropped from activity 2.3.(i) because of shifts in the GoA’s priorities.
1.7 Other Significant Changes
21. There were three project restructurings processed, including a closing date extension by 18 months from December 31, 2014 to June 30, 2016.
22. #1 Level 2 restructuring, September 2011 - reallocation of funds (Report No. 64564). Reallocations were processed to respond to a detailed feasibility study that identified a US$3.5 million shortage in funding for the GTC, while significant private sector interests were identified to finance the national broadband network. As a result, funds were reallocated from Disbursement Category 2(a) ‘National Broadband Backbone and Government Network’ to Category 1 ‘Goods, works, consultants’ services, Training and Incremental Operating Costs’. In addition, the unutilized amount of US$2.95 million from the Project Preparation Advance was reallocated to Category 1.
23. #2 Level 2 restructuring, June 2014 - including a closing date extension (Report No. RES13152). In addition to changes in the project description (see section 1.6), the restructuring composed of the following changes: (a) a legal covenant11 was removed as project financing was no longer required for the Sales Force Representation Office in Silicon Valley; (b) detailed reallocations were made between disbursement categories in the Loan Agreement to reflect changes in scope of Components 1 and 2 as described in section 1.6, including the addition of
11 Section 1.A. Institutional Arrangements “furnish to the Bank an evidence showing a private sector co-financing is available for Part B.3 (a) (i) of the Project on the terms and conditions as set forth in the Sales Force Representation Office Manual.”
6
VTC; and (c) 18-month closing date extension was granted to accommodate new activities, establishment of VTC and rural Wi-Fi deployment, and to recover initial implementation delays.
24. #3 Level 2 restructuring, July 2015 - eligibility of disbursements for Venture Fund (Report No. Not recorded). A restructuring was processed to better define the eligible expenditure and to streamline the requirements related to the Venture Fund component of the project. It also included a reallocation of funds US$300,000 from the cost savings incurred from the matching grants component to the Refurbishment of the International Business Machines (IBM) Center for Innovative Solutions and Technology. The restructuring facilitated disbursements and significantly reduced complexity of the Venture Fund component (see section 2.2)
2. Key Factors Affecting Implementation and Outcomes
2.1 Project Preparation, Design, and Quality at Entry
25. Preparation and soundness of background analysis. The PAD reflects comprehensive efforts by both the GoA and the World Bank team to design a highly innovative project for growing the nascent innovation and entrepreneurial ecosystem in Armenia. The project design adequately took into account lessons learned from recent analytical work and operations in Armenia, and international experiences. For instance, the Enterprise Incubator LIL, Access to Finance for Small and Medium Enterprises (SMEs) Credit Line Project, and the Public Modernization Project in Armenia provided valuable lessons, including innovative schemes to promote the IT/knowledge- intensive industry, financial intermediary approaches of channeling the World Bank financing through commercial banks, and implementation of e-Government systems.
26. World Bank experts also reflected lessons learned from projects in other countries that had similar activities for the broadband PPP component (see section 1.1), and the Republic of Korea, India, and Mexico, among others that had some form of Venture Fund components though they were limited to non-lending TA or World Bank–executed grants. At the same time, the EIF had been building institutional capacity and knowledge through the implementation of programs that built upon the LIL project, and had also been proactive in exploring partnerships with the private sector and the governments (for example, Microsoft, India). A limited-scale Computer for All pilot was implemented successfully in 2009 and had sold 3,500 computers. This pilot helped to test the financing and sales mechanisms for the program. All of these activities and lessons learned helped inform the preparation of the project.
27. A Program for Human Resources Development (PHRD) grant of close to US$1 million was secured to finance preparatory studies, including broadband backbone design, spectrum policy, e-ID card design, Computer for All support, design of the GTC, design of Venture Fund, assistance for design of programs for IT industry development, preparation of the M&E framework, and a thorough economic analysis. The study on broadband carried out extensive consultations with the private sector and concluded at the time that the project will be leveraged to support the deployment of 600 km of fiber-optic broadband backbone throughout the country. Qualified consultants were hired to assist with the preparation of the bidding documents for the broadband network and the e-ID component which were deemed to have high risks of experiencing delays.
7
28. Assessment of project design. The sheer complexity and innovative nature of the project are notable. The project activities were carefully determined based on a clear government strategy and coincided with a wave of private sector investments in the country. The project was adequately designed to stimulate both the supply (Component 1) and demand sides (Component 2) of developing the Armenian e-Society by incorporating state-of-the-art and out-of-the-box approaches. The supply-side activities aimed to strengthen the digital foundation by increasing Internet and computer access across Armenia, and the demand-side activities were aimed to address the increased shift from an outsourcing model to a higher value-added entrepreneurship model to boost growth of the IT industry and enhance skills of Armenian youth. The establishment of the CA was fundamental to facilitate citizen’s access to services and for developing a digital foundation for improving efficiency, accountability, and transparency of government services.
29. The demand-side activities were ambitiously designed in a way that it would support technology entrepreneurs throughout the incubation cycle. Activities spanned across IT training programs, mentoring, idea generation, prototyping (Innovation Matching Grants), and start-up development (Seed and Early-stage Venture Fund), among others. In particular, the setup of a Venture Fund was a groundbreaking product/initiative to be implemented under a World Bank lending operation, and the first to be set up in Armenia.
30. Given the innovative nature of project activities and the evolving nature of the IT industry, it was expected that significant adjustments may be needed during implementation. However, in hindsight, the ambitious design most likely warranted a deeper dive assessment on the practicalities of implementing the project in the context of a World Bank lending operation, which may have contributed to better disbursement flows in the initial years. In particular, the preparation of the Operations Manual for the Venture Fund before Board approval may have helped identify solutions for resolving the fiduciary bottlenecks for establishing a new market-based financial instrument at earlier stages of the project (see section 2.2 for details). Disbursement categories in the loan agreement were too detailed and later posed multiple reallocations of funds by the implementation team.
2.2 Implementation
31. The project was implemented successfully in that it achieved (and in most instances surpassed) the PDO and intermediate targets. However, the project faced a number of implementation challenges particularly during the initial years. A midterm review (MTR) was held in April/May 2013, at which time the loan had disbursed only US$3 million (12.5 percent of the loan), and the project was rated Moderately Unsatisfactory because of serious delays across Component 2. In particular, critical adjustments were needed to adapt the project to the fast- evolving industry, shifts in Government priorities, and to fine-tune business models for the Venture Fund and the Computer for All Program which were complex and did not have equivalents in other World Bank–funded projects. Changes in government leadership had also affected the speed of decision making during the initial years.
32. The MTR reviewed each activity in detail and discussed options to restructure the project. The GoA and the World Bank agreed on a concrete set of milestones to be achieved within a six- month period, after which a decision on the restructuring would be made. This contributed to turning around project performance and to the successful restructuring of the project in 2014,
8
including an extension of the closing date by 18 months (see section 1.7). Another restructuring took place in 2015 to finalize disbursement mechanisms for the Venture Fund, after which all project activities were implemented and concluded by the project’s revised closing date. A summary of the key factors that affected implementation are the following:
a) Rapid advancements in broadband technologies and increased private sector investments. The first restructuring in 2011 reduced the project financing needed for the broadband component given increasing private sector investments (see section 1.7). By the end of 2012, private sector telecommunications providers were investing in infrastructure at previously unexpected levels as the market matured and technologies advanced, having reached a household broadband penetration of over 50 percent. These included the deployment of broadband infrastructure, through a variety of technologies, such as backbone fiber-optic infrastructure, and broadband wireless access through 3G mobile broadband and LTE cellular technologies, among others. This led to a decision by a commission chaired by the prime minister to realign the broadband component to focus solely on rural broadband (‘digital village’) aimed at demonstrating the socioeconomic impact of access to broadband, in last mile rural and isolated villages. Due to these changes, the regulatory TA to the Public Services Regulatory Commission was also deemed unnecessary, and was not implemented.
b) Redesign of Computer for All Program, including flexibility in offering of computing devices that are aligned with global technological shifts. The Computer for All Program started in December 2012. However, the program quickly ran into difficulties because of two main factors: (a) the complexity of the vendor, logistics center selection process, which led to technical specifications of computers that were largely obsolete by the time they reached the stores, and (b) the provisions of the master loan agreements with the banks that provided refinancing for only 50 percent of the costs of the computers sold. To resolve these issues, the program’s operations manual was radically simplified by removing the selection of vendor, logistics centers, which made any computer, including tablets and smartphones, eligible for consumer credit. The refinancing was also brought up to 100 percent. These adjustments, coupled with increased consumer awareness, resulted in accelerated sales and all project funds disbursed by the end of 2013. This was particularly relevant given the global technological shift from computers to tablets/smartphones. According to estimates by the German-Armenian Foundation (implementing partner), in 2013, desktops and laptop computers composed nearly 80 percent of sales under the program, which has gone down to about 40 percent to 50 percent by 2016.
c) Addition of VTC. A second technology center located in the city of Vanadzor was requested by the GoA and approved in the 2014 restructuring (see section 1.7). This addition was built on the preliminary successes of the technology center in Gyumri and in line with the Government’s strategy for economic development outside of Yerevan. The aim was to develop a foundation for Vanadzor -- the third-largest city in Armenia and the capital of the Lori Province in the northern part of the country -- to become the third technology hub in Armenia, and to create linkages to job opportunities particularly for the youth. This addition became one of the factors that contributed to the extension of the closing date. Given accelerated efforts, the
9
refurbishment of VTC was completed and became operational by the project closing in June 2016.
d) Need to adjust disbursement mechanisms for Venture Fund. The third restructuring in 2015 was necessitated by the need to adjust the disbursement model for the Venture Fund. The original model had treated the Venture Fund as a temporary intermediary that disbursed the World Bank loan proceeds in the same way as a grant fund would work, which is to ‘expend’ instead of ‘invest,’ and required that loan proceeds be fully disbursed by the beneficiary firms before the project’s end disbursement date. This approach was incompatible with the investment model of venture funds, and would have undermined its purpose of making capital commitments to firms as they achieve milestones then deployed as needed for capital improvement, working capital, and research and development (R&D). Furthermore, it was contrary to the intent of the project that aimed to establish a market-based financial instrument, from scratch, as a catalyst for attracting private investors and growing Armenian tech firms, which impact would go beyond the project period. Accordingly, a restructuring was undertaken so that when the Venture Fund called upon its investors, including the Government (under the project), to make a capital commitment, those monies were treated as fully disbursed under the World Bank Loan Agreement regardless of when the Venture Fund deployed them to pay its expenses or to invest in the firms. These changes significantly turned around implementation for the Venture Fund subcomponent.
e) Changes in Government leadership. Changes in leadership at the Ministry of Economy also contributed to delays. During the five years of project implementation, there were four changes of the ministers and deputy ministers. With each incoming minister the new administration required time to familiarize itself with the project, and in some instances put on hold targeted activities for a prolonged period particularly during the early stages of project implementation. Implementation was also affected by the policy introduced by the prime minister’s office, which required all foreign aid project procurement to be reviewed at the prime minister’s level.
2.3 Monitoring and Evaluation (M&E) Design, Implementation, and Utilization
33. Design. The M&E results framework was prepared under the PHRD grant, and arrangements for M&E were made specific in the PAD. It called for data collection and reporting to be the responsibility of the implementing agencies, and that the Ministry of Economy will provide the World Bank with a comprehensive M&E report ahead of every supervision mission, at least semiannually. It also called for the data collection and reporting to be automated where possible, and be written into the terms of reference and technical specifications to be implemented by the vendor(s) to enable efficient collection of data. There were 4 PDO indicators and 16 intermediate outcome indicators in the original results framework. Despite the complexity, the inter-relationships of the intermediate outcome indicators with the PDO were generally well defined in the PAD, and were supportive of the components/subcomponents at the time of project preparation.
10
34. There were some minor shortcomings in the results framework. For instance, the PDO indicator ‘IT/ITES Employment (number of people)’ posed a challenge in measuring the project’s attribution to the achieved outcome for this indicator. The intermediate outcome indicators included the ‘number of manpower trained’, among other relevant indicators, but this did not provide a clear linkage to increasing employment in IT/ITES. Another minor shortcoming is that the ‘number of direct beneficiaries (number), and of which female (percentage)’ was not included in the results framework. The World Bank guidelines require the adoption of this indicator for investment projects that have an approval date after July 1, 2009, and a mechanism to measure the project’s direct impact aggregated by gender would have been useful.12
35. Implementation and use. A consulting firm (KPMG) was hired under the project to assist the Ministry of Economy with preparing semiannual performance indicator updates. M&E was downgraded from Moderately Satisfactory to Moderately Unsatisfactory because of delays with hiring of the firm, but was subsequently signed and finalized (with amendments to respond to the first restructuring) in October 2013. It is worth noting that, until the MTR in April/May 2014, the project had only disbursed 12.5 percent of the loans, thus the delayed hiring of the firm had negligible impact on tracking progress. Five semiannual performance reports were prepared and submitted to the World Bank between 2014 and May 2016 and were found generally to be of good quality. M&E was upgraded in the March 2014 Implementation Status and Results Report (ISR) and remained Moderately Satisfactory until the project closing.
36. The 2014 restructuring made comprehensive efforts to adjust the results framework to the modified components. These included fine-tuning of the scope for the outcome indicator ‘Residential broadband household penetration (added “in villages”)’ and the deletion of four intermediate outcome indicators, including ‘Export of IT/ITES services’ and ‘Labor productivity in IT/ITES sector’ which were deemed as unobservable indicators for measuring project results.
37. The indicator ‘Vanadzor Technology Center (VTC) facility is ready for occupancy’ was added as a PDO indicator in the 2014 level 2 restructuring (see section 1.7). However, it is obvious that this output indicator did not belong as a PDO indicator and should have been added as an intermediate outcome indicator. The project team cited to the ICR authors that the team considers this an inadvertent error that occurred during the comprehensive restructuring efforts, and was not meant to be added as a PDO indicator. Nonetheless, this remained uncorrected until the project closing.
2.4 Safeguard and Fiduciary Compliance
38. Safeguards. The project was classified as Category ‘B’ and triggered OP/BP 4.01 Environmental Assessment, OP/BP 4.04 Natural Habitats, and OP/BP 4.11 Physical Cultural Resources. At the time of project preparation, an Environmental and Social Management Framework for the broadband backbone infrastructure and a simplified Environmental Management Plan for the GTC were prepared and disclosed. The restructurings in 2014 and 2015 did not trigger any additional safeguards policies. The EIF had dedicated qualified staff working on safeguards issues, and safeguards compliance remained satisfactory throughout the project life.
12 OPCS guidelines, http://intresources.worldbank.org/INTOPCS/Resources/380831-1177599583121/3719948- 1248469457617/6332446-1330008912550/BENEFICIARIES-CSIGuidance2-21-12.pdf.
11
39. Procurement. The procurement officer at the FFPMC had substantial experience with the World Bank’s procurement and processes, and procurement administration under the project was generally good. Ratings in the ISRs were either Satisfactory (ISR#1–3) or Moderately Satisfactory (ISR#4–12). Interviews carried out for this ICR identified that a number of procurement delays that contributed to the Moderately Satisfactory ratings pertained to resolving substantive technical issues rather than procurement itself.
40. Financial management (FM). FM was managed by FFPMC staff who were highly competent and experienced in the World Bank’s procedures. Internal controls within the FFPMC were adequate, and financial reports and project audits were submitted on time and were of high quality. Counterpart funding was also provided by the Government on time. FM remained Highly Satisfactory or Satisfactory throughout the project.
2.5 Post-completion Operation/Next Phase
41. The project was designed to be fully self-sustainable at completion. The project did not have a project implementation unit (PIU) and was implemented by regular staff at the EIF, EKENG, and FFPMC. Operational costs of the EIF, including the GTC and VTC, remain sustainable from state budget as well as revenues earned from business consulting, trainings, and facilities services. The EKENG has earned substantive revenue from issuance of e-IDs and from the provision of other e-Government services (50 percent goes to the state). With respect to individual programs, the Computer for All Program is sustained under the revolving fund and is managed by the German-Armenian Foundation. In addition, ICT vendors, banks, and telecom operators that saw the benefits are scaling up similar programs using its own fund. The Venture Fund has successfully contributed to increasing investor confidence in Armenia, and an additional three investments (total US$900,000) were in the pipeline at the time of this ICR. Investor portfolio included domestic as well as Russian, U.K., and the Armenian diaspora around the world. The sustainability of the tech labs/innovation centers are assured by continuing Government, donor, and private sector support. Finally, the EIF had signed a contract with Ucom (mobile network operator) which has committed to provide free Wi-Fi Internet services to the 344 rural villages for a three-year period until 2019, during which the potential collection of user fees will be assessed.
42. In September 2016, structural changes were implemented in which the Ministry of Economy and the Ministry of Transport and Communication were renamed the Ministry of Economic Development and Investment and the Ministry of Transport, Communication, and Information Technologies, respectively. At the time of the ICR, it remains unclear to which ministry the mandate for developing the IT industry would belong to, which may or may not affect future directions of project-related activities. The Trade Promotion and Quality Infrastructure (P146994), currently being implemented by the Ministry of Economic Development and Investment, has a US$19.3 million component on PPPs to promote research, skills, and cluster development in which the EIF is the implementing agency for proposals related to IT and engineering. Select activities for further growing the IT/knowledge-intensive industries are expected to be implemented under the project. The EIF is also exploring additional funding from the European Union and U.S. Agency for International Development, among other donors, to scale up some of the high-impact activities for accelerating growth of technology entrepreneurship.
12
3. Assessment of Outcomes
3.1 Relevance of Objectives, Design, and Implementation Rating: High
43. The project remains highly relevant to the current World Bank Group CPS for Armenia 2014–2017. The CPS specifies two clusters of outcomes, ‘supporting competitiveness and job creation’ and ‘improving efficiency and equity in social services’, of which the project firmly supports the former. The project also remains highly relevant to the recently published World Bank report ‘World Development Report Digital Dividends 2016’ which provides strong linkages between broadband access, growth and jobs. For instance, access to high speed broadband promotes inclusion of firms in the world economy by expanding trade, raising the productivity of capital, and intensifying competition in the marketplace. It also brings opportunities to the citizens by creating jobs and transforming public service delivery. However, for countries to reap these benefits, they need to be complemented by a strong enabling environment, including skills.
44. Armenia’s Development Strategy 2014–2025 continues to support the development of e- Society as part of its growth strategy. The Armenia’s Development Strategy clearly states that competitiveness and innovations are key to achieving the vision of transforming Armenia into a knowledge-based economy, and stresses the importance of ICT-based innovation that has been built upon the e-Society project.
45. The project design was adequately focused on strengthening the underlying infrastructure and enabling environment for e-Society development, which included both hard and soft elements, and training programs to increase Armenia’s wealth of human capital. The project design incorporated meaningful instruments, institution building, and development approaches that were critical to the country’s development. Yet, when dealing with fast-changing new knowledge-based, innovative economy activities, not all elements of the original design remained relevant. The project’s plan to support building of a broadband backbone became obsolete when market forces kicked in, and the private sector decided to do the same. Accordingly, those loan proceeds were reallocated to replicating the GTC model in the city of Vanadzor to scale up regional economic development efforts. The Venture Fund is now firmly established in Armenia backed by private investors, along with tech labs and other institutions, making the relevance of the project design, as adjusted through restructurings, High.
3.2 Achievement of Project Development Objectives Rating: High
46. The ICR considers that the PDO contains two main outcomes: (a) to address constraints to competitive e-Society and enterprise innovation in Armenia and (b) to strengthen the underlying infrastructure and enabling environment.
47. Although a new PDO indicator ‘Construction of VTC building is complete’ was added late in the project in the 2014 level 2 restructuring, it was obvious that it did not belong as a PDO indicator, and should have been added an intermediate outcome indicator (see paragraph 37). More importantly, this additional indicator had no logical effect on the assessment of outcomes, as the original outcome indicators already solidly support a rating of High for efficacy, and the new
13
indicator, which also was fully achieved, does nothing to alter or qualify this finding. In this light, the ICR concludes that a split evaluation is not warranted for the purpose of evaluating the outcomes of the project.
PDO 1: Address constraints to competitive e-Society and enterprise innovation in Armenia
48. This PDO is addressed by activities that were carried out under Component 2: Fostering Enterprise Innovation.
49. Although it is not made explicit in the PAD, the ICR evaluates that the two original PDO indicators were related to this outcome. These indicators remained unchanged throughout the project and were the following: (a) development of knowledge and technology-intensive industries (ratio of IT/ITES sector revenue in GDP) and (b) IT/ITES employment (number of people).
50. According to the PAD and subsequent restructurings, this outcome was supported by seven intermediate outcome indicators whose targets were either achieved or surpassed by project closing, with the exception of the venture fund investments which were short by one investment (92 percent achieved). These include (a) number of matching grants approved; (b) number of projects financed through the Venture Fund; (c) number of projects prepared and presented to potential investors; (d) number of enterprises benefitted from GTC; (e) occupancy rate of GTC; (f) number of sales and investments generated by the representative office; and (g) number of manpower trained and certified under the project.
51. Project outputs relevant to these outcomes include the following (see annex 2 for details):
(a) Three calls for Innovation Matching Grants were announced during 2015, and a total of 31 grants were awarded to technology start-ups in Yerevan, Gyumri, and Vanadzor.
(b) The seed and early-stage Venture Fund was set up, became fully operational in 2015, and provided financing to a total of 11 projects (8 companies).
(c) A total of 210 projects were prepared and presented to potential investors, including through the U.S. Sales Representative office and conferences hosted by the GoA.
(d) The GTC was refurbished, became fully operational, and provided office space/trainings/business consultancies to over 102 multinational and local start- ups/firms (occupancy rate reached 99.2 percent by June 2016)
(e) The Armenian U.S. representative office in Silicon Valley was setup and became operational in 2012, and generated over US$3 million in sales and investments to Armenian tech companies. Establishment of IBM-Innovation Solutions and Technologies Center (ISTC) was also negotiated through the representative office.
(f) IT and entrepreneurial skills trainings were provided to a total 16,572 personnel through the GTC, VTC, and mLAB, among other innovation acceleration centers that were set up fully or partially funded by the project.
14
52. The specific outcomes of the PDO are as follows:
a) Development of knowledge and technology-intensive industries (ratio of IT/ITES sector revenue in GDP). The industry’s contribution to GDP rose from just 1.5 percent in 2010 to 5.29 percent by the project closing,13 greatly exceeding the end target of 2.2 percent. Currently, the IT/ITES sector is one of the fastest growing in the Armenian economy. It created US$559.1 million in revenue in 2015, an increase of 17.7 percent from the previous year. Today around 540 ICT companies operate in Armenia, in which the number of companies outside of Yerevan is growing year by year, because of the development of critical infrastructure, particularly in the Shirak and Lori regions with the establishments of GTC and VTC (see Annex 3). In 2015 alone when project activities were most active, 70 new ICT companies were established in Armenia, out of which 8 were investees of the Venture Fund and about 20 were recipients of Innovation Matching Grants in Yerevan, Gyumri, and Vanadzor. This growth was also fueled by project activities, including the trainings provided in GTC, VTC and the various technology centers. In particular, the GTC is already flourishing as a second technology hub in Armenia and is attracting local and multinational tech firms and research institutions as tenants that are helping to mentor and hire local youth as well as local entrepreneurs. 14 This demonstrates the catalytic role the project is playing in developing the Armenian IT industry. The Venture Fund has also contributed to increasing investor confidence in Armenia.
b) IT/ITES employment (number of people). IT/ITES employment grew dramatically from 5,190 to 12,685 during the project period, greatly exceeding the target of 6,900. While acknowledging the fluid nature of technology entrepreneurship, the EIF estimates that project activities have contributed to the creation of about 350–400 direct jobs in Armenia, out of which 220–250 are IT/ITES specialist jobs. While the direct contribution of the project to IT/ITES employment growth during the period is seemingly modest at about 2.9 percent to 3.3 percent, the indirect and longer-term impact of the project on employment is notable. For example, firms that benefitted from the Innovation Matching Grants have collectively employed more than 150 engineers and IT specialists in Yerevan, Gyumri, and Vanadzor and the numbers are continuing to grow. In particular, the GTC has provided tremendous opportunities for youth in a city where youth unemployment had previously reached 52 percent, according to reports.15 Further, the 11 investments (8 companies) of the Venture Fund have grown to employ over 70 engineers and IT specialists in Armenia, and is continuing to grow.16 Finally, trainings and incubation support that are being provided in the GTC, VTC, and the various tech centers are also boosting the pipeline of potential start-ups and are linking workers to employment opportunities.
13 The EIF estimates that it has increased to 6 percent at the time of the ICR. 14 Tenants include D-link, National Instruments, Pomegrante, Global Cleantech Innovation Programme, American university of Armenia, among many others. 15 http://www.epress.am/en/2015/06/03/high-unemployment-in-gyumri-is-the-main-cause-of-migration-among- youth-study.html. 16 Companies include SoloLearn, WiCastr, CodeFights, News Deeply, ggTaxi, SkyCryptor, FlatClub, Menu Group (UK) Limited, covering mobile education, communication, e-commerce, and digital media.
15
PDO 2: Strengthen the underlying infrastructure and enabling environment
53. This PDO is addressed by activities that were carried out under Component 1: E-Society Infrastructure Development.
54. The ICR evaluates that two original PDO indicators were related to this outcome: (a) Access to Internet services (% population using Internet) and (b) Access to computers (% population using a computer). The 2014 restructuring added one additional PDO indicator related to this outcome: (c) Construction of VTC building is complete (see section 2.3 for background of this addition to the PDO).
55. The achievement of this part of this PDO is considered ‘High’. The two original indicators surpassed the target values by the end of the project, reaching 73.36 percent for access to Internet services and 57.4 percent for access to computers, both against the target of 50 percent. The additional indicator (c) also achieved its target value of completing construction of VTC by June 2016.
56. According to the PAD and subsequent restructurings, this outcome was supported by five intermediate outcome indicators whose targets were all achieved or surpassed. These include (a) residential broadband household penetration in villages under the project scope; (b) price of retail broadband access as a percentage of average monthly salary; (c) number of electronic certificates issued; (d) percentage of households owning a computer; and (e) number of households owning a computer purchased under the Computer for All Program.
57. Project outputs relevant to this outcome include the following (see annex 2 for details):
(a) Wireless (Wi-Fi routers) Internet access devices were installed in 344 villages in rural villages of Armenia, providing free Internet access to over 172,000 people.17 Villages included high-altitude and mountainous villages at the borders, with population of up to 1,000 people.
(b) The Computer for All Program was launched and provided computers/devices to 37,171 households (about 40,000 devices) across all regions of Armenia, greatly exceeding the target of 15,000. The program offered consumer loans at a low interest rate of 16 percent compared to 20 percent to 25 percent offered in the markets. The program initially focused on desktop/laptop computers but expanded to include Internet-enabled tablets and smartphones, adjusting flexibly to the evolving market (see section 2.2)
(c) The CA was set up in 2013 and servers, database, and software were developed/installed for the CA. This combined with significant public awareness resulted in issuance of 1,022,262 electronic certificates by the end of the project, far exceeding the target of 5,000. This signifies that nearly one-third of the population are now engaged in e-transactions.
17 EIF information.
16
58. The specific outcomes of the PDO are as follows:
a) Access to Internet services (% population using Internet). Access to Internet services increased from 15 percent of the population in 2010 to 73.35 percent by the end of the project, greatly exceeding the target of 50 percent. Calculating from the available data,18 the estimated increase in population that gained access to Internet services during the period is 1,768,968, out of which at least 9.7 percent (172, 000 people) were directly attributed to the project. The EIF notes that the free services are being actively used in the villages, with an average monthly usage accounting to about 6,500 GB (37.8 MB per capita).19 The growth in Internet penetration was also driven by a dramatic price drop in retail broadband access as a percentage of monthly salary, which was 14 percent in 2009 and fell to 1.95 percent by 2016. This was a result of the improved broadband infrastructure and generally increased competition in the sector during the project period. As a result, residential broadband household penetration in villages rose from 8.3 percent in 2014 to 41.6 percent in 2016, greatly exceeding the target of 30 percent. Internet-enabled devices that became accessible to the citizens under the Computer for All Program also contributed to the increased number of Internet users.
b) Access to computers (% population using a computer). Access to computers increased from 15 percent of the population in 2010 to 57.4 percent by the end of the project, greatly exceeding the target of 50 percent. The achievements of the Computer for All Program significantly helped to boost computer penetration in Armenia, as well as in the households. Now, 57 percent of the total households have access to computers, a significant increase compared to just 10 percent in 2010. Using the number of households in Armenia from the 2011 population census (763,584),20about 358,885 households gained ownership of a computer between 2010 and 2015, of which 10.4 percent (37,171 loan agreements/households) were beneficiaries of the Computer for All Program.
c) Construction of VTC building is complete. The refurbishment of the VTC was completed by June 2016, and achieved the target. At the time of this ICR, the VTC had already started to provide IT trainings and incubation support to entrepreneurs and students. It was housing one tenant (Fligh Tech Gateway) that employed three technical specialists, and an additional three to four firms were in the pipeline. The official opening of the VTC took place in October 2016.
59. As evidenced above, significant relevant tangible outcomes have emerged from the project and greater impacts are expected in the future. It could be argued that achievements of both PDO outcomes were also influenced by exogenous factors such as the rapid growth in the use of ICTs globally and the rapid evolution of the sector in general, including the introduction of 3G/4G
18 World Bank data record population of 2,963,496 (2010) and 3,017,712 (2015). About 15 percent of 2010 population is 444,524, and 73.35 percent of 2015 population is 2,213,492. Estimated increase in access to Internet services is 1,768,968. 19 6,500,000 MB/172,000 people = 37.8 MB per capita. According to available information, 50 emails (no attachments) per day would account to around 40 MB of data usage per month. 20 National Statistical Service of the Republic of Armenia, http://armstat.am/en/?nid=517.
17
mobile broadband technologies that continued to make ICT services more accessible at lower costs. However, it should be noted that before the project, Internet penetration in Armenia (15 percent) was lower than neighboring country Georgia (20.1 percent), but by 2015 Armenia had reached 58.2 percent while Georgia’s growth remained limited and penetration reached only 45.2 percent. This signifies the project contributions to achieving the PDO, particularly with respect to PDO 2: Strengthening the underlying infrastructure and enabling environment.
3.3 Efficiency Rating: Substantial
60. The overall efficiency is rated as Substantial. The ICR does not calculate an economic rate of return, but Annex 3 assesses economic benefits that can be quantified, including a number of socioeconomic benefits that were identified in the PAD. Even with a large margin of error, the costs of the project appear negligible compared to the broad economic and social benefits achieved under the project. This section examines efficiency from the standpoints of project implementation and from efficiencies created by project activities.
61. Implementation. Despite the initial delays and one project extension by 18 months, the project was able to accelerate implementation after 2014 and achieve positive results of the PDO and intermediate outcome indicators. The project remained within the original budget envelope (97.09 percent IBRD funds disbursed), and leveraged US$6 million from the GoA financing. In addition, project activities contributed to mobilizing more than US$45 million in co-investments from private sector partners, which includes over US$26 million private capital raised by the Venture Fund, at the time of this ICR (see Annex 1). There were undisbursed loan proceeds of US$700,000, including US$400,000 from the Venture Fund, but this stemmed from changes in Forex rates between the U.S. dollar and the Armenian dram that went from 1 US$ = AMD 358.21 (appraisal) to AMD 476.63 (closing).21 The ICR concludes that the delayed implementation at the beginning of the project and the closing date extension are not considered major shortcomings, and can be justifiable as a sign of growth, learning, and adaptation required for implementing such an innovative project that broke the grounds for future World Bank-funded operations. Given the above, the ICR rates the implementation efficiency as Substantial.
62. Project activities. The activities of the project demonstrated good value for money. For instance, the rural broadband component (total cost US$256,570) enabled 172,000 previously unserved people with access to Internet services. This accounts for a unit cost of just US$1.5 per person, while the project successfully secured the private sector to provide free Wi-Fi Internet services to the population until 2019. Further, a total of 16,572 people were trained in IT and entrepreneurial skills under the project in the GTC, VTC, and the various technology centers implemented under PPP arrangements with Microsoft, National Instruments and the Government of India, among others. This was a cost-effective way to maximizing results and for ensuring sustainability beyond the project. In addition, the e-ID (Digital Citizen) Program disbursed a total of about US$1.56 million (US$1.17 million IBRD and US$0.39 million GoA), under which over 1 million certificates were issued by project closing, and is now supported fully by the fees collected and regular state budgets for the EKENG. The program is producing major benefits in improving tax services, land administration, business registration, and the pension system, among
21 http://www.xe.com/currencycharts/?from=USD&to=AMD&view=5Y.
18
others, in Armenia by replacing paper-based government functions for both the Government and its customers. Individuals and businesses are now able to complete registering and licensing online and pay for the service if they have access to computers and credit cards.22 In summary, the ICR rates the efficiencies/value created by the project activities as High.
3.4 Justification of Overall Outcome Rating Rating: Satisfactory
63. The project’s high achievement of its PDO, its high continued relevance to economic and social development of Armenia, and its substantial efficiency merit an overall Satisfactory rating of outcomes.
3.5 Overarching Themes, Other Outcomes and Impacts
(a) Poverty Impacts, Gender Aspects, and Social Development
64. The project had no overt poverty objectives and did not explicitly address gender either in design or during implementation. However, it is noteworthy that the project has helped enhance skills and employment opportunities for previously disadvantaged youth in the cities of Gyumri and Vanadzor that were suffering from high unemployment and a lack of industry and jobs. A gender impact assessment could have been useful; however, it was still an emerging agenda within the World Bank and had not been incorporated in the project design. Informal interviews carried out with the German-Armenian Fund, an implementing partner of the Computer for All Program, identified that nearly half of the loans signed under the Computer for All Program were signed by a female member of the household (age group unknown), and stemmed across all 11 regions (including Yerevan) of the country. Although they were not quantitatively tracked, the EIF provided a rough estimate that about half of the trainees and entrepreneurs supported under the project were estimated to be female, and interests by female youth to participate in the programs are continuing to grow. In the rural areas, anecdotal evidence suggests that increased Internet access and computer/device penetration have allowed the previously isolated elderly population to communicate with families abroad through Skype and other platforms at affordable costs. This is also because of the Computer for All Program that provided targeted trainings for the elderly population.
(b) Institutional Change/Strengthening
65. The project significantly improved the technical and operational capacities of the staff in the EIF, EKENG, and several other stakeholders in their respective areas. The EIF grew its capacity to assist start-ups/IT companies in the areas of business development, skills training, and start-up formation and entrepreneurship. The EIF now serves as a major channel for linking investors with the Armenian IT/knowledge-intensive firms, and is providing consulting services to other countries, sharing the experiences gained from the project (for example, Jamaica, Montenegro, Moldova). Through the project, the EKENG has continued to develop its capacity as the CA. This has helped lay the foundation for growing its role as overall coordinating agency for e-Government in Armenia. For instance, the e-ID system has been integrated with many e-services including digital signature, tax service, e-payment, e-banking, electronic system of the Real
22 EKENG website, https://www.ekeng.am/en/certificationauthority/.
19
Property Cadaster, State Population Register, licensed persons’ reports, and State Register of Legal Entities. Finally, interviews carried out for the ICR revealed that project activities have contributed to transforming the business models of ICT vendors and banks selected under the Computer for All Program, as they are now continuing the program under their own funding.
(c) Other Unintended Outcomes and Impacts (positive or negative)
66. The project achieved important unintended outcomes. Interviews carried out for the ICR revealed the creation of technological innovations that are transforming the Armenian economy across sectors and region. For example, Digital Pomegrante, one of the GTC tenants, has launched Digital Gyumri, an app that is helping develop the tourism industry in Gyumri. The app has collaborated with museums, hotels, retail shops, restaurants, and even the library, and now tourists are able to readily access information about the city. Further, investees of the venture fund are providing apps and services that range from e-commerce, digital media to transport, and start-up teams in Vanadzor are in the process of developing apps to match Armenian youth to online work opportunities (for example, ITfreelancer.com, AvatarMe). Tumo (tenant of the GTC), an innovation center for children of ages 12–18, is providing IT trainings/creative space for hundreds of children and is helping develop a strong pipeline of talent for the region. Finally, GTC has helped to create a foundation for unintended foreign investments in Gyumri. For example, a Taiwanese company, D-Link, a large manufacturer of computer networking equipment, has established its Research & Development Center right next to the GTC building, taking advantage of the ability to access IT specialists and engineers that have been trained in GTC. These are just some of the examples of the project’s indirect impacts.
3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops
67. Beneficiary surveys and stakeholder workshops were not carried out as part of the ICR, but informal interviews were carried out with targeted beneficiaries, which are included in annex 5. Comments raised by InecoBank, a participating bank in the Computer for All Program, are also included.
4. Assessment of Risk to Development Outcome Rating: Low
68. The risk of not maintaining the development outcomes is rated ‘Low’. The contributions of the project to the sustainable strengthening of Armenia’s IT/knowledge-intensive industries are reflected in the overall growth of the industry whose revenue reached US$559.1 million in 2015, a 17.7 percent growth from the previous year.23 The Government’s sustained commitments to industry development is also clearly outlined in Armenia’s Development Strategy 2014–2025, which represents a further evolution of the Government’s approach. These include (a) continuation of establishment of technoparks, incubators, and other IT infrastructures being implemented through PPP toolsets; (b) support to universities in implementation of modern curricula and acquisition of necessary laboratories, through promotion of their cooperation with private sector; (c) implementation of favorable tax policy promoting export potential in the sector; (d) implementation of state‐support programs for SMEs and start‐ups involved in the IT sector,
23 Armenian ICT Sector, State of Industry Report 2015.
20
including access to markets, professional trainings, and ensuring access to information; and (e) direct support to IT companies offering innovative products or services. The ICR team was informed that the EIF is receiving adequate budget allocations, and the wide base of ownership of the various programs across a range of public and private sector stakeholders helps assure that it will continue to be well sustained.
5. Assessment of Bank and Borrower Performance
5.1 Bank Performance
(a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory
69. The World Bank team was co-led by seasoned specialists in ICT and private sector development that brought innovative international experiences to the design of the project. The team successfully mobilized a client-executed PHRD grant, which contributed significantly to preparatory activities, including technical feasibility studies, safeguards, M&E, and economic analysis. The project design was complex, but this reflected the strong preference of the client for a comprehensive approach to supporting e-Society development. The multiplicity of components and disbursement categories required that funds be transferred across numerous categories during implementation. However, the need for ex ante adjustments of these in the Loan Agreement was not fully foreseen at the time of project preparation. In hindsight and given the ambitious nature of the project, more attention could have been placed on the fiduciary aspects of the project, which may have contributed to smoother implementation in the initial years. The results framework was complex and required some adjustments during implementation. Based on these factors, the World Bank’s performance in ensuring quality at entry is considered Moderately Satisfactory.
(b) Quality of Supervision Rating: Highly Satisfactory
70. The World Bank team continued to be co-led by seasoned ICT and private sector development specialists until the project closing. Despite changes in the task team leader (TTL) once, and co-TTL twice, the continuity of the program was maintained through adequate hand over and continuity of core team members. The team fielded 11 supervision missions between loan approval and closing, averaging two missions per year. Comprehensive aide memoires were prepared after each mission, and a total of 12 ISRs were filed by the end of the project. The World Bank’s implementation support also included constant remote support, proactively responding to project needs and adapting to complex challenges that arose during implementation (see section 2.2). The team was consistently engaged in providing TA on substantive matters, including on fiduciary matters and safeguards, and technical experts were mobilized at critical timings including mobilization of venture fund experts and civil engineering consultants for the GTC and VTC. Client representatives cited to the ICR team the important role of the World Bank team in resolving complex issues to ensure that the project’s objectives are met. The MTR in April/May 2013 identified specific bottlenecks and adjustments needed to accelerate implementation, and subsequent project restructurings were significant in bringing the project back on track, and to the high achievement of outcomes. The minor shortcoming in the results framework (for example, incorrect addition of PDO indicator) is considered a negligible error compared to the overall
21
performance. Based on these factors, the World Bank’s performance in supervision is considered Highly Satisfactory.
(c) Justification of Rating for Overall Bank Performance Rating: Satisfactory
71. Overall, Bank performance is rated Satisfactory reflecting the two dimensions: quality at entry and supervision/implementation support. Despite some shortcomings in quality at entry, significant resources and technical expertise were proactively allocated during supervision to assist the client in adjusting implementation to new circumstances and resolving complex issues under challenging circumstances including on technical matters, safeguards, FM, and procurement.
5.2 Borrower Performance
(a) Government Performance Rating: Moderately Satisfactory
72. The project received strong government commitment to the PDO, which is illustrated by the fact that the GoA invested US$6 million in cofinancing, representing 25 percent of the total project costs. Notwithstanding, the GoA through the EIF, EKENG, and FFPMC financed part of the project’s operating costs not registered as project cofinancing but under the Government’s own budget (for example, staff costs). Nonetheless, records indicate that the lack of continuity of leadership at the Ministry of Economy contributed, to a certain extent, to the slow implementation during the first half of the project. Major delays were also caused by the new procurement policy that was introduced by the prime minister’s office (see section 2.2). Given the above, the Government’s performance is considered Moderately Satisfactory.
(b) Implementing Agency or Agencies Performance Rating: Highly Satisfactory
73. The performance of the implementing agencies is considered Highly Satisfactory. The EIF provided strong technical and operational leadership for the project’s design and implementation, and was responsible for overall coordination, M&E, communications, and safeguards. The EIF made major strides in efficiency, management, and oversight under difficult circumstances, and sustained its commitment to achieve the high development outcomes of the project. Despite initial delays because of various factors (see section 2.2), project implementation advanced at an accelerated pace from 2014 until the project closing. The EIF’s focus on extensive consultations with private sector partners, project beneficiaries, and the Government throughout the project life were also fundamental to mobilizing private investment and promoting the project’s long-term sustainability. Compliance on safeguards aspects remained satisfactory, and all project covenants were complied with. The FFPMC was highly competent and ensured compliance on fiduciary aspects of the project, submitting reports and audits efficiently and on time. The EKENG’s commitment to the Digital Citizens Program led to the good results achieved for the subcomponent at early stages of the project. Given the above, implementing agencies’ performance is considered Highly Satisfactory.
(c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory
22
74. Overall, borrower performance is rated Satisfactory reflecting the two dimensions: the Government performance and implementing agency performance. Although the Government’s performance is rated Moderately Satisfactory, the strong performances of the implementing agencies and the outstanding results achieved under difficult circumstances warrant an overall borrower performance of Satisfactory.
6. Lessons Learned
75. A number of lessons emerge from the project implementation experience:
(a) Introduction of new and catalytic activities that have longer-term impacts rather than short-term is not usually a big sell but can have enormous impacts that go beyond the PDO, and within the boundaries of the project/country. The project broke the grounds in terms of adapting to the World Bank policies for creating a first Venture Fund in the country from scratch, and its success has had tremendous impact on increasing investor confidence in Armenia’s tech sector and on implementing VCs in World Bank projects. Projects that are active across regions are now referring to the instruments and lessons learned from the EIC to facilitate access to finance and promoting SME growth. These include the Croatia Innovation and Entrepreneurship Venture Capital Project (P152130), Promoting Innovation for Inclusive Financial Access (P146244), Innovate in India for Inclusiveness (P156241), and Piloting Equity Financing for SMEs in Morocco (P150928), among others. In this regard, it would have been useful if the project had carried out a beneficiary assessment and/or impact evaluation of the project interventions, which results would have helped inform the design of future projects.
(b) Both the GoA and the World Bank teams noted that the World Bank’s operational policies and guidelines are not optimal for the fast-evolving dynamic IT industry, and these need review. For instance, a more flexible approach could have been adopted in a relatively high-capacity context such as Armenia, and would have been a perfect pilot for applying agile principles to World Bank projects. This was particularly relevant for the multiple needs to amend the project description, fine-tune disbursement mechanisms, and reallocate funds across disbursement categories. The Program for Results instrument was not available at the time of project preparation, but going forward it may be a good choice of instrument for this type of project, as it would allow projects to link disbursement of funds directly to the achievement of results, instead of locking in the project to specific activities in such a fast-evolving sector.
(c) Establishing partnerships, not only in terms of mobilizing private sector resources but also collaborating with universities, industry associations, and research institutions, were fundamental for maximizing the impact and increasing sustainability of investments. For instance, the VTC is built on the premises of Vanadzor Branch of the State Engineering University, and IBM-ISTC and Armenian-Indian Center of Excellence for ICT are within the Yerevan State University campus. These have helped create synergistic effects with the university curriculum, prepare a strong talent pool in the country that is aligned with industry demands, and help mitigate the supply
23
gap of IT specialists. Together with the wide base of ownership of the various programs across a range of public and private sector, stakeholders are helping to assure that achievement of outcomes will continue to be well sustained.
(d) Finally, mainstreaming the project into existing institutions has ensured sustainability and continued buildup of project activities. Unlike many other World Bank–financed projects, PIUs were not created for the EIC, and core project tasks were carried out by regular staff of the EIF, EKENG, and FFPMC. This has helped institutional strengthening and avoid the loss of knowledge and expertise after project closure. It has also enabled the project to allocate sufficient resources to finance project activities, rather than escalate project management costs for hiring high-paying PIU consultants.
7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners
(a) Borrower/implementing agencies
76. Comments received from EIF were minor, mostly on correction of data, and have been incorporated in the ICR. The Borrower’s evaluation report is attached in Annex 7.
(b) Cofinanciers
There were no comments provided by co-investors.
(c) Other partners and stakeholders
There were no comments provided.
24
Annex 1. Project Costs and Financing
(a) Project Cost by Component (in USD Million equivalent)
Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions)
1. E-Society Infrastructure 13.8 (IBRD 11.04) 4.95 (IBRD 4.48) 35.87% (IBRD 40.58) Development
2. Fostering Enterprise Innovation 11.09 (IBRD 8.87) 20.51 (IBRD 17.35) 184.94% (IBRD 195.60) 3. Project Management 1.20 (IBRD 0.96) 1.85 (IBRD 1.47) 154.17% (IBRD 153.12) Total Baseline Cost 26.09 (IBRD 20.87) n.a. n.a. Contingencies + Front end Fee 3.85 (IBRD 3.13) n.a. n.a. Total Project Cost 29.94 (IBRD 24.00) 27.31(IBRD 23.30) 91.22% (IBRD 97.09)
Note: There were slight discrepancies between the appraisal estimates by component in the Project Description of the PAD and Annex 5 (Project costs). For the purpose of this Annex, the numbers in the PAD’s Annex 5 are used.
(b) Financing
Appraisal Actual/Latest Percentage of Source of Funds Estimate Estimate Appraisal (USD, millions) (USD, millions) Borrower 6.00 4.01 100% International Bank for Reconstruction and 24.00 23.30 97.09% Development Japan Policy and Human Resources Development 0.97 0.97 100% Fund (TF093097) Private Sector co-investments in project activities 1. E-Society Infrastructure Development — 2.19 24 — 2. Fostering Enterprise Innovation — 43.45 25 — ‐
24 Rural WiFi-$80,545, Computer for All- $2,105,263, 25 Venture Fund-$25,947,782, Innovation matching grants- $822,710, GTC- $1,523,580, VTC- $256,884, Skills development
(ISTC- 6,086,550; MIC-2,457,592; ANEL-6,354,742)- $14,898,884
25
Annex 2. Outputs by Component
Actual Accomplishments at the Time of ICR and Appraisal Plans Reasons for Deviation Component 1: E-Society Infrastructure Development 1.1 Nationwide Broadband Backbone and Government Network TA for telecommunications regulatory This activity was dropped when Component 1 1.1.1 framework and backbone implementation was restructured to include only rural Wi-Fi. Restructured in 2014 as a last mile rural broadband component; provided free Wi-Fi Nationwide broadband backbone and 1.1.2 Internet access in 344 villages with populations Government network development over 1,000, benefitting a total population of 172,000. 1.2 Support to Digital Citizen Program Trainings provided for the CA staff
1.2.1 TA for the CA and e-ID document system Equipment and TA financed for the e-ID system, modernization of State Population Register, and E-mail Armenia system The CA launched in October 2013 within the Ministry of Economy’s EKENG Hardware and software for the CA developed, 1.2.2 Establish the CA tested, and implemented CA staff financed (project coordinator, technical experts, system administrator) 1,022,262 electronic certificates issued 1.3 Computer for All Program Delivered loans for the purchase of desktop/laptop computers, tablets, and Line of Credit for the Computer for All smartphones to 34,491 households in 11 regions 1.3.1 Program of Armenia Additional 13,498 loans financed from the revolving fund Financed operating expenses (PR campaign, TA to implement the Computer for All 1.3.2 trainings, audits, M&E) implemented by the Program German-Armenian Foundation and the EIF Component 2: Fostering Enterprise Innovation 2.1 Financial Support for Innovation in Knowledge and Technology-intensive Firms Ideas Generation Mini Grants and Innovation Ideas Generation Mini Grants dropped in 2014 Matching Grants Operations Manual for Innovation Matching Grants completed Innovation Matching Grants call for proposals 2.1.1 Innovation Matching Grants for Vanadzor announced in Yerevan, Gyumri, and Vanadzor (added in 2014 restructuring) Total of over 200 applications received, and 31 firms selected for awards that totaled over US$1.2 million Operations Manual completed Venture Fund established in 2012 2.1.2 Seed and early-stage Venture Fund Total 11 projects (8 firms) financed with investments totaling over US$1.6 million Additional US$1.1 million in pipeline This component was removed in 2014 2.1.3 Innovation brokerage restructuring 2.2 Establishment of Gyumri Technology Center
26
Operations Manual completed The GTC refurbished and launched in January 2.2.1 Establishment of GTC 2014 Equipment, furniture, and technology infrastructure installed The GTC management team, training coordinator, and engineering supervisors hired, 2.2.2 Support to GTC operation and operating expenses covered 100 percent occupancy rate achieved at time of ICR (99.2 percent by project closing) Operations Manual completed VTC refurbishment completed, and center operational in June 2016 Business development manager hired Business development consultancy providing Added Establishment of VTC mentoring services to eight enterprises and helped establish three start-up teams Vanadzor start-up grants provided to four teams IT skills trainings provided to over 13,000 trainees Specialized laboratories set up for training, prototyping, and testing (for example, Engineering, Multimedia, Cleantech, Mobile Technologies) 2.2.3 GTC Technology Extension Program Skills training delivered to over 2,000 people, and business development consultancies provided to over 50 firms Summer school classes offered to school children Armenian IT sales representative hired in Silicon Valley and Armenian-U.S. representative office became operational in December 2012 Total of US$200,000 contracts signed Support to IT/knowledge-intensive industry 2.3 development Armenian IT Industry Report and Guide to IT Companies 2013, 2014, and 2015 published Total of 38 Armenian IT companies traveled to Silicon Valley in June 2016 to interact with potential investors Industry promotion and managerial 2.3.1 Dropped in 2014 restructuring secondment Following centers established under PPP approach (project financing indicated in italics): Regional Mobile Applications Laboratory for Eastern CIS (mLAB ECA) - Staff, renovation, and equipment Skills development and productivity 2.3.2 Microsoft Innovation Center (MIC) – Elevator improvement Armenian-Indian Center for Excellence in ICT - Furniture Armenian National Engineering Lab - Elevator IBM-ISTC - Staff, trainings, refurbishment, furniture Component 3: Project Management
27
Operating expenses for the EIF, EKENG, and FFPM Consulting firm hired for M&E performance indicators updates on biannual basis between This component financed necessary activities 2014 and May 2016 (five total) for adequate project management, including
M&E and PR communications campaigns. Project audits Communications and PR campaigns delivered for Computer for All, Rural Wi-Fi, Innovation Matching Grants, Venture Fund activities, GTC, and VTC, among others.
28
Annex 3. Economic and Financial Analysis
1. The PAD acknowledged that the project will have more significant positive impacts in the long run rather than in the short run. However, an economic and social impact assessment was carried out with an attempt to quantify some of these impacts, including net tax gains from growth in Internet penetration, economic impacts of broadband and Venture Fund Investments, and the project’s direct impact on jobs.
2. The analysis concluded the following:
a) Net tax gains from facilitated Internet penetration growth versus no intervention. Estimated increase in a 10-year time frame includes net industry revenue (US$17 million), Government tax revenue (US$27 million), and net present value for the Government (US$12.6 million at 10 percent discount rate).
b) Economic impact of broadband growth. This was estimated according to a World Bank study (10 percent increase in broadband penetration contributes to economic growth by 1.38 percent), concluding that if Armenia realizes a 15 percent increase in penetration, GDP growth will account for about 2 percent.
c) Economic impact of Venture Fund (on R&D). Assuming that the VC investment generates a 20 percent internal rate of return within seven years, a US$6 million VC investment could generate US$25–US$55 million cumulative volume of private R&D within the period.
d) Direct impact on jobs and IT/ITES revenues. It was estimated that US$2.5 million sales generated from the U.S. representative office will create 60–90 direct jobs, and facilitate increased linkages to exports from the sector.
3. The ICR did not replicate the same economic analysis given the rapid transformation of the industry at both global and national levels plus changes in the project scope, which have diminished relevance of many of the assumptions. Further, much of the core project activities were carried out in 2014–2015 and the longer-term impacts are yet to be seen. However, the ICR notes strong evidence in growth of the IT/knowledge-intensive industry in Armenia between the period 2010 and 2015, which were fueled by activities carried out by the project (see table3.1).
Table 3.1. IT Industry Growth and Jobs October 2016 (EIF 2010 2015 estimates) Industry contribution to GDP 1.5 percent 5.29 percent 6 percent Number of IT/tech firms 197 450 540 Local 125 288 n.a. Foreign-owned 72 162 n.a. Industry revenue (US$, millions) 148.8 425.5 500 Employment 4,960 15,000 16,000 Technical specialists 4,110 10,250 10,850 Management and others 850 4,750 5,150 Students in IT-related specialties 6,800 9,352 9,500 Source: Armenian ICT Sector: State Industry Report 2010 and 2015, EIF.
29
4. The industry now accounts for 5.29 percent of Armenia’s GDP compared to 1.5 percent in 2010, and preliminary estimates by the EIF note that this may have risen to 6 percent in 2016. The number of IT firms grew from 197 in 2010 to 450 by the end of 2015, and is estimated to have grown to about 540 at the time of this ICR. In 2015 alone, when project activities were most active, 70 new companies were established. By 2015, there were about 10,250 IT technical specialist employees26 in the country compared to just above 4,000 in 2010, and is estimated to have grown to 10,850 in 2016. The industry as a whole is estimated to have reached 16,000 jobs by 2016. ICR interviews with the EIF and industry stakeholders in the GTC revealed that salaries for IT specialists can be two to three times higher compared with other industries reaching US$300– US$500 per month even as a new graduate, contributing to an improved quality of life.
5. While acknowledging the fluid nature of tech entrepreneurship, the EIF estimates that project activities have contributed to the creation of about 350–400 direct jobs in Armenia, out of which 220–250 are IT specialists. For example, the US$1 million disbursed in Innovation Matching Grant awards led to 31 products/services being developed, 20 new companies being registered in Yerevan, Gyumri, and Vanadzor, and collective employment of more than 150 engineers and IT specialists. Further, the US$2.87 million of project proceeds invested in the Venture Fund and corresponding investments from private investors includes a portfolio of 11 investments (8 companies), which have directly employed over 70 engineers and IT specialists in Armenia.27 Interests in and capacity building in IT, entrepreneurial skills have been stimulated by the various innovation centers/tech labs and the two regional technology centers supported by the project. The GTC has also set a strong precedent for regional ICT development in the country.
6. The project’s indirect contributions to income-generating potentials are also tremendous, and are helping the previously unemployed labor force in Armenia. For example, ggTaxi (recipient of the venture fund investment) has helped over 200 people in Armenia gain access to earnings as contracted drivers; and Menu.AM, another VC investee, has hired over 1,000 delivery personnel as full-time staff. The average monthly salary for the delivery personnel is reportedly about AMD 100,000 per month (about US$208), which is still low compared to the average monthly salary of AMD 184,992 (about US$386),2829 but has contributed to improving individual livelihoods by providing regular earning opportunities. Calculating from available data, the unemployed labor force had reached 92,038 in 2014, out of which at least 1 percent were able to gain full-time employment at Menu.AM, assuming that most of them were not fully employed in 2014.30 This admittedly limited and simple comparison demonstrates the transformational socioeconomic impact the project is having on the Armenian economy and society.
26 Technical specialist jobs include software engineers, analysts, developers, and IT project managers, among others. 27 Companies include SoloLearn, WiCastr, CodeFights, News Deeply, ggTaxi, SkyCryptor, FlatClub, and Menu Group (UK) Limited, covering mobile education, communication, e-commerce, and digital media. 28 EIF confirmations with ggTaxi and Menu.AM. 29 National Statistical Service of the Republic of Armenia, November 18, 2016. http://www.armstat.am/en/?nid=126&id=08001. 30 World Bank data. Labor force in 2014 was 1,559,971, and the unemployment rate was 5.9 percent. This accounts for 92,038 people who were unemployed in 2014. Using available data, this increased to 5.9 percent of the total labor force in 2014, accounting for around 92,038 people.
30
7. The project’s impact has also extended to neighboring countries. ggTaxi now operates in Georgia and Belarus, contributing to improving transportation in these countries, and contributing to increasing Armenia’s exports of services, and further facilitating the growth of the country as a regional technology hub. Longer-term impacts of the project also include technology-enabled product innovations across sectors that are stimulated by the start-ups and individuals trained under the project. This may contribute to the further emergence of new firms, new sectors, and thus new jobs. According to a World Bank study, each job in the IT and ITES sector creates three to four jobs in sectors such as transportation, construction, health, entertainment, and others. 31 The maturity of the industry today is notable and is evidently seen as one of the most prospective sectors in the country by young people, which is evidenced by the 38 percent growth of students enrolled in IT specializations between 2010 and 2015 (see Table 3.1).
31 World Bank. 2010. The Global Opportunity in IT-based Services..
31
Annex 4. Bank Lending and Implementation Support/Supervision Processes
(a) Task Team members
Responsibility/ Names Title Unit Specialty Lending Alexander Astvatsatryan Consultant GGO03 Ani Balabanyan Senior Energy Specialist GEE03 Deepak T. Bhatia Lead e-Government Specialist GTIDR Kashmira Daruwalla Senior Procurement Specialist GGOGI Michael Edwards Lead Financial Sector Specialist GFM09 Karen Grigorian Senior Private Sector Specialist GTC10 Co-TTL Aleksan Hovhannisyan Senior Operations Officer ECCAR Grace T. James Temporary GTC06 Saman Karunaratne Finance Analyst WFAFO Artur Kochnakyan Senior Energy Specialist GEE03 CTRFC- Hannah M. Koilpillai Senior Finance Officer His Yevgeny N. Kuznetsov Consultant GTC03 Anarkan Lilly Counsel LEGCF Davit Melikyan Senior Public Sector Specialist GGO15 Tatevik Mnatsakanyan Public Information Assistant ECCAR Juan Navas-Sabater Lead ICT Policy Specialist GTI09 TTL Seda Pahlavooni e-Government Specialist GTI09 Sandra Sargent Senior Operations Officer GTI09 Randeep Sudan Adviser GTIIC Arman Vatyan Senior Financial Management Specialist GGO21 Anna L Wielogorska Lead Procurement Specialist GGO08
Supervision/ICR Supervision Sandra Sargent Senior Operations Officer/TTL GTI09 TTL Feyi Boroffice Senior Private Sector Specialist/co-TTL GTC10 Co-TTL Edgar Karapetyan Consultant Juan Navas-Sabater Lead ICT Policy Specialist GTI09 Prior TTL Ara Grigorian IT Analyst ITSCR Francis Joseph Skrobiszewski Consultant Irene Guadalupe Rubio Gonzalez Consultant Karen Grigorian Senior Private Sector Specialist GTC10 Prior co-TTL Mather B. Pfeiffenberger Operations Analyst GTI09 Alan Carroll Operations Advisor/Consultant GTI09 Yevgeny N. Kuznetsov Consultant GTC03 Seda Pahlavooni e-Government Specialist GTI09 Garik Sergeyan Senior Financial Management Specialist GGO21 Darejan Kapanadze Senior Environmental Specialist GEN03 Lela Shatirishvili Safeguards Consultant GSU03 Rocio Mariela Malpica Valera Senior Counsel LEGES Benedicta T. Oliveros Procurement Specialist GGO03 Gayane Davtyan Program Assistant BPSEM
32
Angela Prigozhina Senior Financial Sector Specialist GFM09 Prior co-TTL Grigor Gharabaqhtsyan Civil Engineer Consultant ICR Junko Narimatsu ICT Policy Specialist GTI09 TTL/Primary Author Arsen Nazaryan Senior Private Sector Specialist GTCEE co-TTL Marta Lucila Priftis Administrative GTI09 Administrative
(b) Staff Time and Cost
Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including travel No. of staff weeks and consultant costs) Lending FY09 11.08 87.8 FY10 42.38 231.1 FY11 34.58 129.2 Total: 88.04 448.1 Supervision/ICR FY11 8.95 46 FY12 23.24 106.7 FY13 30.58 152.2 FY14 25.78 111.7 FY15 22.32 111.3 FY16 15.55 69.9 FY17 7.74 30.8 Total: 134.16 622.2
33
Annex 5. Beneficiary Survey Results
Beneficiary surveys were not carried out as part of the ICR, but informal interviews were carried out with targeted beneficiaries, and are recorded below.
Figure 5.1. Project’s Impact and Beneficiary Voices
Source: ICR author’s visits to the GTC, VTC, ISTC, mLAB ECA, MIC, Armenian-Indian Center for Excellence in ICT, Armenian National Engineering Lab, and ISTC between September 20 and 28, 2016.
(a) The ICR author met with local youth employed by GTC tenants, Digital Pomegrante (app development, web design, tech education), and Skyline Studio (2D/3D animation), and was presented some of the products that they are developing for the local (for example, Digital Gyumri) and international markets (for example, 2D/3D animation for the United States, the United Kingdom, and Russia). They cited critical trainings (Java, C++, and so on) provided at the GTC, which have led to enhancing their skills and employment opportunities with the start-ups.
(b) Informal discussions with young app developers/entrepreneurs at mLAB ECA (for example, Smart TV, 4Car.AM, Dr.Math, Police tool#1) revealed that many of the recipients of the project’s Innovation Matching Grants are now receiving acceleration support from the mLAB ECA program (figure 5.1). For example, SmartTV received a US$50,000 grant, and since its launch in late 2015 has reached 26,000 downloads, providing streaming of Armenian television programs and movies to the Armenian diaspora. 4Car.AM, which provides an online transportation platform that facilitates
34
driver’s communication on speed camera locations, was also a matching grant recipient in Gyumri, and now has 65,000 downloads, and concurrent online users can reach up to 300 in a given day. 4Car.AM is also partnering with the police to offer online payment system for penalties.
(c) At the ISTC, the ICR author met an Armenian diaspora entrepreneur/app developer from Silicon Valley (Figure 5.1) who had moved to Yerevan to provide trainings, meetups, and mentorship to local youth using ISTC as a platform, with an aim to help boost tech entrepreneurship in the country.
Comments from InecoBank (participating and beneficiary bank of the Computer for All Program)
1. Since 2012, InecoBank has been effectively involved as a participating bank in the ‘Computer for All’ Program supported by the World Bank, and has witnessed a number of positive developments since then, including constant growth of client base and increased ability to offer added value. The program implementation went smoothly enough, considering that InecoBank has always been a pioneer in offering point-of-sales products and cooperating with a wide network of retail resellers of computer equipment. The main prerequisites for success were low interest rates as compared to regular point of sale loans, as well the option of 0 percent prepayment. These terms were crucial for the clients with low income and overall helped increase the availability of computer equipment in hundreds of families throughout Armenia.
2. The main challenge of this project was to ensure awareness among the bank’s current and potential clients through diversified channels. The project helped us position ourselves even closer to our customers and also helped increase quality of life especially for low- and middle-class households in Armenia. From another point of view, this project was a good opportunity for InecoBank to extend volumes of customers because of strong competitiveness in private sector. Considering this, the program could have been even more efficient if lower interest rates were available. As the sales of this product were mostly performed on the spot through a cooperating shop network, the feedback has been collected on a regular basis from the respective shop assistants who mentioned that a vast majority of the clients not only left positive feedback but also advised their friends and relatives to use this product. Another positive feedback received was from the shops witnessing increased sales of eligible products.
3. The most of awareness raising was implemented through shops and stores, the volumes and good geographical spread of which is a huge advantage for InecoBank compared to other banks in Armenia. Also, there was an organized special group of phone sales from InecoBank staff, which also contributed to awareness raising among the broader public.
4. InecoBank is looking forward to participating in new projects aimed to serve our retail clients and being able to offer better terms and opportunities.
35
Annex 6. Stakeholder Workshop Report and Results
No stakeholder workshop was undertaken.
36
Annex 7. Borrower's ICR
Government’s evaluation report
The E-Society and Innovation for Competitiveness Project (hereinafter referred to as ‘the project’) has been successfully implemented, meeting all the determined core objectives and left a tangible positive socioeconomic impact on the Armenian IT sector and public in general. The project had a tremendous impact on creation of enabling environment for e-society, growth of internet penetration, residential broadband household penetration in villages, workforce development, jobs creation not only in Yerevan but also in the regions, support technology entrepreneurs throughout the whole incubation chain from idea generation/ acceleration to investment opportunities, prevention of migration from regions to Yerevan. This demonstrates the catalytic role of the project also in positioning of Armenia as a regional technology hub.
Key activities affecting the project’s outcomes: Nationwide broadband backbone and government network development- More than 172,000 people are benefitting from the provided free WiFi Internet access devices in 344 villages. Computer for All- The program is self-sustainable through the revolving fund. From year 2009-2016, internet penetration grew 58.5%, and percent of population using computers grew 42.4% resulting 57.4% penetration. Particularly, in rural areas, the internet penetration reached to 41.6% by 2015. The first Armenian Venture Fund- More than 70 jobs are created in IT /Engineering sectors and more than 2000 in other sectors; 4 international startups established core value-add in Armenia; 6 companies have generated revenue; 2 companies have expanded their operations to Georgian and Belarusian markets. More than $25 mln additional investment was raised by the investee companies. Innovation Matching Grants, Gyumri Matching Grants and Vanadzor Matching Grants- More than 150 new jobs are created in Yerevan, Gyumri and Vanadzor. More than $1mln additional funding/investment was raised by the companies. Additionally, the component contributed to the higher volume of exports (increase from 2.7% in 2014 to 5.3% in 2015. 15 companies are currently generating revenue. from 2009-2015, number of employment in IT/ITES sector grew by 75%. Several projects funded under both Venture Fund and Grants have already succeeded in international market, thus proving that the defined model is very successful. Gyumri Technology Center- More than 5,900 beneficiaries were involved in the training programs; 900 jobs are created and business development consultancies were provided to 50+ firms; 150 people are employed by the companies located in GTC, as well as American, Taiwanese and local companies have established their branches in GTC, thus ensuring almost 100% occupancy rate. Vanadzor Technology Center- Successful model of GTC is replicated in Vanadzor. More than 4000 beneficiaries were involved in the training programs, business development consultancies were provided to 40+ firms, more than 25 teams received assistance in establishment of startups. The establishment of Gyumri and Vanadzor Technology Centers created long-term development hubs for local IT companies and enterprises. Skills Development- More than 5000 students are being trained annually in the areas of Programming, Engineering, Mobile, Business/management and more than 100 startups have been accelerated. The Centers continue to cooperate with private sector to provide trainings based on the industry needs.
37
Project Development Objectives- The project is highly successful, as all the project Development objectives are achieved and more of them are overachieved with respect to the set targets, which is also stated in the World Bank’s ICR. Project Rating- Altogether, the implementation of the project influenced the development of the ICT sector in Armenia, particularly by joint efforts of the World Bank and local implementation agencies. Rating: World Bank team- Highly Satisfactory Local Implementing agencies- Highly Satisfactory Overall Project- Highly Satisfactory
38
Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders
There were no comments from cofinanciers, and other partners/stakeholders involved for this project.
39
Annex 9. List of Supporting Documents
PAD
Financing Agreement
Restructuring papers (September 2011, July 2014, July 2015)
Mission aide memoires
Implementation status and results reports
FM and procurement supervision reports
EIF project annual reports
KPMG performance monitoring reports
Armenian ICT Sector: State of Industry Reports (2010-2015)
40
SEPTEMBER 2004 SEPTEMBER
41
40
°
°
N N 43 43 °
E
shown on this map do not imply, the part of The World Bank This map was produced by the Map Design Unit of The World Bank. Group, any judgment on the legal status of territory, or The boundaries, colors, denominations and any other information endorsement or acceptance of such boundaries. ° E ARMENIA TURKEY
INTERNATIONAL BOUNDARIES PROVINCE (MARZ) BOUNDARIES RAILROADS MAIN ROADS RIVERS NATIONAL CAPITAL PROVINCE (MARZ) CAPITALS SELECTED CITIES AND TOWNS Aras GEORGIA
A
r
p Kars
To Borjomi
a To
SHIRAK SHIRAK
ARMA ARMAVIR Ar Armavir mavir
Gyumri
ARAGA ARAGATSOTN
44 VIR 44 Artik Ar °
° E
L L L E
tik tik
e e e
s s s
s s s
TSOTN
Tashir T
e e
Ashtarak e Ashtarak
Aragats (4090 m) ashir
Stepanavan Stepanavan
r r r
ISLAMIC REPUBLIC To T’bilisi
C C C
LORRI
a a a
u u u
OF IRAN
YEREV YEREVAN 02 30 Miles 20 10 0 102030400 c c c V Vanadzor
Alaverdi
anadzor a a a
s s s Aras
Ar Artashat
u u YEREV u YEREVAN
AN
AN Hrazda tashat tashat
s s
n s To T’bilisi
KOT KOT KOTAYK’
M M M
ARARA
ARARAT AN Ararat Ararat
A A
o o o YK
YK Hrazdan Hrazdan
u u u ’ ’
n n Sevan n Sevan
t t
Dilijan t Dilijan
a a
Naxçivan a
T
i i
45 i To
n n
Gavar n Gavar
° A
E r s s s a
50 Kilometers s
TAVUSH T
GEGHARK GEGHARK’UNIK’
A
Ijevan Ijevan VUSH V VUSH
A a AZERBAIJAN Karmir Kar
r
p r
a Sevan
V VAYOTS’
mir
d Lake
A
A Martuni Mar Y Yegegnadzor
DZOR DZOR e
egegnadzor
YOTS YOTS n tuni
i To Gäncä
s
’
UNIK
R Vaik V
Artsvashen Ar To Naxçivan Gäncä
aik
a
tsvashen ’ To Vardenis V Jermuk Jer
ardenis
n To Gäncä
muk
g
’
e
K
u
r a
Z Z
Z 46 To Ordubad
a a a Angekhakot Angekhakot
n n n °
E
g g g
e e e
z z z
SYUNIK
SYUNIK’ GEORGIA
u u u
r r r
V
o
R R R
Megri r Megri
o a a a
t n n n
a
n g g g
e e
e Kapan Kapan
AZERBAIJAN Goris Goris
’
Qubadli
To Qubadli
ARMENIA
To
To Füzili
Mingechevir Reservoir
Aras 47 ° E 47 ° E
39
40 41
°
° °
N
N N IBRD 33364 IBRD