Ex post evaluation of cohesion policy interventions 2000-2006 financed by the Cohesion Fund – Work Package B: Cost-benefit analysis of selected transport projects FINAL REPORT
20th June 2011
© Frontier Economics Ltd, London.
June 2011 | Frontier Economics, Atkins, ITS i
Cost-benefit analysis of selected transport projects
Acknowledgements vii Disclaimer xi Glossary xiii Executive Summary 1 1 Introduction 21 1.1 Context...... 21 1.2 Background ...... 21 1.3 Objectives ...... 22 1.4 Structure ...... 23 2 Approach and methodology 25 2.1 Task 1: Selecting ten transport projects for the ex post cost- benefit analysis ...... 25 2.2 Task 2: Ex post project impact analysis ...... 34 2.3 Task 3: Assessing cost-benefit analysis as method ...... 41 3 The role of cost benefit analysis: a brief review 45 3.1 The roles of CBA in transport appraisal ...... 45 3.2 Strengths and weakness of ex ante CBA ...... 53 4 Evaluation of selected projects 55 4.1 Overview of selected projects ...... 55 4.2 Review of ex ante CBA ...... 63 4.3 Results of ex post evaluation ...... 73 4.4 Comparison of ex ante and ex post analyses ...... 84 5 Cost-benefit analysis as a project appraisal method: findings from the ten transport projects 93 5.1 Findings on ex ante CBA ...... 93 5.2 Ex post CBA findings ...... 104
Contents ii Frontier Economics, Atkins, ITS | June 2011
6 The role of CBA in predicting the macroeconomic impacts of transport investment 111 7 Conclusions and recommendations 113 7.1 Ex post evaluation summary ...... 113 7.2 Cost-benefit analysis as a method ...... 114 7.3 Recommendations ...... 118 Annexe 1: Project summaries 123 Annexe 2: Strengths and weaknesses of ex ante CBA appraisal 145 Annexe 3: References 155 Annexe 4: Ex-post CBA of transport projects – experience and lessons to date 159 Annexe 5: Modelling the macroeconomic impacts of transport investment using CBA output 167 Appendices 1 to 10: Detailed case studies
Contents
June 2011 | Frontier Economics, Atkins, ITS iii
Cost-benefit analysis of selected transport projects
Figure 1. Illustration of calculation of overall score 33 Figure 2. Summary of ex ante stated objectives. 58 Figure 3. Impacts considered in the ex ante CBA. 64 Figure 4. Role played by the ex ante CBA 73 Figure 5. Sources of benefits – railway projects 78 Figure 6. Sources of benefits – road projects 79 Figure 7. Summary of wider impacts from the ten transport projects. 80 Figure 8. Level unit costs availability 81 Figure 9. Comparison of utilisation rates 83 Figure 10. Comparison of ex ante and ex post NPV 85 Figure 11. NPV comparison (without more extreme observations). 86 Figure 12. BCR Comparison 87 Figure 13. BCR Comparison (without outlier: Road IE) 87 Figure 14. IRR Comparison 88 Figure 15. IRR Comparison (without more extreme observations) 89 Figure 16. Capital Costs comparison. 90 Figure 17. Main reasons for discrepancies between ex ante and ex post results 91 Figure 18. Spatial pattern of economic impact (TEN+TINA priority projects) 150 Figure 19. Inclusion of Other Environmental Impacts in Ex Ante CBA 151 Figure 20. The POPE Evaluation and Appraisal Cycle 160
Table 1. Regional distribution in the original 40-project list. 27 Table 2. Transport mode distribution in the original 40 projects list. 27 Table 3. Distribution of funding between regions and modes (initial 40 projects) 28
Figures and Tables iv Frontier Economics, Atkins, ITS | June 2011
Table 4. 20-project shortlist 29 Table 5. Distribution of EU funding between regions and modes (20 project shortlisted) 30 Table 6. Date-Related Prioritisation Scoring Matrix (Range of points depending on the Standard reach across criteria) 32 Table 7. Summary of candidate projects 33 Table 8 Summary of topic areas considered in semi-structured interviews 40 Table 9. Explicit “Value for Money” thresholds 47 Table 10. Use and Scope of CBA in the EU25 + Switzerland 48 Table 11. Use of CBA by transport mode in EU25 + Switzerland 50 Table 12. Summary of candidate projects 55 Table 13. Overview of projects considered 56 Table 14. Project objectives 57 Table 15. Overview of discount rates and appraisal periods 65 Table 16. Overview of value of time assumptions used in the ex ante analyses and comparison with HEATCO values 67 Table 17. Unit costs of vehicle operating costs 68 Table 18. Unit cost of accidents – comparison with HEATCO values, when available 70 Table 19. Risk analysis. Sensitivity tests run and risk factors considered 72 Table 20. Ex post evaluation – economic appraisal indicators 75 Table 21. Ex post evaluation - Financial appraisal indicators 76 Table 22. Overview of project funding. 77 Table 23. Unit cost comparison for rail projects 81 Table 24. Unit cost comparison for road projects 82 Table 25. Identified weaknesses in the ex ante CBA analyses 95 Table 26. The treatment of network effects in the ten transport projects 97 Table 27. Optimism Bias Adjustments 146 Table 28. Transport impacts utilised by CBA and GVA modelling 172
Figures and Tables
June 2011 | Frontier Economics, Atkins, ITS v
Figures and Tables
June 2011 | Frontier Economics, Atkins, ITS vii
Acknowledgements
This report has been written by a team led by Frontier Economics together with Atkins and the Institute of Transport Studies (University of Leeds), following the call for tenders 2009CE160CAT051. The report has been completed thanks to the help, feedback and information received from a series of individuals and officials both in the Commission and in Member States. First, the authors are grateful for the helpful comments and advice received from members of the Evaluation Unit in DG REGIO, including Veronica Gaffey, Jurate Vaznelyte, Adam Abdulwahab, Lucia Pacillo, Patty Simou and Kai Stryczynski; and from Commission officials who participated in several meetings of the Steering Committee which oversaw the implementation of the study. Second, we are also grateful to the national representatives who participated in the Workshop with Member States on Ex Post Evaluation of Cohesion Fund (including former ISPA) 2000-2006 held in Brussels on the 3rd of February 2011 for sharing their experience about the Cohesion Fund instrument and their views about the findings of our study. Third, we would like to extend our thanks to the three external experts who advised the team throughout the implementation of the study providing detailed feedback on methods and findings: (in alphabetical order) Professor Antonio Estache (ECARES, Université Libre de Bruxelles), Professor Emile Quinet (Paris School of Economics - Ecole des Ponts ParisTech) and Professor András Timar (Professor Emeritus, University of Pécs, Faculty of Engineering Mihály Pollack). Finally, this study could have not been completed without the support of staff in the DG REGIO Geographic Unit and project stakeholders at Member State level. Specifically, we received invaluable help from the following individuals:
High Speed railway Madrid – Barcelona in Spain Javier Asensio (Ajuntament de Barcelona); Carme Bellet Sanfeliu (Universitat de Lleida) ; Miguel Benito (DG REGIO, European Commission); Alfonso Abengozar Tejero (RENFE); Oriol Clos (Ajuntament de Barcelona); Ana Raquel García Rubio (Ministerio de Economía y Hacienda); Sergi Lozano (Institut d‟Estudis Territorials); María Muñoz (Ministerio de Economía y Hacienda); Isabel Meléndez (Ajuntament de Barcelona); José Antonio Nieves (Ministerio de Economía y Hacienda); Luis Nistal Martínez (ADIF); Antonio Rodríguez (RENFE); Gaspar Ros Moreno (DG REGIO, European Commission); Concepción Sanz (Ministerio de Fomento); Rosa Sebastián Escolano (ADIF); Manuel Villalante (Departament de Política Territorial i Obras Públiques, Generalitat de Catalunya).
Acknowledgements viii Frontier Economics, Atkins, ITS | June 2011
A2 motorway in Poland Magdalena Adamczuk (CUPT); Malgorzata Bronowicz (GDDKiA - Warsaw); Michal Grzybowski (CUPT); Andrzej Jankowski (Gmina of Strykow); Roman Jankowski (Town office in Konin); Krzysztof Jan Kalinski (Mayor of Lowicz); Rafal Karolczak (Lodz Special Economic Zone); Barbara Kielar (GDDKiA - Lodz); Paul Knight (EIB); Szymon Milczarek (GDDKiA - Lodz); Monika Milwicz (GDDKiA - Warsaw); Gregorz Obara (GDDKiA - Warsaw); Emilia Pieczara (Ministry of Infrastructure); Ewelina Rapela (Konin County Council); Steve Richards (EIB); Tomasz Smollen (GDDKiA - Warsaw); Wieslaw Tomczyk (Gmina of Strykow); Robert Wojdynski (GDDKiA - Warsaw); Przemyslaw Wrobel (CUPT)
Algarve railway in Portugal Marta Albuquerque (REFER); Helena Azevedo (POVT); Ana Paula Barreiros (REFER); Duarte Ladeira (POVT); Maria Lopes (CP); Germano Martins (POVT); Edoardo Pires (REFER); Mario Rodrigues (DG REGIO, European Commission); Luisa Maria Soares (REFER); Lilia Sousa (CP); Helena Vieira (POVT); Miquel Vila (DG REGIO, European Commission)
A23 motorway in Spain Miguel Benito (DG REGIO, European Commission); Carlos Casas (Unidad de carreteras del Estado en Teruel); Justo Borrajo Sebastián (Ministerio de Fomento); Pedro Galán Bueno (Ministerio de Fomento); Antonio Garcia Cañada (Ministerio de Fomento); Jesús Iranzo (Unidad de carreteras del Estado en Teruel); Rafael López Guarga (Ministerio de Fomento); Joaquín López Sanchez (Ministerio de Fomento); Gaspar Ros Moreno (DG REGIO, European Commission).
Agiou Konstantinou bypass in Greece John Krassakopoulos (Ministry of Transportation); Soultana Lyngeri (DG REGIO, European Commission); George Logothetis (Ministry of Economy); Mr Evangelos Psathas (Mayor of Agios Konstantinos); John Voloioutis (PATHE)
M1 northern motorway in Ireland Jim Deane (Department of Finance); Richard Evers (NRA); John Fitzsimons (NRA); Philip Hopkins (Department of Transport); Alan O'Brien (AECOM); Fergal Trace (NRA)
Acknowledgements
June 2011 | Frontier Economics, Atkins, ITS ix
Railway Thriassio-Pedio-Eleusina-Korinthos in Greece Isaia Karamida Linta (Ergose); Soultana Lyngeri (DG REGIO, European Commission); George Logothetis (Ministry of Economy); Ms Vitou Katerina (OSE)
IX B corridor in Lithuania (including Vilnius Southern bypass) Albertas Arūna (Transport Investments Directorate); Laura Bogušienė (Ministry of Finance); Laura Danielutė (DG REGIO, European Commission); Arvydas Domatas (Transporto ir kelių tyrimo institutas); Zita Dubickienė (Ministry of Transport and Communications); Mantas Kaušylas (UAB "Kelvista"); Ieva Kriščiūnaitė-Kačiuškienė (Ministry of Finance); Ilona Lankininkienė (Vilnius Municipality); Jurgita Matiliauskaitė (Ministry of Finance); Aurelija Meškauskaitė (Transport Investments Directorate); Mindaugas Pakštys (Ministry of Finance); Kristina Petrauskienė (Lithuanian Road Administration); Gintautas Predkelis (Transport Investments Directorate); Mindaugas Šidagis (Ministry of Transport and Communications)
Bratislava Rača – Trnava Railway Upgrade in Slovakia Rozália Dzvoníková (Ţeleznice Slovenskej Republiky); Stefan Elek (Národná diaľničná spoločnosť, a.s.); Marian Hantak (Ministry of Transport, Posts and Telecommunications); Zuzana Krajcirikova (Národná diaľničná spoločnosť, a.s.); Juraj Majtan (Bratislava Regional Chamber of SCCI); Miroslav Matusek (Ţeleznice Slovenskej Republiky); Štrba Peter (Ţeleznice Slovenskej Republiky); Rastislav Prokopic (DG REGIO, European Commission)
M0 Budapest ring road (eastern section) in Hungary György Bátovszki (Mayor of Csömör municipality); Péter Bucsky (Nemzeti Infrastruktúra Fejlesztő Zrt.); Beatrix Horváth (Nemzeti Fejlesztési Ügynökség); Csaba Kelen (KOZLEKEDES Ltd, Budapest); Kyriacos Ktenas (DG REGIO, European Commission); Violetta Tarnócziné (Nemzeti Infrastruktúra Fejlesztő Zrt.); Attila Tompos (Nemzeti Infrastruktúra Fejlesztő Zrt.)
Acknowledgements
June 2011 | Frontier Economics, Atkins, ITS xi
Disclaimer
The views expressed in this report are those of the authors and do not necessarily reflect the official opinion of the European Commission or indeed of any of the authors of the Work Packages. Any errors or omissions remain the responsibility of the authors. The European Commission and the authors accept no responsibility or liability with regards to this report. Specifically: this report is of general nature and is not intended to address the specific circumstances of any particular entity or individual; and this report does not constitute professional or legal advice. Reproduction or translation is permitted, provided that the source is duly acknowledged and no modifications to the text are made.
Disclaimer
June 2011 | Frontier Economics, Atkins, ITS xiii
Glossary
Acronym Definition
AADT Average Annual Daily Traffic
ADIF Administrador de Infraestructuras Ferroviarias (The railway infrastructure manager in Spain)
AHVV Average Hourly Value of time for one Vehicle
AVE Alta Velocidad Española (Spanish high-speed railway service)
B/C ratio Benefit cost ratio. Also BCR.
BIR Benefit investment ratio
BOE Boletín Oficial del Estado (Spanish official state gazette)
CBA Cost Benefit analysis. Also COBA
CF Cohesion Funds
CP Comboios de Portugal (A state-owned company which operates freight and passenger trains in Portugal).
CSO Irish Central Statistical Office
CUPT Centre for EU Transport Projects, in Poland. Body set up to implement programmes and projects of the transport infrastructure development, particularly programmes and projects co-financed by EU funds in the programming period of 2007 - 2013.
DIA Environmental Impact Declaration (Declaración de Impacto Medioambiental), in Spain.
DoF Irish Department of Finance
EBRD European Bank for Reconstruction and Development
EC European Commission
EIA Environmental Impact Assessment
Glossary
xiv Frontier Economics, Atkins, ITS | June 2011
Acronym Definition
EIB European Investment Bank.
EIRR Economic internal rate of return.
ENPV Economic net present value.
ERDF European Regional Development Fund
FDI Foreign Direct Investment
FNPV(C) Financial net present value of the investment.
FNPV(K) Financial net present value of capital.
FRR Financial rate of return.
FRR(C) Financial rate of return of the investment.
FRR(K) Financial rate of return of capital.
GDDKiA Generalna Dyrekcja Drog Krajowych Autostrad (Polish General Directorate for National Roads and Highways). Established in the Polish Ministry of Transportation, GDDKiA is the authority in charge of managing the national roads and the implementation of the state budget in the sector.
GDP Gross Domestic Product
HEATCO EC funded research project in charge of "Developing Harmonised European Approaches for Transport Costing and Project Assessment".
HGV Heavy goods vehicle.
INE Instituto Nacional de Estadística (Spanish National Statistics Institute)
IRR Internal rate of return.
ISPA Instrument for Structural Policies for Pre – Accession
Glossary
June 2011 | Frontier Economics, Atkins, ITS xv
Acronym Definition
JASPERS Joint Assistance to Support Projects in European Regions. A joint initiative of the EIB, EBRD and EC to assist the 12 Central and Eastern EU Member States in the preparation of major projects to be submitted for grant financing under the Structural and Cohesion Funds.
LAV Linea de Alta Velocidad (Spanish High-speed line)
LGV Light goods vehicle.
LRA Lithuanian Road Administration
MCA Multi-Criteria Analysis
NDS Národná diaľničná spoločnosť (Slovakian highway authority)
NRA National Roads Authority
NTDP Lithuanian National Transport Development Programme
OKA Hungarian National Highway Databank
PATHE Patras, Athens, Thessaloniki, Evzoni
PCU Passenger Car Units
PIA Personal Injury Accidents
PLN Polish Zloty
PSV Passenger Service Vehicle (i.e. bus or coach)
PTE Portuguese Escudos
POVT Programa Operacional Temático Valorização do Território (The Portuguese government programme for the implementation of infrastructure projects)
PVB Present Value Benefit
REFER Rede Ferroviária Nacional (The railway infrastructure manager in Portugal)
Glossary
xvi Frontier Economics, Atkins, ITS | June 2011
Acronym Definition
RENFE Red Nacional de Ferrocarriles Españoles (The railway public operator in Spain).
RSA Irish Road Safety Association
SEZ Special Economic Zone
SME Small and Medium size Enterprise
TEN-T Trans European transport network
TOR Terms of Reference
VAT Value Added Tax
VOC Vehicle Operating Costs
VOT Value of Time
ŢSR Železnice Slovenskej Republiky (Slovakian rail authority)
ŢSSK Železničná spoločnosť Slovensko (Slovak state-owned passenger train operating company)
Glossary
June 2011 | Frontier Economics, Atkins, ITS 1
Executive Summary
Frontier Economics, together with Atkins and the Institute for Transport Studies (University of Leeds), is pleased to present this Final Report for the study “Ex post evaluation of cohesion policy interventions 2000-2006 financed by the Cohesion Fund – Work Package B: Cost-benefit analysis of selected transport projects”.
Context The Cohesion Fund was established in 1993 to strengthen the economic and social cohesion of the European Union.1 The eligibility criterion is that the GNP per capita in the applicant country is 90% or less than the EU average. During the period 2000-2006, 17 countries have received funding from the Cohesion Fund and ISPA (Instrument for Structural Policies for Pre-Accession, aimed at accession countries) for transport and environment projects. The Council Regulation2 states that “in order to ensure the effectiveness of Community assistance, the Commission and the beneficiary Member States shall, in cooperation with the EIB where appropriate, carry out a systematic appraisal and evaluation of projects,”. This study focuses on the ex post evaluation of a selection of ten transport projects in EU countries benefitting from EU Cohesion and ISPA funding during the period 2000-2006. The study forms part of three studies commissioned by DG REGIO to assess the effect of the Cohesion Fund and ISPA on economic and social cohesion and draw lessons for the future. These studies are package A (overall); package B (transport); and package C (environment). The Terms of Reference (TORs) for Work Package B identify the three key questions that this study addresses:
Question 1: What were the impacts of the examined projects?
Question 2: How can ex post cost benefit analyses (CBA) contribute to the practice of ex ante cost-benefit analysis?
Question 3: What are the potential and the limits of ex post CBA as a tool to identify the impact of infrastructure projects? The first question is specific to the ex post evaluation of the ten transport projects selected. During the period 2000-2006, there were 260 individual
1 Council Regulation (EC) No 1164/94 of 16 May 1994 establishing a Cohesion Fund [Official Journal L 130, 25.05.1994]. 2 Ibid.
Executive Summary
2 Frontier Economics, Atkins, ITS | June 2011
transport projects co-financed by the Cohesion Fund, for a total eligible value of about €8.6bn and contributions from the Cohesion Funds of about €7.2bn. The TORs of this study shortlisted 40 transport schemes, covering a total of 103 individual transport projects. Therefore, out of the 40 schemes identified in the TORs, we selected 10 of them for in-depth ex post evaluation. In aggregate, these schemes encompass 38 individual transport projects, that is about 15% of the individual projects co-financed by the Cohesion Fund in the period 2000- 2006.
Approach To address the three questions above, as specified in the TORs, we completed three tasks in sequence:
In Task 1, we selected, out of the list of 40 transport projects provided in the TORs, a shortlist of 20 projects. For each project, we carried out an initial review. We presented the results of this review in a three-page project description, which included an initial overview of the quality of the ex ante CBA, information on the degree of completion of the project, and the identification of the data requirements to complete a full ex post CBA. We used the results of this review to select ten transport projects for an in-depth ex post CBA analysis.
Task 2 was the core of the study and focused on the ten selected transport projects. In this task, we first undertook a review of any existing ex ante cost-benefit analysis carried out at the time of the funding application. We also reviewed any interim progress report where available. This initial review acted as background material for the in-depth ex post evaluation of these projects, which provided an assessment of the overall performance of the investment. We based the ex post evaluations both on the data already available and, where necessary, on data collected by the project team. We then compared the results of the ex post evaluations with any available ex ante evaluations, to assess the appropriateness of the CBA methodology being applied and to identify any cause for possible discrepancies between expected results and actual outcomes.
Finally, in Task 3, we assessed the potential for ex post cost-benefit analysis to become an integral part of the infrastructure funding appraisal and evaluation process in the EC and in Member States. We identified the lessons that can be drawn from ex post evaluation to help improve the methods of ex ante evaluation. In particular, we looked at the extent to which the use of ex post CBA can strengthen the evidence base underpinning the ex ante analysis in several areas of project appraisal. We also focused on identifying methodological strengths and weaknesses of cost-benefit analysis.
Executive Summary
June 2011 | Frontier Economics, Atkins, ITS 3
Projects considered for the ex post evaluation Before presenting the overall results of our analysis, Tables I and II contain a brief summary of the ten projects included in the study..
Table I. Rail projects
Project Description
The railway Madrid – Barcelona – French border is a high speed line connecting Madrid to the French border via Barcelona. At the moment of writing, only the section Madrid – Barcelona is operational. The segment AVE Madrid - between Barcelona and the French border is still under construction and it is Barcelona (Spain) expected to be operational in 2013. The LAV is part of the TEN-T Priority Project 3 (high-speed railway axis of south-west Europe), whose main objective is to provide high-speed rail connections between the Iberian Peninsula (Portugal and Spain) and the rest of Europe.
This project covers the modernisation of the railway line between Coina (near Algarve Railway Lisbon) to Faro (in the Algarve region) in Portugal, including the branch to Porto de Sines. The total extension of both rail segments is around 275 km (Portugal) and 50 km respectively. The project is part of the TEN-T Priority Project 8 (Multimodal Axis Portugal/Spain – Rest of Europe).
The project covers a rail upgrade between Thriasio (north west of Athens) and Kiato in Greece. The new line has replaced a former single track metric line passing through a number of urban areas. The 112km section runs parallel to the European TEN-T network Motorway Priority Axis 7 Railway Thriassio – (Igoumenista-Patras-Athens-Sofia-Budapest) and forms part of longer term Kiato (Greece) proposals to upgrade the Piraeus-Athens-Patra line. The route is covered by four funding applications for cohesion funding. Three of the sub-projects are related to the section of track running from Thriasio to Korinthos. An additional funding application was made for the section of track between Korinthos and Kiato.
This project covers a rail upgrade between Bratislava Rača and Trnava in Slovakia. The 39km section forms part of the Corridor Va TEN-T rail corridor which runs in Slovakia between Bratislava, Ţilina and Košice, then onto the Bratislava Railway border with Ukraine. The section of line also forms part of Priority Axis No. 23. The modernization project includes two separate applications for funding, Upgrade (Slovakia) the first including the upgrade of the line between Bratislava Rača to Šenkvice (FM001), and the second upgrading the line between Šenkvice and Cífer and all the stations along the section between Bratislava Rača and Trnava (FM003).
Source: Funding applications and supporting material
Executive Summary
4 Frontier Economics, Atkins, ITS | June 2011
Table II. Road projects
Project Description
This project involved the construction of approximately 100km of new motorway between the towns of Konin and Strykow in central Poland. The route is of national and international importance and forms part of the A2 Motorway European Route E30 between Berlin and Belarus. This section of the A2 is (Poland) part of a wider package of projects to construct a high standard road transport link between Berlin and Warsaw, as well as improving connectivity at local and regional scale. The section between Strykow (at the eastern end of the project) and Warsaw is currently under construction.
This project covers the construction of 75 km of the A23 motorway in Spain, covering two separate segments, one of 63.5 km between Teruel and Calamocha and another of 11.5 km between Huesca and Nueno. The A23 A23 Motorway (also called Autovía Mudéjar) is a high capacity road, long 440 km (370km in (Spain) service), connecting Sagunto, on the Mediterranean coast north of Valencia, and the Somport road tunnel, which connects France and Spain through the central Pyrenees. The A23 partially belongs to the European route E07 connecting Pau (France) and Zaragoza (Spain).
This project involved the construction of a bypass around the town of Agios Konstantinos and an upgrade of an existing Bypass around Kamena Vourla, Agios Konstantinos both in Greece. This is part of the PATHE (Patras-Athens-Thessaloniki- Evzoni) corridor. The route is of national importance because it links the two Bypass (Greece) largest cities in Greece (Athens and Thessaloniki), internationally the route supports an important link between Greece, Central Europe, and the Balkans.
This project covers the construction of two sections of the M1 motorway in Ireland. The first section runs from Cloghran to Lissenhall and the second M1 Northern section joins this road with the Balbriggan Bypass further north. The M1 Motorway (Ireland) Motorway is part of the TEN-T Priority Axis 13 (Ireland – United Kingdom – Benelux). The main objective of the investments in this corridor is the reduction in journey times between Ireland, the UK and mainland Europe.
This project covers the modernisation of one of the two main motorway routes in Lithuania. This is part of the IX B transport corridor, 315km of IX B Corridor motorway linking the port city of Klaipeda with the capital city of Vilnius, via (Lithuania) Kaunas. The route then links Lithuania with other destinations in Eastern Europe. The project also involved the construction of the Vilnius Southern bypass.
The M0 Eastern Sector is the eastern component of the Budapest orbital motorway, whilst the M31 is a linking motorway between it and the M3 (the motorway to the north east of Hungary). The project has important international, national and local dimensions. Budapest sits on the junction of M0 Budapest Ring three land based Helsinki corridors, is the focus of the national transport Road (Hungary) network and is the economic centre and capital of Hungary. Congestion in the city centre, particularly in the vicinity of the Danube is severe. This project therefore performs an important function in facilitating the movement of international and national traffic while also providing some traffic relief to Budapest.
Source: Funding applications and supporting material
Table III shows the opening dates of the various projects. It also indicates when the most recent ex ante CBA was undertaken for each project. We also obtained
Executive Summary
June 2011 | Frontier Economics, Atkins, ITS 5
information on project capital costs and Cohesion Funds contribution from each project‟s funding applications. Most projects were divided into various subprojects, each with its own different completion dates and, in some cases, different ex ante analysis.
Table III. Opening dates and cost of the projects considered
Capital costs - EURm Opening Date of ex ante date(s) appraisal (% Cohesion Fund contribution)
Rail projects
AVE Madrid - 1,719 2003 - 2008 2001 Barcelona (Spain) (61%)
Algarve Railway 419 2003 - 2006 1999 (Portugal) (77%)
Railway Thriassio – 619 2005 - 2007 2000 Kiato (Greece) (47%)
Bratislava Railway 234 2003 - 2009 2001 Upgrade (Slovakia) (39%)
Road projects
A2 Motorway 476 2006 2003 (Poland) (82%)
A23 Motorway 203 2001 - 2005 1999 and 2003 (Spain) (83%)
Agios Konstantinos 441 2008 2002 Bypass (Greece) (55%)
M1 Northern 232 2003 1995 Motorway (Ireland) (66%)
IX B Corridor 154 2006 - 2009 1999 (Lithuania) (79%)
M0 Budapest Ring 367 2008 - 2010 2004 Road (Hungary) (74%)
Source: Funding applications
Table IV lists the main objectives of each of the ten projects as stated in their funding applications. The table shows that safety was the main stated objective in 8 of the 10 projects, followed by the objectives of reducing travel time (6 projects) and increasing capacity (4 projects).
Executive Summary
6 Frontier Economics, Atkins, ITS | June 2011
Table IV. Summary of main project objectives
Project Stated project objectives
Rail projects
AVE Madrid - Reduction in travel times on the existing corridor. Barcelona (Spain) Increase capacity and safety conditions on the rail line. Algarve Railway Reduction in travel times and increase productivity. (Portugal) Reduction in operating costs on the line. Increase safety conditions. Railway Thriassio – Reduction in travel times. Kiato (Greece) Increase safety conditions on the rail line. Bratislava Railway Reduction in travel times. Upgrade (Slovakia) Increase safety conditions on the rail line. Meet interoperability requirements.
Road projects
A2 Motorway Reduction in congestion on existing roads. (Poland) Improve safety conditions. A23 Motorway Increase capacity on existing corridor. (Spain) Improve communications in low density areas. Improve safety conditions. Agios Konstantinos Reduction in travel times. Bypass (Greece) Increase safety conditions. Improve connectivity between Greece and the EU. M1 Northern Reduction in travel times. Motorway (Ireland) Facilitate growth at Dublin airport. IX B Corridor Increase pavement strength. (Lithuania) Divert traffic away from Vilnius city centre. Improve safety conditions on the existing corridor. M0 Budapest Ring Reduction in transport costs (travel times and congestion). Road (Hungary) Divert traffic away from Budapest city centre Source: Funding applications In the next section we present a summary of the results of the in-depth ex post evaluation of these projects.
Executive Summary
June 2011 | Frontier Economics, Atkins, ITS 7
Results of the ex post evaluation Tables V and VI present the economic appraisal indicators (Present Value of Benefits, Present Value of Costs, Net Present Value, Benefit Cost Ratio, and Internal Rate of Return) for the ten transport projects included in the study. In carrying out the ex post evaluation, we have calculated a High case (optimistic) outcome and a Low case (pessimist) outcome. For each indicator, tables V and VI show the values for both High and Low case.
Table V. Ex post evaluation of rail projects – economic appraisal indicators
PVB (€m) PVC (€m) NPV (€m) BCR IRR High High High High High Low Low Low Low Low
Rail projects
AVE Madrid - 5,744 7,692 -1,948 0.7 3.7% Barcelona (Spain) 4,856 7,593 -2,736 0.6 2.6%
Algarve Railway 410 331 79 1.2 7.4% (Portugal) 379 331 48 1.1 6.7%
Railway Thriassio – 583 326 258 1.8 9.3% Kiato (Greece) 358 326 32 1.1 6.1%
Bratislava Railway 443 231 98 2.0 10.4% Upgrade (Slovakia) 291 231 40 1.4 7.8%
Note: For all schemes except for Bratislava Railway Upgrade (Slovakia) and Railway Thriassio – Kiato (Greece) figures reported are in factor prices as oppose to market prices.3
3 Market prices (MP) refer to the prices for which a good or service is offered in the marketplace. Factor prices (FP) are equal to market prices excluding the effects of taxes and subsidies on products.
Executive Summary
8 Frontier Economics, Atkins, ITS | June 2011
Table VI. Ex post evaluation of road projects – economic appraisal indicators
PVB (€m) PVC (€m) NPV (€m) BCR IRR High High High High High Low Low Low Low Low
Road projects
A2 Motorway 1,168 268 900 4.4 22.8% (Poland) 791 268 523 3.0 18.2%
A23 Motorway 253 225 28 1.1 6.3% (Spain) 198 225 -28 0.9 4.6%
Agios Konstantinos 488 206 283 2.4 13.4% Bypass (Greece) 438 206 233 2.1 12.6%
M1 Northern 4,140 235 3,905 17.6 53.0% Motorway (Ireland) 4,040 235 3,805 17.2 53.0%
IX B Corridor 300 88 212 3.4 56.0% (Lithuania) 288 88 200 3.3 55.0%
M0 Budapest Ring 1,187 213 974 5.6 24.8% Road (Hungary)
As tables V and VI show, most projects yield a positive net present value, indicating that their economic benefits exceed their costs. In the AVE Madrid – Barcelona project the NPV is negative for both the High case and the Low case due to the high upfront capital costs of the project. The cost-benefit analysis of this project, however, does not account for the wider socio-economic impacts of the investment. These wider economic benefits include the impact on mobility and accessibility, socio-economic structures, urban image and spatial effects. These impacts are expected to be significant, although it is still too early to appreciate their extent. Moreover, the wider economic benefits are difficult to quantify in monetary terms, and therefore be added to the cost-benefit analysis in a consistent way, as there is no agreed methodology for doing so. The benefit-cost ratio of the ten projects ranges from 0.6-0.7 (Low-High) of the AVE Madrid-Barcelona high speed train project in Spain to 17.2-17.6 (Low- High) of the M1 Northern Motorway project in Ireland. The average benefit- cost ratios for the ten projects are 3.6 (Low) and 4.0 (High); while the average IRRs are 19.1 (Low) and 20.7 (High). If we exclude the M1 Northern Motorway project in Ireland and the IX B Transport corridor in Lithuania (both projects have a very high IRR), the average IRRs are 10.4% (Low) and 12.3% (High).
Executive Summary
June 2011 | Frontier Economics, Atkins, ITS 9
These findings confirm that, overall, the projects analysed have delivered value for money with the exception of the AVE Madrid-Barcelona high speed line whose high capital costsexplain its unfavourable cost-benefit ratio. Tables VII and VIII present the financial appraisal indicators obtained from the ex post financial analysis of the ten selected projects. For each project, the tables present the Net Present Value and the Internal Rate of Return of Investment (“I”), and the Net Present Value and the Internal Rate of Return of Capital (“C”), showing the results for both the High and Low case scenarios.
Table VII. Ex post evaluation of rail projects - Financial appraisal indicators
FNPV (I) FIRR (I) FNPV (C) FIRR (C) (€m) (€m) High High High Low High Low Low Low
AVE Madrid - -4,288 0.6% -351 4.5% Barcelona (Spain) -4,766 0.4% -919 3.7%
Algarve Railway -299 -3.6% -111 0.2% (Portugal) -299 -3.6% -111 0.2%
Railway Thriassio – -101 3.4% 98 7.3% Kiato (Greece) -184 1.8% 10 5.3%
Bratislava Railway -11 4.4% 81 15.1 Upgrade (Slovakia) -28 3.3% 65 13.6
Source: Funding applications and supporting material
Executive Summary
10 Frontier Economics, Atkins, ITS | June 2011
Table VIII. Ex post evaluation of road projects - Financial appraisal indicators
FNPV (I) FIRR (I) FNPV (C) FIRR (C) (€m) (€m) High High High Low High Low Low Low
A2 Motorway -263 -3.5% 47 11.6% (Poland) -263 -3.5% 47 11.6%
A23 Motorway -230 -55 n/a n/a (Spain) -230 -55
Agios Konstantinos -166 - 1.3% -78 1.0% Bypass (Greece) -173 -1.7% -85 0.5%
M1 Northern -235 -81 n/a n/a Motorway (Ireland) -235 -81
IX B Corridor -95 -28 n/a n/a (Lithuania) -95 -28
M0 Budapest Ring 220 n/a -49 n/a Road (Hungary)
Source: Funding applications and supporting material
The financial NPV of seven of the ten projects is negative. However, this is not surprising since most of these projects do not generate direct commercial revenues and the financial analysis does not take the Cohesion Fund contributions into account. Without taking those EU contributions into account, some projects have a negative financial NPV but a positive economic NPV. The interpretation is therefore that Cohesion Fund contributions help to „unlock‟ the economic benefits of these projects.
Wider economic benefits Our analysis also considered the wider social and economic impacts of each project. Quantifying these wider impacts of transport projects on economic, social and physical environments can be challenging. Normally, these are impacts which cannot be included in the standard CBA, for a variety of reasons: the lack of an agreed methodology to identify their monetary value; the lack of data, which would make the quantification impossible even if an agreed methodology was available; and,
Executive Summary
June 2011 | Frontier Economics, Atkins, ITS 11
the difficulty to isolate a project‟s true impacts with a great deal of certainty, especially in the context of rapidly changing environments (for example, in the new Member States). Therefore, the approach we used to assess the wider economic impacts was largely qualitative. Below we summarise our findings in this area for each of the ten projects included in the study. AVE Madrid - Barcelona (Spain) Most studies carried out on wider socio-economic impacts conclude that high- speed rail is not a sufficient condition to cause major transformations in the cities and regions connected by it. High-speed rail only facilitates socio-economic changes that may be already underway. Access to high-speed rail services may provide important competitive advantages to those cities that are on the high- speed rail network compared with those that are not on the network and have therefore fewer train services. According to the economic literature and experience in other European countries where high-speed rail services have also been introduced, the main wider economic impacts of high-speed rail infrastructure and services appear to be mobility and accessibility, socio- economic structures, urban image and spatial effects. We have found this to be the case for the cities connected by the Madrid-Barcelona high-speed line, particularly Zaragoza and Lleida. Algarve Railway (Portugal) The lack of any form of impact monitoring since the opening makes identifying a causal link between the project and any wider effect very problematic. In this kind of situations, it is difficult to separate the impacts of the project from the impact of other ongoing changes in the economy. Nonetheless, there is some evidence that per capita income might have improved in the regions connected by the railway line. It is however problematic to attribute these changes just to this project. Railway Thriassio – Kiato (Greece) This new line is approximately three years into its expected economic life, whilst economic and social impacts often take many years to materialise. In addition, the recent economic downturn is likely to have offset some of the potential benefits of the project. As a result, the wider impacts of the project are hard to isolate. Nonetheless, we were able to identify the following findings: a number of stations on the new line are in peripheral locations relative to their previous town centre positions. As a result they are now less accessible compared to the stations located on the old routes; the new out-of-town stations do offer a focal point for development and have been integrated into local town master planning exercises;
Executive Summary
12 Frontier Economics, Atkins, ITS | June 2011
the removal of trains from the built-up areas, and associated level crossings, has offered local congestion benefits; We note, however, that the true wider socio-economic benefits of the project are unlikely to materialise until the line completes the rail link between Patra port and Athens. Bratislava Railway Upgrade (Slovakia) The project was implemented during a period of rapid economic growth and increasing car ownership. As a result, passenger numbers on the line have actually fallen relative to 2005 levels. Moreover, the continued works on other segments of the line have caused delays and cancellations further contributing to the fall in passenger numbers. Finally, due to the global economic recession in 2008-2009, it is difficult to attribute regional development impacts directly to the project. However, we found that the project had contributed to meeting aspirations of Slovak transport policy to increase accessibility to wider EU markets. It has also been successful in improving communication and trade links between the two major economic centres of Bratislava and Košice. The improved link also offers increased accessibility to jobs located in the major urban areas of Bratislava and Trnava. A2 Motorway (Poland) The project was implemented against a backdrop of rapid economic growth in Poland. Therefore, it is difficult to establish a causal link between the A2 project and specific impacts. Nonetheless, it is clear from our consultation with stakeholders that the project (and future proposals to extend the A2 to Warsaw and proposals for the north south A1 route) is regarded as a key enabler of economic growth in central Poland. Key benefits of the project identified by stakeholders include: faster and more reliable connectivity between Poland and neighbouring countries; increased foreign direct investment in Poland; quality of life improvements through reduced traffic on the bypassed routes; increased land values along the corridor; and supported tourism access Adverse impacts reported include noise and poorer air quality on the existing R14 (at the eastern end of the A2 motorway). A23 Motorway (Spain) We note that the subprojects under analysis in this evaluation are relatively minor compared to the whole of the A23. It is therefore difficult to establish causality
Executive Summary
June 2011 | Frontier Economics, Atkins, ITS 13
links between these investments and any wider impact. Moreover, the A23 is one of the many interventions that affected the region during the same period. However, for the purpose of this evaluation, we managed to isolate the impacts that are more specific to the segments considered. In relation to the three subprojects located in the province of Teruel, we obtained information about two new infrastructure projects (an industrial park and an airport) developed in the area since the opening of the first road segments. In relation to the subproject Huesca – Nueno, new infrastructures (an industrial park, a technology centre, and new university buildings) have been developed along the A23 near Huesca since the opening of the new road. Agios Konstantinos Bypass (Greece) This project was implemented during a period of economic change, making it difficult to isolate its wider impacts. That said, stakeholders felt that the project had helped unlock the following benefits: it allowed the removal of traffic from bypassed settlements to enable urban re-modelling; it facilitated drainage works to address historical flooding issues; it led to improved air quality and noise conditions; and it enhanced ferry port access. Collectively such impacts are helping Agiou Konstantinos to develop as a tourist location. M1 Northern Motorway (Ireland) It is difficult to establish a link between the project and wider socio-economic impacts as the project was implemented during a period of rapid growth in the Irish economy. It is however clear that the project has formed an important component of the general upgrade of the Ireland‟s transport network. The construction of the project was a key factor in facilitating the construction of the second terminal at Dublin Airport. There is also evidence of substantial growth in the towns adjacent to the project, where congestion levels have dropped as a result of the M1 Northern Motorway. IX B Corridor (Lithuania) Due to the rapid economic growth experience by Lithuania during the construction phase, the specific wider impacts of this project are hard to isolate. Nonetheless, it is clear that the project contributed significantly to the upgrading of the Lithuanian transport network. Together with other projects, it helped set up Free Economic Zones and open new logistic centres. It also contributed to the regeneration of brownfield locations along the motorway (for example with the set up of new shopping and residential centres).
Executive Summary
14 Frontier Economics, Atkins, ITS | June 2011
M0 Budapest Ring Road (Hungary) The M0 Eastern Sector and M31 are youthful in terms of their expected economic life. The M0 opened 2 years ago and the M31 four months ago. As such, their impacts on society and the economy have so far been limited. Additionally, the recession has affected the economy throughout the period the M0 has been open. Primarily as a consequence of their youthfulness, quality of life benefits/costs (accessibility, air quality and noise disturbance) are thoe impacts most readily observable. Accessibility benefits have been included in the CBA and therefore we have not taken them into account in this case. There has been, however, some economic development along the M0 corridor (e.g. retail and logistic centres) and it is expected that more will occur, as municipalities are zoning more land in the locality of the M0.
Key findings from the ex post evaluation Some of the main contributions of ex post evaluation programmes are: raised transparency and accountability; strengthening the evidence base; and providing feedback on the ex ante process. We use these categories to illustrate our findings from the ten transport projects regarding the benefits of ex post evaluation to project stakeholders (including promoters), Member States and the European Commission.
Transparency and accountability. Overall, the ten transport projects examined in this report demonstrate an acceptable or good value for money from the perspective of the European taxpayer. There were however some exceptions. Whilst these general finding represent positive news, in the spirit of transparency and accountability, the publication of such ex post findings, including those that identify a less satisfactory outcome, does allow stakeholders, Member States and the Commission to ask hard and sometimes uncomfortable questions of each other and the Member State‟s planning authorities, which fosters accountability.
Strengthening of the evidence base. With only 10 transport projects considered, the sample is too small in itself to demonstrate the existence of any statistically significant patterns. However, with a full ex post programme, a more extensive evidence base can be developed. This can then aid in the planning of future transport projects. We summarise below our findings in key areas. Capital costs. We have found that the average of the difference between ex ante and ex post capital costs across the ten projects is
Executive Summary
June 2011 | Frontier Economics, Atkins, ITS 15
13.5%. This value is rather low and in line with current experience. Nine projects have experienced cost overruns. The following factors may contribute to explaining this difference between ex ante and ex post values:
Firstly, the large cost overrun is associated with ex ante costs that were developed well in advance of construction;
then project delays, alterations in scope and other unforeseen circumstances all increased capital costs;
finally, we note that some Member States took the opportunity when applying for Cohesion Funds to update the ex ante CBA with a more up to date set of capital costs. Utilisation rates. Generally, the utilisation rates we found are compatible with the objective to build in sufficient spare capacity to give room for growth over a project‟s lifetime. However, in one case (A23 Motorway in Spain), the utilisation rate was very low, while another project (M1 Northern Motorway in Ireland) shows very high levels of utilisation. These examples sugges an imbalance between ex ante demand forecasts and actual demand. NPV. Seven out of the 10 case study projects yielded an NPV that was lower ex post than expected ex ante. We have also found that in eight out of ten transport projects differences between ex ante and ex post travel demand are a primary cause of the differences in NPV, whilst in five transport projects differences in capital costs also contribute to this discrepancy. Whilst a sample size of 10 cannot provide any statistically robust conclusions, it appears to us that these findings, in conjunction with some low utilisation rates and the actual cost overruns, are indicative of optimism bias being prevalent in the ex ante analysis.
Feedback. An important attribute of CBA ex post analysis is the feedback it can provide on demand modelling, ex ante appraisal and infrastructure design. Some of the lessons learnt from our analysis of the ten transport projects are: Modelling practice. Ex ante demand modelling would benefit from improvements in study area definition, modal coverage and inclusion of induced traffic/variable demand.
Improved definition of the counterfactuals. We have identified inappropriate assumptions regarding travel demand, traffic growth and vehicle speeds in the „without‟ project scenarios. For rail and toll motorway projects it is essential that the pricing policy is defined correctly. Not only does the pricing policy affect the
Executive Summary
16 Frontier Economics, Atkins, ITS | June 2011
distribution of benefits between operators and users but it also strongly influences the demand for a project. CBA practice. The CBA analyses would have benefited from:
more consistency in scope, the inclusion of environmental externalities in the CBA and the incorporation of network effects;
a disaggregation between business and non-work transport user classes;
an opportunity to improve the comparability of CBAs for decision- makers, by moving towards harmonised definitions, measures (units), methods and assumptions where appropriate – so that for example the values of time, safety and environmental impacts used in CBA will be on the same basis, even if they will not (rightly) be exactly the same throughout the EU. Risk analysis. The main sources of differences between ex ante and ex post economic analyses were capital costs and travel demand. Differences in travel demand occurred due to variances between ex ante and ex post in terms of economic growth, development impacts and whether other transport projects in the locality had been constructed. A more comprehensive risk analysis at the ex ante stage would have enable the assessment of these effects. Identification of wider impacts. The lack of project monitoring frameworks, to be implemented at project opening, makes the identification of wider impacts particularly challenging. This is especially the case for projects being implemented as part of a wider modernisation strategy, during periods of economic change. In these cases, identifying a causal link between a project and wider socio- economic impacts is challenging. Project design. The ex post analyses did not identify any unintended consequences aside from impacts on noise that should have been mitigated against. This suggests improvements in the design of such mitigation measures are needed, however, the sample size is too small to make a general statement based on this study. Good practice guides and regulation. The desire for and need of good practice guides on economic appraisal and demand modelling was identified both in our interviews and in the Member States workshop. There was a mixed response to the need for more regulation.
Executive Summary
June 2011 | Frontier Economics, Atkins, ITS 17
Cost-benefit analysis as a method One of the objectives of the study was to assess cost-benefit analysis as a method. To do so, we considered five aspects of such an assessment as specified in the TORs.
Strengths and weaknesses of CBA (as applied by Member States): Ex ante CBA is a mature method regularly used worldwide. There is a basic level of consistency in the use of CBA methodology for the appraisal of transport infrastructure across EU Member States. Our findings from the ten transport projects confirm the following: the CBA framework used for the appraisal of these projects is both consistent with DG REGIO‟s guidelines and with good practice. All the appraisals consider investment costs (including ongoing maintenance costs) on one side and measure the user benefits as well as some externalities. Future benefits are discounted correctly and some risk analysis, albeit limited to a handful of assumptions, has been undertaken; however, we found that the scope of the ex ante analysis has often been narrower than what would be ideal. Often, the analysis only considers times savings, vehicle operating cost savings and safety benefits, while
other impacts (for example, CO2 emissions, noise, pollution) are not always taken into account; different EU Member States use different appraisal parameters. This in itself is not an issue, but there appears to be little similarity in some instances between the values used and other sources for these values, such as HEATCO; we also identified weaknesses in the demand modelling, the definitions of the counterfactuals and the modelling of network effects. We note that a robust treatment of demand and the counterfactuals is fundamental to the reliability of a cost-benefit analysis.
Effectiveness of CBA in decision-making (supporting the project generation and project decision of the Member States and the Commission). Whilst cost-benefit analysis is regularly used worldwide, it forms only part of a broader decision-making process – even in countries with a long history of using it. For many of the projects we reviewed, we found that CBA had been used mainly to confirm that the project offered value for money and to support the application to access EU funding. Furthermore, it is clear from our discussions with stakeholders that this limited role still remains in some EU Member States. One of the reasons, aside from the time it takes to embed CBA into existing planning processes,
Executive Summary
18 Frontier Economics, Atkins, ITS | June 2011
is that often projects are motivated by environmental and regional development objectives. As these factors are not generally fully included in the CBA, there may be concerns that CBA may lead to the prioritisation of less desirable projects. However, we think that even within these constraints, CBA can offer relevant information and be a useful tool to decision-makers compared to the extent to which CBA is currently used. at a technical level, CBA is particularly strong in comparing projects that have relatively similar impacts. Thus, it can be used to help make design choices, alignments (depending on the environmental impact), and help prioritise different elements of a transport investment strategy; at a political level, CBA also adds transparency to the decision making process, and improves the accountability of the decision-makers.
Utility of ex post CBA (from the point of view of the project promoters, Member States and the Commission). Ex post CBA can add value to the planning process. t adds transparency, strengthens the evidence base and provides feedback on the methods and techniques used to design and appraise the infrastructure. For these reasons, active ex post CBA programmes have been in place for some time in the UK and in France, for example. Perhaps one of their key contributions has been to develop an improved understanding of “optimism bias”: the systematic bias during the planning phases of a project, whereby the benefits are overestimated, whilst costs and completion times are underestimated. Ex post evaluation programmes can help identify instances of optimism bias by identifying cost overruns and their causes.
Ex post CBA as a tool for evaluating project impacts. Ex post CBA‟s main strengths are its holistic approach, that is, its ability to consider a project as a whole, as well as its ability to deliver an unambiguous indicator of a project‟s economic worth. A further strength of ex post transport CBA is that it focuses on the direct impacts of the transport project and its associated externalities. This makes the project monitoring programme reasonably straightforward. The focus on direct impacts is also a weakness, since there can be a gap between project objectives (e.g. increased employment and wealth to a region) and the inputs to a CBA.
Relevance and potential utility of CBA for macro-economic modelling. A key strength of CBA is its ability to deliver an unambiguous indicator of a project‟s worth. Macroeconomic models on the other hand focus on a narrow range of impacts. Transport impacts included in a CBA that are not relevant for macroeconomic modelling include those associated with non- work time savings, safety and environmental externalities. At best, therefore, only a fraction of transport CBA output is relevant for macroeconomic
Executive Summary
June 2011 | Frontier Economics, Atkins, ITS 19
modelling ,namely, that associated with business and freight impacts and any productivity changes due to agglomeration benefits. The value of transport CBA output as a GDP indicator is undermined for two reasons: it is a „net‟ benefit, that is, a series of gains and losses will occur as the economy adjusts to a change in transport supply and economic activity relocates in response to changes in transport accessibility; and transport CBA measures the added value of bringing new resources into use (increased output and employment), while macroeconomic modelling measures the gross value. These differences between CBA and macroeconomic modelling mean that in practice the macroeconomic impacts of changes in accessibility are best modelled by linking transport accessibility models directly with macroeconomic models. This can lead to some inconsistencies between the economic forecasts from the macroeconomic model and those that underpin the transport model. These inconsistencies can be overcome through either the use of a Land Use Transport Interaction (LUTI) model or a Spatial Computable General Equilibrium (SCGE) model – neither of which are trivial modelling exercises.
Recommendations The study offers a set of recommendations on how to improve the practice of ex ante project appraisal in the context of current EU regional policy instrument processes and practices. Specifically, based on the findings of our study, we recommend the Commission to undertake a review of the following areas:
Quality and consistency of ex ante CBA. Consider changes that could be introduced into current regulations to improve the quality and consistency of the ex ante CBA analysis applied to major transport infrastructure projects supporting EU funding applications. These changes could include a set of objective indicators regarding scope and depth of the CBA that the application of the DG REGIO CBA guidelines should adhere to. Such indicators would relate to, among others, issues such as appropriate network definition to account for induced and redistributed traffic; appropriate demand analysis and forecast modelling; thorough assessment of environmental impacts – including climate change impacts – on the basis of the latest available guidelines in the CBA literature; quantified risk and sensitivity analyses; or compulsory inclusion of a lower-cost alternative option to the proposed investment projects in the analysis submitted as part of the funding application.
Ability to undertake ex post evaluation. Examine the possibility of changing current regulations to improve the “evaluability” of major
Executive Summary
20 Frontier Economics, Atkins, ITS | June 2011
transport infrastructure projects that benefit from EU funding to better assess their “value for money” ex post. Such improvements would require, for example, a definition of “value for money” in terms of the opportunity cost of the public funds used in the project, reaching from control of costs through to delivery of outcomes, including system-wide impacts and more focused outputs; requirements to set up a monitoring framework for data collection and evaluation; set up fixed intervals to monitor ramp-up and distribution effects and carry out ex post evaluations; systematic data collection efforts; and assessment of potential role of alternative project appraisal techniques;
Provision of training and further guidance. Create or facilitate ways to provide guidance and train public sector officials on the use of ex ante CBA analysis and ex post project “evaluability” design. This could be achieved by, for example, sharing examples of good practice amongst practitioners; offering qualified and targeted training on specific aspects of the CBA analysis; and, creating opportunities for project promoters and potential consultants to learn and share their knowledge.
Executive Summary
June 2011 | Frontier Economics, Atkins, ITS 21
1 Introduction
Frontier Economics, together with Atkins and the Institute for Transport Studies (University of Leeds) is pleased to present this Final Report for the study “Ex post evaluation of cohesion policy interventions 2000-2006 financed by the Cohesion Fund – Work Package B: Cost-benefit analysis of selected transport projects”.
1.1 Context The Cohesion Fund was established in 1993 with the objective to strengthen the economic and social cohesion of the European Union.4 The eligibility criterion is that the GNP per capita is 90% or less than the EU average. During the period 2000-2006, 17 countries have received funding from the Cohesion Fund and ISPA (Instrument for Structural Policies for Pre-Accession, aimed at accession countries) for transport and environment projects. The Council Regulation5 states that “In order to ensure the effectiveness of Community assistance, the Commission and the beneficiary Member States shall, in cooperation with the EIB where appropriate, carry out a systematic appraisal and evaluation of projects” However, the objective of the ex post evaluation is not just to comply with regulatory requirements but crucially to establish the economic and social impacts of such EU policy interventions, to learn from experience gained in practice, and to reflect on the methods used to select, design and appraise projects. The aim is to use the results of ex post evaluation to try to improve project design and appraisal in future cohesion policy programmes.
1.2 Background This study focuses on the ex post evaluation of ten selected transport projects in EU countries benefitting from EU Cohesion and ISPA funding during the period 2000-2006. The study forms part of three studies commissioned by DG REGIO to establish the effect of the Cohesion Fund and ISPA on economic and social cohesion and draw lessons for the future. These studies are package A (overall); package B (transport); and package C (environment). The Terms of Reference (TORs) for Work Package B identify the three key questions that this study addresses:
4 Council Regulation (EC) No 1164/94 of 16 May 1994 establishing a Cohesion Fund [Official Journal L 130, 25.05.1994]. 5 Ibid.
Introduction
22 Frontier Economics, Atkins, ITS | June 2011
Question 1: What were the impacts of the examined projects?
Question 2: How can ex post cost benefit analyses (CBA) contribute to the practice of ex ante cost-benefit analysis?
Question 3: What are the potential and the limits of ex post CBA as a tool to identify the impact of infrastructure projects? The first question is specific to the ex post evaluation of the ten transport projects selected. During the period 2000-2006, there were 260 individual transport projects co-financed by the Cohesion Fund, for a total eligible value of about €8.6bn and contributions from the Cohesion Funds of about €7.2bn. The TORs of this study shortlisted 40 transport schemes, covering a total of 103 individual transport projects. Therefore, out of the 40 schemes identified in the TORs, we selected 10 of them for in-depth ex post evaluation. In aggregate, these schemes encompass 38 individual transport projects, that is about 15% of the individual projects co-financed by the Cohesion Fund in the period 2000- 2006. The analysis focuses on whether the analysed projects have achieved the social and economic objectives for which they were designed by national authorities and co-financed by the Cohesion Fund. The analysis also verifies whether the project objectives were achieved in the most cost-efficient manner, without cost overruns or time delays. The analyses of Questions 2 and 3 follow on from the findings of the analysis of Question 1 but with a review of methodological issues. The questions focus on (i) assessing the potential future role of ex post CBA as an evaluation tool; and (ii) on how ex post CBA can be used to improve the practice of ex ante project appraisal. This assessment also takes into account the institutional capacity of the Cohesion Fund beneficiaries to undertake CBA exercises.
1.3 Objectives This Final Report concludes the study. It builds on the findings of the Second Interim Report and the discussion and conclusions of the workshop with EU Member State representatives which took place in Brussels on the 3rd of February 2011, Specifically, to answer the fundamental questions of the evaluation listed above this report has two main objectives.
Summary of results of the evaluation. To address Question 1, we summarise the findings of the evaluation of ten transport projects. First, we compare the methodologies used for the ex ante analysis carried out by Member States as part of the applications for EU funding. Second, we
Introduction
June 2011 | Frontier Economics, Atkins, ITS 23
present the results of the ex post evaluation, introducing a framework that should facilitate the comparison of future projects with those considered in this study. Finally, we explores the causes for differences between the ex ante appraisal and the ex post evaluation that we have observed.
Assess cost benefit analysis as a method. To address Questions 2 and 3, as requested by Task 3 of the TORs, we assess: the strengths and weaknesses of CBA methodologies as applied by the Member States; effectiveness of CBA as a tool supporting the project generation and project decision of the Member States and the Commission; utility of ex post CBA from the point of view of the project promoters, Member States and the Commission; the extent that ex post CBA is an appropriate tool for evaluation of impact assessment; and, relevance and potential utility of data obtained from CBA for macro- economic modelling.
1.4 Structure The structure of this Final Report is the following:
Chapter 2 provides an overview of the overall approach and methodology we have used for this study.
Chapter 3 describes the roles of cost-benefit analysis (CBA) in European transport infrastructure planning taking account of its strengths and weaknesses for ex ante appraisal of transport infrastructure projects.
Chapter 4 summarises the findings of our evaluation of the ten transport projects; describes the results of our review of the ex ante analysis; presents the results of the ex post evaluation; and discusses potential causes for discrepancies between the ex ante and ex post analyses.
Chapter 5 presents our assessment of CBA as a project or policy appraisal method based on the findings from the ten transport projects taking into account the experience and lessons learnt from the use of ex post CBA across Europe.
Chapter 6 provides a summary of the macroeconomic impacts of transport investment (a requirement included in the TOR of the study) and considers
Introduction
24 Frontier Economics, Atkins, ITS | June 2011
the relevance and potential utility of data obtained from CBA for macro- economic modelling.
Chapter 7 contains our conclusions and recommendations. The annexes and appendices of the report cover the following ground: Annexe 1 provides a one-page summary for each project, summarising the main findings of the evaluation. Annexe 2 complements chapter 2 with a survey of the strength and weaknesses of CBA as appraisal method and of multi-criteria analysis, an alternative project appraisal method; Annexe 3 provides a list of references we have used for the literature review presented in Chapter 2 and Annexe 2; Annexe 4 complements the findings of chapter 3 and 4 with a survey of the experience of ex post CBA across Europe; and, Annexe 5 complements chapter 5 with a more detailed discussion of the links between transport CBA and macroeconomic modelling Appendices 1 to 10 contain the complete analysis of the ten transport project covered in this study.
Introduction
June 2011 | Frontier Economics, Atkins, ITS 25
2 Approach and methodology
This chapter presents a summary of the approach we have followed to carry out this study. In addition, it provides details on the technical methodology we have used to undertake the evaluation of the ten transport projects. As requested by the TORs, we have based our methodology on the most recent European Commission‟s guide to cost-benefit analysis („the EC Guide‟)6 and integrated it, when necessary, with existing CBA best-practice guidance. The study has involved undertaking three sequential tasks.
Task 1: Selecting 10 major transport projects for ex post cost-benefit analysis.
Task 2: Ex post project impact analysis.
Task 3: Assessing cost-benefit analysis as method. In the remainder of this chapter, we describe how we carried out each task.
2.1 Task 1: Selecting ten transport projects for the ex post cost-benefit analysis The TORs provided a list of 40 transport projects that could potentially be included in this study. The objective of Task 1 was the selection of ten transport projects from the original list. These projects would then be the subject of a complete ex post cost-benefit analysis. Three sequential subtasks have been necessary to complete Task 1: Task 1.1: set criteria to generate a shortlist of 20 projects; Task 1.2: prepare, for each of the shortlisted projects, a three-page description; and, Task 1.3: select, on the basis of these descriptions, ten projects for an ex post evaluation in Task 2. We describe each in turn below.
2.1.1 Task 1.1 – Set criteria for short-listing 20 projects To prepare a short-list of 20 projects out of the proposed list provided in the TORs, we defined a list of criteria. Our overarching objective was to maximise
6 European Commission (2008) Guide to Cost Benefit Analysis of Investment Projects. Published on: http://ec.europa.eu/regional_policy/sources/docoffic/working/sf2000_en.html
Approach and methodology
26 Frontier Economics, Atkins, ITS | June 2011
the representativeness of the shortlist, ensuring that the information available for the chosen projects would be sufficient to carry out a meaningful ex post CBA. Specifically, our selection aimed at balancing the following key criteria.
Size of EU funding: other things being equal, the project shortlist should be relevant in terms of the amount of taxpayer moneys allocated by the European Commission.
Regional distribution of EU funding: the project shortlist should also be representative of the distribution of funding across various regions. Specifically, the selection should be representative of the relative size of the funding for projects in Greece, Ireland, Portugal and Spain compared to the size of funding for projects in Central and Eastern Europe.
Modal split of EU funding: the project shortlist should also reflect the distribution of EU funding across the various transport modes (rail, road, airports and ports)
Quality and availability of information: finally, while compiling the shortlist, we should also consider information quality and data availability issues, including the quality of ex ante project appraisal. As a general rule, the shortlist will tend to include those projects for which information is either available or readily obtainable. An implication of this criterion is that we should select only those projects which are complete as this allows the comparison between ex-ante analysis and ex-post outcomes. In the remainder of this section, we show how we applied these criteria to come up with a 20-project shortlist for Task 1.2.
Amount of EU contributions: The projects included in the shortlist should try to reflect the amounts of EU contributions received by the 40 projects included in the TORs. The 20 projects with the highest Cohesion Fund contributions account for 80% of the total amount of contributions in the original list of 40 projects during the 2000-2006 period. Therefore, subject to other criteria, we have used the amount of EU funding as a first filter to select those projects which account for the largest shares of EU funding.
Regional distribution: the distribution of funding across regions in the shortlist of 20 projects should be similar to that distribution in the original list of 40 projects. Table 1 below shows that, in the original list of 40 projects the number of projects in Greece, Ireland, Portugal and Spain is the same as that in Central and Eastern Europe. However, the projects in Greece, Ireland, Portugal and Spain account for approximately 64% of the
Approach and methodology
June 2011 | Frontier Economics, Atkins, ITS 27
EU contributions, while projects in Central and Eastern Europe countries account for 36%.
Table 1. Regional distribution in the original 40-project list.
Regional distribution Number of projects % of CF contributions
Greece, Ireland, Portugal 20 63.8% and Spain
Central and Eastern 20 36.2% European countries
Source: Own calculations using ITT data for 40 selected projects.
Modal split: The distribution of funding across different transport modes in the 20 projects shortlist should be similar to that distribution in the original list of 40 projects. Table 2 shows that there are 19 rail projects, 19 road projects, 1 port project and 1 airport project in the original list of 40 projects included in the TORs. However, rail projects tend to have larger capital costs than road projects, thus accounting for 62% of total funding. Road projects account for 34%, while the only port and airport projects account for 3% and 1% of EU funding, respectively.
Table 2. Transport mode distribution in the original 40 projects list.
Sector Number of projects % of CF contributions
Airport 1 0.7%
Port 1 3.4%
Road 19 34.3%
Rail 19 61.5%
Source: Own calculations using ITT data for 40 selected projects.
The airport project (Tallinn Airport) and the port project (Ampliación del Puerto de Gijón) are relatively small in size. Even if selected for the ex-post CBA, it is not clear that those projects are representative enough to be able to draw meaningful conclusions for the same or other transport modes. For this reason, we decided, in agreement with the Evaluation Unit of DG REGIO, to exclude these two projects a priori from the selection process. Table 3 brings together the regional and transport mode distributions after excluding from the original list of 40 projects the port and the airport projects. It
Approach and methodology
28 Frontier Economics, Atkins, ITS | June 2011
shows the percentage of funding for each region-transport mode combination. The funding for rail projects in Greece, Ireland, Portugal and Spain is approximately three times as large as the funding for road projects. This picture is somewhat different in Central and Eastern Europe, where a larger share of EU funding is channelled to road projects.
Table 3. Distribution of funding between regions and modes (initial 40 projects)
Greece/Ireland/ Central and Eastern Europe Portugal/Spain
Rail 47.3% (11) 15.6% (8)
Road 16.9% (8) 20.2% (11)
Source: Own calculations using ITT data for 40 selected projects.
In preparing the 20-project shortlist we tried not to deviate too much from the distribution of funding in the list of 40 projects shown in Table 3. However, as noted above, we have had to balance the criteria of representativeness (both for transport mode and for region) with the need to include projects with good and sufficient information to facilitate the successful completion of Tasks 2 and 3. Table 4 presents the list of 20 projects we selected for Task 1.2. and the amount of EU funding received by each project.
Approach and methodology
June 2011 | Frontier Economics, Atkins, ITS 29
Table 4. 20-project shortlist
Cohesion Fund Transport Country Project name contributions Mode (Euro)
Spain High speed railway Madrid–Barcelona–French border Rail 1,014,006,060
Spain High speed railway Madrid – Levante Rail 541,281,940
Spain Línea de Altā Velocidad Ferroviaria Madrid-Valladolid Rail 490,847,732
Greece Egnatia Odos Motorway Road 341,449,971
Poland A2 Motorway (Konin – Strykow) Road 325,325,094
Portugal Modernisation of Ligne du Sud (Lisboa – Faro) Rail 323,486,946
Portugal Modernisation of Ligne du Nord (Lisboa – Porto) Rail 214,227,610
Spain Motorway A23 (between Levante and France) Road 156,734,306
Greece Agios - Konstantinos by-pass Road 146,062,798
Ireland M1 Road 144,604,730
Greece New Railway Line (Thriasio – Korinthos) Rail 128,500,000
Lithuania IXB Transport Corridor Road 121,868,535
Rehabilitation of Budapest – Cegled – Szolnok – Hungary Lokoshaza Railway Line Rail 119,818,800
Slovak Trnava – Nové Mesto nad Váhom, track modernisation Rail 116,730,099 Republic
Ireland Heuston terminal and South-West rail corridor Rail 74,970,000
Romania Bucharest - Cernavoda motorway Road 71,712,000
Slovak Šenkvice to Cífer, track modernisation Rail 58,429,500 Republic
Eastern Section of the M0 Budapest Ring Road Hungary between National Road 4 and M3 Road 284,659,050
Czech D8 Motorway (Usti nad Labem – Czech - German Road 52,000,000 Republic Border)
Greece New railway (Korinthos - Kiato) Rail 49,900,000
Source: European Commission for the funding data. Frontier Economics, Atkins and ITS for elaboration
The total amount of Cohesion Fund contributions received by the 20 projects included in the shortlist amounts to 66% of the total amount of contributions received by the original 40 projects. Therefore, we believe that our selection was representative of the total amount of EU funding in the original list of 40 projects.
Approach and methodology
30 Frontier Economics, Atkins, ITS | June 2011
Table 5 shows the distribution of EU funding across modes and regions in the 20-project shortlist. It can be compared with Table 3 to verify the representativeness of this selection.
Table 5. Distribution of EU funding between regions and modes (20 project shortlisted)
Greece/Ireland/ Central and Eastern Europe Portugal/Spain
Rail 59.4% (8) 6.2% (3)
Road 16.5% (4) 17.9% (5)
Source: Data from the EC REGIO TORs and elaboration of Frontier Economics.
Rail projects in Greece, Ireland, Portugal and Spain account for a relatively larger share of EU funding than in the original 40 project list from the TORs. However, these differences are mainly due to the need to select projects with sufficient data availability in order to ensure that the ability to carry out meaningful analysis should these projects be selected for Task 2. Overall, we believe that the outcome of this project selection process did not deviate too much from those the initial list included in the TORs.
2.1.2 Task 1.2 – Project descriptions As requested in the TORs, we prepared a short description for each of the 20 shortlisted projects. We relied on a variety of different sources to prepare these short project descriptions. First of all, we used EC Funding Decision documents and, in rare cases, Final Reports, which provided information about project objectives, outcome of ex ante CBA analysis and completion data. At times, they also identified relevant contacts at the Member State level. We sourced additional information by liaising with desk officers at the DG REGIO‟s Geographic Unit. We also obtained documents, data or confirmation of their existence from the relevant contacts in the Members States. Finally, whenever possible, we relied on publicly available information, such as traffic count databases. After filling the short project description with all the information gathered, we assigned scores to each project based on a set of mostly qualitative criteria. Each criterion has a different weight, to express its relative importance in the project selection.
Likely level of support from key contacts involved in the evaluation process. This criterion considers the extent to which, , we would be able to obtain support from the relevant contacts at Member State level if we were
Approach and methodology
June 2011 | Frontier Economics, Atkins, ITS 31
to carry out an ex-post CBA. We have assessed this criterion on the basis of the level of support we have received for the completion of the project descriptions. We have assigned a maximum 10% weight to this criterion for the purposes of calculating the project score.
Comprehensiveness of ex ante evaluation baseline data which is readily available. This criterion measures the completeness of baseline data that were used for the ex ante CBA. It is the most important criterion since the availability of comprehensive baseline data facilitates the preparation of the ex post CBA framework and its comparison with the ex ante appraisal. We have assigned a maximum 55% weight in the project score to this criterion since we believe it is the most important one.
Availability of primary / secondary data from existing sources. This criterion captures the extent to which the information needed for ex post cost-benefit analysis is already available. The availability of this information will also have an impact on the ability to compare ex ante and ex post analysis in Task 3. We have assigned a maximum 20% weight to this criterion in the project score..
Amount of new ex post evaluation primary data required to support evaluation. This criterion measures the extent to which new data is needed to carry out an ex post CBA. We have assigned this criterion a maximum 15% weight in the project score. To arrive to a total score for a given project, we assigned a rating between 0 (minimum or very low standard) and 5 (maximum or very high standard) to each criterion. The project score is the weighted sum of the ratings assigned to each criterion. Table 6 summarises the rationale for each rating and criterion.
Approach and methodology
32 Frontier Economics, Atkins, ITS | June 2011
Table 6. Date-Related Prioritisation Scoring Matrix (Range of points depending on the Standard reach across criteria)
Criteria (weights) High Medium Low Standard Standard Standard (4 or 5) (2 or 3) (0 or 1)
Likely level of support We have established We have been able to Getting hold of from key contacts very cooperative and establish a contact documents and involved in the helpful contact(s) who who can provide most information to support evaluation process is able to facilitate the information but there the evaluation process (10%) data collection and are still a number of has been very overall evaluation outstanding data challenging and largely process. requirements that will unsuccessful. Unless require resolving if we the EC can assist further are to proceed with an this is not a suitable ex post evaluation candidate for further evaluation
Comprehensiveness of Comprehensive pre- Sufficient data is There is no observed ex ante evaluation project data has either available to support an information regarding baseline data that is been provided or can evaluation of the most traffic flows, passenger readily available (55%) be provided easily and crucial aspects of the numbers, journey times, is sufficiently detailed evaluation. accident rates, etc. to to allow a function as a baseline. comprehensive project evaluation
Availability of primary / The amount of The amount of There is little or no secondary data from information available information available information which existing sources (20%) to us is likely to allow to us is variable in provides very little for a very quality or has gaps opportunity for us to comprehensive and as such will learn lessons about evaluation and provide some project appraisal or comparison with ex contribution to the evaluation methods as ante appraisal in Task conclusions derived part of Task 3. 3 from Task 3.
Amount of new ex post Little or no extra data A relatively significant There is currently no evaluation primary is required as it has level of primary data is post-opening data data required to already been collected required to support the available to support an support evaluation through a process of evaluation. Specific evaluation. New data (15%) post-opening examples might will need to be collected evaluation include traffic counts at significant cost. or journey time surveys, passenger surveys, etc.
NB: The maximum score for any criterion is 5
Figure 1 illustrates how we have aggregated the weighted ratings for each criterion into a single score for each project (out of 100).
Approach and methodology
June 2011 | Frontier Economics, Atkins, ITS 33
Figure 1. Illustration of calculation of overall score
Level of support Partial 10% (0 to 5) = weighted
Ex ante data availability 55% Partial (0 to 5) = weighted + Ex post data availability 20% Partial (0 to 5) = weighted
New data requirements 15% Partial (0 to 5) = weighted
Final score FINAL SCORE 20 (weighted) = (Out of 100)
2.1.3 Task 1.3 – Selection of ten transport projects for ex post CBA Using this scoring formula explained above, we identified the ten projects with the highest scores, which we proposed to DG REGIO for consideration in Task 2 of the study. On this basis of our proposal, DG REGIO selected ten projects to be taken forward in the study. Table 7 summarises the final selection. This completed Task 1 of the study.
Table 7. Summary of candidate projects
Road Rail
M1 Northern Motorway Railway line Thriassio-Pedio- (Ireland) Eleusina-Korinthos-Kiato Greece/Ireland/ (Greece ) A23 Motorway (Spain) Portugal/Spain Algarve Railway (Portugal) Agios Konstantinos Bypass (Greece) AVE Madrid - Barcelona (Spain)
IX B Corridor (Lithuania) Bratislava Railway Upgrade (Slovakia) Central and A2 Motorway (Poland) Eastern Europe M0 Budapest Ring Road (Hungary)
Source: Frontier Economics, Atkins and ITS – Projects scores are out of 100.
Approach and methodology
34 Frontier Economics, Atkins, ITS | June 2011
2.2 Task 2: Ex post project impact analysis The key objective of this task was to answer the first questions asked by the TORs, specifically:
Question 1: What were the impacts of the examined projects? This task was at the core of the study, and its success depended on the ability to carry out in-depth ex post CBAs on the ten selected projects. In accordance with the TORs, we first carried out an in-depth ex post cost- benefit analysis for two pilot projects for the First Interim Report. We then completed the remaining eight ex post evaluations for the Second Interim Report. We also updated the analysis of the pilot projects based on the feedback on the First Interim Report we received. The final version of the analysis for each of these projects is provided as appendices to this report. As indicated in the TORs, this task has three subtasks for each of the selected projects: Task 2.1 – A review of ex ante cost-benefit analysis; Task 2.2 – An ex post cost-benefit analysis; and Task 2.3 – A comparison of ex ante and ex post cost benefit analyses. In accordance with the TORs, we first carried out an in-depth ex post cost- benefit analysis for two pilot projects. These projects were:
Pilot 1: Construction of the A2 motorway between Konin and Strykow, Poland
Pilot 2: Modernisation of the Lisbon - Algarve railway, Portugal We reported the results of the pilot analysis in the First Interim Report of this study. The final ex post evaluations of these projects are provided as appendices to this report.
2.2.1 Task 2.1 – Review of ex ante cost-benefit analysis For this subtask, we undertook a detailed review of the ex ante cost-benefit analysis of each project. This analysis was prepared by stakeholders at Member State level to support the funding application. For each project, we considered: the technical quality of the analysis; and the role that ex ante CBA played in the decision-making process.
Approach and methodology
June 2011 | Frontier Economics, Atkins, ITS 35
Quality of the Analysis In reviewing the technical quality of the ex ante CBA, we have relied on existing guidance from the Commission (the EC Guide) and integrated it with existing good practice in the field of transport appraisal. Our review focused on assessing the following key aspects.
Capital and operating costs. We considered whether the project costs had been correctly identified and included in the analysis.
Demand analysis. Demand analysis underpins the estimation of most of the benefits generated by a transport project. We reviewed the analysis carried out in each case to determine whether future traffic volumes had been identified correctly.
Economic analysis. We reviewed the analysis of the expected economic benefits of each project. These included benefits from time savings, reductions in vehicle operating cost, safety benefits and other positive externalities (such as reduction in pollution). For each of these impacts, we assessed whether the analysis underpinning their estimation had been carried out according to good practice guidelines.
Financial analysis. In addition to reviewing the economic analysis, we also considered the financial analysis of each project. We note however that most of these projects were not meant to be revenue-generating or self-funded. Rather, the key reason for their development was that they would bring about economic benefits. For this reason, this type of analysis played a secondary role in our review.
Sensitivity analysis and Risk Assessment. We reviewed the sensitivity analysis and, when available, the risk assessment undertaken in the ex-ante CBA.
Alternative options. Finally, we also considered whether the ex ante analysis took into account alternative implementation options in addition to the solution for which a funding application was made. We carried out this analysis mainly by reviewing the available documentation provided by DG REGIO (as part of the supporting material to the actual funding applications) and by stakeholders at Member State level. However, to supplement our knowledge on the issues above, we also had meetings with project stakeholders.
Approach and methodology
36 Frontier Economics, Atkins, ITS | June 2011
Role of ex ante CBA in the decision-making process In addition to discussing the analysis underpinning the ex ante CBA, we used the stakeholder interviews to understand the role that this type of analysis played in the decision-making process that led to the funding application. Specifically, we addressed the following areas of interest: at what stage in the development process of a project was CBA used; the key information provided by the CBA which were used in the decision-process; what decisions the CBA was expected to support (for example, the choice between alternative projects, or, for a specific project, the choice between alternative implementation options); what performance indicators (for example NPV, benefit-cost ratio, Economic IRR) were most important for the decision-makers; and the overall usefulness of ex ante CBA in developing the project and reaching a decision about it.
2.2.2 Task 2.2 – Ex post cost-benefit analysis of selected projects This subtask was the central component of this study. In undertaking this analysis for each project, whenever possible we attempted to follow the same structure that was used for the ex ante analysis. This helped ensure that the results of the two analyses would be comparable, thus facilitating the completion of Task 2.3. In addition to considering the key quantifiable impacts of each project, as requested by the TORs we also attempted to assess, in a qualitative way, any wider socio-economic impacts that the project generated. These include effects on factor markets, society and the environment that are not generally captured by the standard cost-benefit analysis. In this sub-section, first we briefly present the methodology that we used to carry out the estimation of the main impacts we considered in the core cost-benefit analysis. Then we outline the methodology that we have followed to capture the wider socio-economic impacts of each project.
Core cost-benefit analysis
Costs. We considered two categories of costs: capital costs and ongoing costs. With regards to capital costs, we obtained information about actual expenditure incurred for the construction of the project. In order to ensure a like with like comparison, we paid particular attention to ensure that we would consider only the costs specific to the project identified in the TORs.
Approach and methodology
June 2011 | Frontier Economics, Atkins, ITS 37
This was particularly necessary for projects that were part of larger modernisation plans where some cost categories were common to various projects. Regarding ongoing costs, these were covered annual operations and maintenance. As the projects we considered were mostly already operational, we obtained the latest actual operating expenditures from project promoters or operators and assumed that these costs would continue to be incurred for the remaining life of the project.
Demand volumes. Traffic and patronage drive the calculation of a project‟s impacts. For example, the number of travellers switching from road to rail following the introduction of a new rail connection would have a direct implication on travel time savings, as well as on the reduction in vehicle operating costs, polluting emissions and road accidents. As we generally undertook the ex post evaluation a few years after project opening, we used two different approaches to calculating demand volumes for entire life of a project. Specifically: Period from project opening to the time of evaluation: to cover this period, which ranged from one year to six years depending on the project, we used observed data on traffic volumes. In some cases, we obtained this information directly from the project stakeholders. In other cases, we carried out traffic survey or used independent public available sources. Period from the time of evaluation to the end of the cost-benefit analysis horizon: to estimate future demand volumes, we generally used current demand levels as a starting point and extrapolated recent growth. However, to capture the uncertainty surrounding future demand, we defined both a pessimistic scenario (Low Case, with reduced or even no traffic growth) and an optimistic scenario (High Case, assuming that the latest available growth rate would continue indefinitely). Accordingly, as the assumption on demand volumes drives the estimation of the other impacts, we presented the results of the analysis as a range, rather than a single point estimate.
Time savings. Time savings are generally the main source of economic benefits of a transport intervention. In order to estimate total time savings, first we obtained information on improvements in travel time achieved by the project. Then we applied these savings to the number of travellers expected to use each mode of transport, following project opening. These passengers included existing travellers (now benefiting from faster speeds on the same transport mode), passengers switching from other transport modes (air or road) and new travellers, which were not undertaking these journeys before the opening of the new connection. In order to express time savings in monetary terms, we relied on country-specific estimates of the value of
Approach and methodology
38 Frontier Economics, Atkins, ITS | June 2011
time. These values vary according to local economic conditions (for example, GDP per capita) as well as differences in individual preferences. Whenever possible we used the most up-to-date value available. Often, this implied using a different value compared to the one used in the ex ante analysis.
Vehicle operating costs savings. Road travel generates costs. These include fuel consumption as well as general wear and tear. Vehicle operating costs mainly vary according to road conditions and travel speeds. There are two ways in which a transport investment can reduce vehicle operating costs: road traffic reduction: the introduction of an alternative transport mode (such as a high speed rail link) would make some travellers switch, thus eliminating the vehicle operating cost component that was associated to these journey; and road condition improvements: improvements in road surface quality, by reducing wear and tear, can lead to significant reductions in vehicle operating costs, depending on the condition of the road before the intervention (we note, however, that increases improvements in road condition can lead to higher speed, thus offsetting some of the benefits via increase fuel consumption). For this analysis, we used estimates on vehicle operating costs (generally measured on a per passenger/km basis) provided by project stakeholders and applied them to historic and future volumes of road traffic.
Safety benefits. Road condition improvements and a reduction in road traffic can often lead to a decline in the number and severity of accidents. This is often the primary positive externality of transport investments. In order to estimate this impact, we used the latest available annual statistics on road fatalities and casualties since project opening and calculated the difference between these data and those for the time before the project. We then multiplied the difference in accident numbers by the economic value of each accident type. These values are country-specific, and, similarly to the value of time, vary according to economic conditions and preferences.
Other positive externalities. In addition to the impacts considered above, transport investments may also generate additional positive externalities. Generally, these include reductions in pollution levels, CO2 emission, noise and congestion. We note that these impacts were not always considered in the ex ante analyses we reviewed. However, for the ex post evaluation, whenever sufficient information was available, we attempted to include estimates of at least some of these impacts. To estimate them, we used available information on pollution, emissions and noise levels, before and
Approach and methodology
June 2011 | Frontier Economics, Atkins, ITS 39
after the intervention. Then we relied on public available estimates of the marginal cost of these effects to express these impacts in monetary terms.
Wider impacts Quantifying the wider impacts of transport projects on economic, social and physical environments can be challenging. Normally, these are those impacts which cannot be included in the standard CBA, for a variety of reasons. These include: the lack of an agreed methodology to identify their monetary value; the lack of data, which would make the quantification impossible even if data was available; and, the difficulty to isolate a project‟s true impacts with a great deal of certainty, especially in the context of rapidly changing environments (for example, in the newer Member States). Therefore, the approach we used for these impacts was largely qualitative. In order to provide an indication of the likely wider impacts of the projects considered here, we held a series of semi-structured interviews with key stakeholders. The structure of the interview was based upon guidance set out in the Commission‟s „Impact Assessment Guidelines‟ published in January 2009. This guidance provides a detailed list of social, economic and environmental themes that should be considered when evaluating infrastructure projects. Clearly not all of these themes were of immediate relevance to all projects and hence the themes and questions have been rationalised for the purpose of the project under evaluation. Table 8 summarises the key topic areas considered in the semi-structured interviews.
Approach and methodology
40 Frontier Economics, Atkins, ITS | June 2011
Table 8 Summary of topic areas considered in semi-structured interviews
Economy Impacts Social Impacts Environmental Impacts
Economic Development Skills, Labour and Impact on Development, and Investment Employment Regeneration and Land Internal Markets, Quality of Life and use Competitive Trading and Community Impacts Sustainable Travel Investment Public Health and Safety Climate and Air Quality Administrative Burdens Cultural Impacts Biodiversity and Public Authorities Water and Soil Quality Property and Land Values Production of Waste Other Impacts
In addition to these impacts, there may be others that should be included in the cost-benefit analysis but, in some specific cases, were not taken into account. These are mainly environmental impacts, ranging from impact on groundwater pollution to noise and CO2 emissions. Whenever applicable, we discussed them, in a qualitative way, during the semi-structured interview on wider impacts. In these cases, we were mindful of not double-counting them in the actual cost- benefit analysis. We note that the wider impact of a project depends largely on its scale and on whether it is a new infrastructure or not. That is the reason why we have identified more impacts for projects that delivered a new railway or road. Accordingly, the wider impact discussion for these projects is more extensive. On the other hand, projects that involved upgrading existing infrastructures are more likely to generate benefits such as time savings and vehicle operating costs savings. These benefits are already fully captured by the core CBA methodology.
2.2.3 Task 2.3 – Comparing ex ante and ex post cost-benefit analysis To conclude the analysis required by Task 2, in this subtask we compared the results of the ex post evaluations with the outcome of the ex ante cost-benefit analysis reviewed in Task 2.1. In carrying out this Task, as required by the TORs, we explicitly addressed: the appropriateness of both the financial and economic discount rates used in the ex ante CBA; the reasons for any difference between ex ante and ex post financial and economic net present values or rates of return;
Approach and methodology
June 2011 | Frontier Economics, Atkins, ITS 41
the possible reasons for delays and cost overruns, taking into account the methodology and findings identified in Work Package 107; and the ex ante risk assessment, in the light of the actual project outcomes.
2.3 Task 3: Assessing cost-benefit analysis as method In this final task, the subject of this Final Report, we sought to provide an answer to the remaining two key questions asked by the TORs, specifically:
Question 2: How can ex post cost benefit analyses contribute to the practice of ex ante cost benefit analyses?
Question 3: What are the potential and limits to carry out an ex post cost- benefit analysis to identify and/or analyse the impact of projects? Is it an appropriate tool for impact analysis? The lessons that can be drawn from ex post evaluation can help improve the methods of ex ante evaluation, thus offering project sponsors and key decision makers a more robust framework to support their investment decisions. As the sample of projects subjected to ex post evaluation increases over time, conclusions will become more robust and statistically significant. It will increasingly be possible to use ex post evaluations to enhance the formal guidance methods set out in documents such as the most recent EC Guide to CBA. These enhancements to ex ante evaluation should, in turn, result in more accurate future ex post evaluations. The possible lessons learnt and the improvements to ex ante CBA methods can be broadly classified under the following three headings:
Strengthening the evidence base: the use of ex post CBA can contribute to develop a more solid evidence base in several areas of project appraisal, including costing, demand responses to infrastructure enhancements, risk analysis of future projects, and environmental impacts.
Identifying and responding to specific methodological strengths and weaknesses: the use of ex post evaluation can, for example, help unearth issues with the approaches used to assess specific cost or benefit items. It can also help identify alternative approaches or techniques to respond to these issues and rectify them in future assessments.
7 Work Package 10 of the ex post evaluation of the ERDF in the 2000-2006 programming period.
Approach and methodology
42 Frontier Economics, Atkins, ITS | June 2011
Identifying and responding to strengths and weaknesses in the evaluation process as a whole: as well as addressing specific issues, ex post evaluation can be used to generate a positive feedback loop to fine tune the ex ante evaluation exercise and make sure that it meets the needs of the intended users, that is the project promoters, governments and transport authorities in the Member States and the European Commission. The last heading is the broadest and introduces issues about the role of CBA in decision-making and ways of mitigating its known weaknesses. The inclusion of some environmental impacts and wider economic impacts in CBA, for example, are often challenging and there is a need for practical ways to take account of them. In line with the TORs, in Task 3 we assessed: the strengths and the weaknesses of the ex ante CBA methods used by funding applicants; the effectiveness of CBA as a tool supporting the decision-making process of Member States and the Commission; the utility of ex post CBA from the point of view of project promoters, Member States and the Commission; the extent to which ex post CBA analysis is an appropriate tool for evaluation of impact assessment; and the relevance and potential utility of data obtained from ex post CBA as an input for detailed transport models, to be used to underpin ex ante CBA analysis. Some of these methodological elements have already been assessed in previous studies and programmes.8 We have taken the results of these studies into account and build upon them to address the TORs requirements. Our work for Task 3 has drawn mainly upon the results of Task 2, the findings of which are described in Chapter 4 of this report. In addition, we integrated these results with: internal working group discussions; discussions with the independent experts who have supported us throughout the study; and
8 Including HEATCO (building on EURET), TENASSESS and TRANSTALK and IASON.
Approach and methodology
June 2011 | Frontier Economics, Atkins, ITS 43
the feedback received directly from Member State stakeholders at the seminar, jointly held with the consortium responsible for Work Package C.
Approach and methodology
June 2011 | Frontier Economics, Atkins, ITS 45
3 The role of cost benefit analysis: a brief review
This chapter describes the role of cost-benefit analysis (CBA) in the appraisal and evaluation of transport infrastructure projects taking account of the relevant literature about the strengths and weaknesses of applying CBA in EU Member States. Such description help set out the context to better understand our findings from the ex post evaluation of the selected 10 transport case study projects (chapter 3). This chapter is structured in two sections.
Section 3.1 describes the main roles of CBA in the context of European transport infrastructure appraisal; and,
Section 3.2 summarises the strengths and weaknesses of CBA as an ex ante project appraisal method.
3.1 The roles of CBA in transport appraisal Cost-benefit analysis is a tool for assessing the value for money of policies and projects from a society-wide perspective. The focus is on measuring and assessing the benefits and costs whoever they accrue to – travellers, operators, government, residents and so on. This distinguishes CBA from commercial (or financial) appraisal where the focus is on the revenue and cost impacts on the investing agency. There are many roles for CBA in transport infrastructure planning. Principally, CBA is used: to optimise the timing of projects – bringing forward or postponing their implementation. to choose between alternative options to carry out a project (e.g. different route options); to help select from a long list of projects to identify a short list and eliminate projects that are over-dimensioned or use excessively high- cost technology and are likely to yield negative economic returns (e.g. so called “prestige” or “white elephant” projects); to help prioritise projects to be funded from a fixed budget; to determine whether a project is financially viable – and the implications over the life of the project for public capital and/or operating subsidy support; and
The role of cost benefit analysis: a brief review
46 Frontier Economics, Atkins, ITS | June 2011
to determine whether a project makes efficient use of resources, yielding social benefits that exceed the social costs using a given social discount rate.
CBA applied to transport project appraisal There is a quite a large literature on CBA in transport.9 This is partly due to the prominent role that CBA plays in the resource allocation systems of many EU Member States (ECMT, 2005; Odgaard, Kelly and Laird, 2005) and in the European Commission (DG REGIO, 2008). The HEATCO project, for example, found that the role of CBA differs somewhat from country to country in the EU25.10 In most countries the CBA is used to: help choose between alternative project options; determine whether a project is efficient or viable; and, for investment prioritisation (Odgaard, Kelly and Laird, 2005: 18). In Germany, for example, social CBA is an integral part of the Federal Transport Infrastructure Plan (BVWP) process. This process determines which projects will form part of the 20-year plan for the country, including roads, railways and inland waterways. Only projects included in the BVWP can legally be taken forward (ECMT, 2005: 14). Another example is England where the selection of transport projects is increasingly driven by value for money criteria based on social CBA. The UK Department for Transport has also introduced thresholds based on the social benefit-to-cost ratio across all type of transport expenditure (DfT, 2009: 58) as shown in Table 9.
9 For an updated review of the use of value for money indicators in transport appraisal see Quinet (forthcoming 2011). 10 HEATCO was a programme for the development of tools, indicators and operational parameters for assessing sustainable transport and energy systems performance (economic, environmental and social). HEATCO primary objective was the development of harmonised guidelines for project assessment at EU level. In Chapter 3, we compare the parameters that were used for the ex ante analysis of the ten case studies with the HEATCO values.
The role of cost benefit analysis: a brief review
June 2011 | Frontier Economics, Atkins, ITS 47
Table 9. Explicit “Value for Money” thresholds
Benefit-to-Cost Value for Money Likelihood of Funding ratio (BRC) category thresholds
„No projects with poor Value for 1 „Poor‟ Money‟
„Very few projects with low Value for 1 to 1.5 „Low‟ Money‟
„Some, but by no means all, projects 1.5 to 2 „Medium‟ with medium Value for Money‟
„Most, if not all projects with high Value 2 to 4 „High‟ for Money‟
„Likelihood greater than for high Value Greater than 4 „Very High‟ for Money projects‟
Source: based on DfT (2009) and DfT (2006a)
Other EU Member States, however, do not use CBA in such a pervasive way. In Italy, for example, not all transport infrastructure projects are subject to CBA, and their prioritisation is often based on criteria other than the results of a CBA. Nevertheless, some CBA has been undertaken for a selection of „priority projects‟ (ECMT, 2005: 83-119). In some of these cases, low and even negative NPVs have been reported, however the implications of these low values for public decision making have not always been followed through. In the new EU Members States of Central and Eastern Europe, CBA is used mainly to support the business case for EU co-funding under the DG REGIO guidelines. However, CBA analysis has started to be incorporated into practice for nationally-funded projects (Odgaard, Kelly and Laird, 2005: 18). Table 10 provides an overview of CBA practice in the EU25 Member States, including the countries where the ten transport projects assessed in this study are located: Greece, Hungary, Ireland, Lithuania, Poland, Portugal, Slovak Republic, and Spain. The symbols (*) and (+) are important in interpreting the table: (*) indicates a recommended or required approach to CBA for transport projects; whilst (+) indicates that the use of CBA is ad hoc. The cell colouring in the table reflects the typical approach each in Member State for those projects where CBA is used.
The role of cost benefit analysis: a brief review
48 Frontier Economics, Atkins, ITS | June 2011
Table 10. Use and Scope of CBA in the EU25 + Switzerland
Construction Construction costs Disruption from construction Maintenance and operating costs Time savings charges User and revenues Vehicle costs operating Freight benefits Safety Noise Air pollution Climate Change Indirect socio- economic effects Austria 1 Belgium 1 1 1 1 Cyprus 1 1 1 1 1 Czech Republic 1 1 Denmark Estonia 1 1 1 1 1 Finland 1 1 France 1 Germany 1 1 Greece 1 1 1 Hungary 1 1 1 Ireland 1 1 1 1 Italy 1 1 1 Latvia 1 1 1 1 1 1 Lithuania 1 Malta 1 1 1 1 1 Netherlands Poland 1 1 1 1 Portugal 1 1 1 1 1 Slovak Republic 1 1 1 Slovenia 1 1 1 1 Spain 1 1 1 1 1 Sweden 1 Switzerland 1 UK 1 1 1
Colour codes: Included with a monetary value 1 Measured quantitatively, qualitatively or not included
Source: Mackie and Kelly (2007). Note: Luxembourg – no project appraisal method was found.
Table 10 clearly shows that all EU25 Member States use ex ante CBA to some extent. Some of the countries where the projects assessed in this study are located use CBA only on an ad hoc basis. This data was compiled during 2004 within the „HEATCO‟ FP6 research project for DG TREN, and represents the most up-to-date comparative information covering all the relevant countries. Some EU Member States have upgraded their CBA methods since 2004, for example with the inclusion of the monetary valuation of noise and CO2 in the UK (see Nellthorp et al, 2007 and DfT, 2009). Based on our experience, there has generally been a steady increase in the use and scope of CBA in recent years, especially in the use of monetary valuations for most impacts. In recent years, and particularly since the economic crisis of 2008, there has been much more attention devoted to the impact of transport projects on the wider economy, economic activity and land regeneration, mainly in urban contexts. The challenges of principle, data and method of appraising projects in conditions of imperfect markets with externalities, such as agglomeration effects, are
The role of cost benefit analysis: a brief review
June 2011 | Frontier Economics, Atkins, ITS 49
discussed in annex 2 (see also Mackie, Graham and Laird (2011). While it can be argued that a full scale shadow pricing11 approach along the lines proposed by Little and Mirrlees (1974) or the UNIDO guidelines (Dasgupta et al 1972) is required in theory, in practice such approaches are seen to be too data and resource intensive and short cut partial methods such as the use of a premium of say 30% as the shadow price of public funds are preferred. „HEATCO‟ also addressed the modal coverage of CBA. Table 11 shows the approaches used by EU25 Member States12 for the appraisal of projects across the various modes of transport. In general, road is more widely covered by CBA than any other mode (25 countries). Rail is almost as widely covered (21 countries), whilst CBA is less common for air (9 countries), inland waterway (10 countries) and sea (10 countries).
11 A shadow price is the maximum price that an economic agent is willing to pay for an extra unit of a given limited resource 12 HEATCO found no information on project appraisal methods was available for Luxembourg. While not a EU Member State, HEATCO included Switzerland, where, there are some important common appraisal methodologies.
The role of cost benefit analysis: a brief review
50 Frontier Economics, Atkins, ITS | June 2011
Table 11. Use of CBA by transport mode in EU25 + Switzerland
Inland Country Road Rail Air Sea waterway Austria 1,2,3 1,2 4 1 5 Belgium 1,3 1,2 4 4 4 Cyprus 1,2,4 5 5 5 5 Czech Republic 1,2 1,2 4 1,2,3,4 4 Denmark 1 1 4 4 4 Estonia 1 1 4 4 4 Finland 1,3 1,3 5 1,3 1,3 France 1 1 5 1 5 Germany 1,4 1,4 5 1,4 5 Greece 1,4 1,4 1,4 5 1,4 Hungary 1,2 4 4 4 4 Ireland 1,3,4 1,3,4 1,3,4 1,3,4 1,3,4 Italy 1 1 1 1 1 Latvia 1 4 4 4 4 Lithuania 1 1 1,3 4 1,3 Malta 1 5 5 5 5 Netherlands 1,3 1,3 1,3 1,3 1,3 Poland 1,2 1 4 4 4 Portugal 1 1 1,4 5 1 Slovak Republic 1,2 1 4 4 4 Slovenia 1 1 4 4 4 Spain 1,2 1,2 1,2 5 1 Sweden 1 1 1 1 1 Switzerland 1,2 1 4 4 4 UK 1,2,3,4 1,2,3,4 1,2,3,4 1,2,3,4 1,2,3,4
Codes: 1: Cost-benefit analysis (CBA) 4: Qualitative assessment / Not Covered 2: Multi-criteria analysis (MCA) 5: No information / Not relevant 3: Quantitative measurement (QM)
Source: Odgaard, Kelly and Laird (2005) . Note: Luxembourg – no project appraisal method was found.
The use of ex ante CBA across countries and transport modes can take the following roles:
Prioritisation of projects. In theory, social CBA can be used for prioritisation. Projects can be ranked either on the basis of their benefit-to- cost ratio (BCR) or their internal rate of return (IRR). Both indicators compare the benefits arising from the project with its costs. Both indicators are relevant when funds are constrained and there is a number of competing projects. When funds are unconstrained, the net present value (NPV) may be the relevant indicator to use.
The role of cost benefit analysis: a brief review
June 2011 | Frontier Economics, Atkins, ITS 51
PIARC (2004) examined the role of CBA in road project decision-making in the PIARC member countries.13 The study found that in many countries, CBA was not the only tool used for prioritisation. Instead, CBA was used alongside multi-criteria analysis (MCA) or a qualitative framework of some kind. For example, it found that “In Denmark, the separately assessed environmental impacts are taken into account when deciding whether to implement a road project and can mean that a project with lower economic return is preferred. […] In Germany projects are selected on the results of CBA plus non-monetary weightings and are then considered by the local and federal politicians. […]” (PIARC, 2004: 67). On the other hand, CfIT (2004)14 found that in Spain economic appraisal is required by law for major infrastructure projects, but that it is used only to determine the prioritisation of projects (“when” and “how”) not whether the project should be undertaken. The CBA practice from countries outside Europe is also relevant. For example PIARC found that “[in] Australia, projects are generally prioritised within a funding constraint and projects with a low BCR are usually not funded. Prioritisation is often not strictly objective, with criteria other then BCR used to subjectively prioritise competing projects. Project priorities are also influenced by higher level strategic outcomes. […] Until 2002, New Zealand was notable for its focus on CBA for the prioritisation of road projects: “Prior to 2002 there was an almost 100 percent correlation between the indicator of project worth (the BCR) and project implementation. This is partly due to the institutional arrangement for central government funding of road projects, which limits political input to high level policy and strategy. Since 2002 the process has been modified, with the BCR used as the initial indicator of project priority but with other factors considered alongside” (PIARC, 2004: 71). Table 11 suggests that in some of the countries considered by this study, pure CBA may be used as the basis for prioritisation. However, in the majority of countries, CBA is combined with another form of project prioritisation assessment.
Testing the efficient use of resources. In this role, a positive social net present value (NPV) indicates that the benefits that a project will generate are expected to justify its costs. From a socio-economic point of view, a project with an NPV greater than zero is said to be „acceptable‟ in cost- benefit terms. This approach is used in England, where the benefit-to-cost
13 PIARC countries include Australia, Canada, Czech Republic, Denmark, France, Germany, Hungary, Japan, Mexico, Netherlands, New Zealand, Norway, Portugal, Switzerland, Sweden, South Africa, UK and USA. 14 The Commission for Integrated Transport (CfIT) is an independent body advising the UK Government on integrated transport policy.
The role of cost benefit analysis: a brief review
52 Frontier Economics, Atkins, ITS | June 2011
ratio is required to be at least equal to one (DfT, 2009), and in Japan, where “CBA is used for acceptance or rejection” (PIARC, 2004: 67).
Testing the financial viability of a project. In this context, a project is said to be „acceptable‟ in financial terms if the financial NPV is greater than zero. The financial NPV is calculated using a financial discount rate instead of a social discount rate. The HEATCO guidelines (Bickel et al, 2006: 3) advocate reporting a financial test alongside the social CBA results. This is commonplace in national-level appraisal. It is also a requirement of the international financial institutions (IFIs), such as the World Bank or the European Investment Bank for funding transport projects (e.g. EC & EIB, 2005; TINA Secretariat, 1999).
Choosing between alternative project options. The choice between mutually exclusive project options – for example, alternative routes for a piece of surface infrastructure – can be informed by incremental analysis, that is by examining the changes in the benefit-to-cost ratio implied by each option. Once again, there is a range of practices among EU Member States and other countries. New Zealand takes a fairly pure CBA approach to decision-making, based on incremental BCR. Conversely, “[in] Norway, project alternatives are evaluated by a local impact assessment, of which BCR is part. However, the final choice of project option is more dependent on local preferences than on economic analysis” (PIARC, 2004: 67). In France, “[selection] of options for road projects [...] starts with economic evaluation but political and environmental acceptability have an increasing effect as the project progresses. […] The Czech Republic uses HDM-4 CBA outputs (NPV and IRR) plus MCA, for assessment of project alternatives” (PIARC, 2004: 65).
Timing and deferment of projects. Cost-benefit analysis can also play a useful role in helping choose the timing and the sequence with which projects are implemented. CBA can help give priority to projects with a more favourable benefit-cost ratio. Note TRN-6 of the World Bank‟s transport evaluation toolkit discusses Timing/Deferment of projects (The World Bank, 2005). Different indicators can be used. For example, PIARC (2004: 61) states that Norway and New Zealand both use First Year Rate of Return (FYRR) to optimise the timing of projects, generally choosing to give priority to those with a higher rate of return. Mackie and Kelly (2007: 11) indicate that France also uses FYRR for this purpose. On the other hand, the World Bank guidance and the English method both use NPV to optimise timing, giving priority to projects with a higher net present value.
CBA applied to transport policy Although we have so far focused on infrastructure projects – which are the main focus of this study – many countries apply CBA to area-wide assessments of
The role of cost benefit analysis: a brief review
June 2011 | Frontier Economics, Atkins, ITS 53
alternative transport strategies and policies. The English method is a good example: a series of area-wide studies were carried out between 1999 and 2006. These „Multi-Modal Studies‟ were also subjected to an ex post process evaluation by AEAT et al. (2004) and Bates et al. (2004). Finally, we affirm that CBA is a decision support tool and therefore is a significant contributor to a broader decision making process. While CBA evidence has been shown to correlate with public decisions on transport projects, the correlation is by no means perfect even in countries with a relatively strong tradition of undertaking CBA in the transport sector (see Nilsson, 1991; Nellthorp and Mackie (2000); Eliasson and Lundberg (2010)).
3.2 Strengths and weakness of ex ante CBA Ex ante CBA is a long established appraisal technique widely used in transport. It was introduced in the late 1950s and it has since developed both in practice and in methodology. As a result, the understanding of CBA is fairly consistent across Europe, as identified by HEATCO, and by earlier studies such as Giorgi et al / Pearman et al, or the EUNET and EURET studies in the 1990s. CBA is therefore a mature method, with a shared knowledge of techniques across national borders Whilst the set of impacts considered does vary from country to country, there is a significant overlap (see Table 3 above). The broad methodological framework including valuation and discounting is common. There are differences in parameter values – for example, HEATCO reports on the differences in discount rates and values of time. There are also different opinions over the most appropriate valuation techniques that should be used – for example, to assess the impact of noise, either stated preference (SP) or hedonic pricing (HP). In general, however, a CBA carried out in one part of the EU is in principle auditable by a cost-benefit analyst from anywhere in the EU. Despite the long history and common approaches across the EU, CBA remains an appraisal technique with both strengths and weaknesses. The principal strengths of CBA are as follows.
Its “potential” to rely on “objective” data and provide common indicators to compare alternative projects. Note the word „potential ‟ since the objectivity of ex ante CBA is challenged, most recently by the literature on optimism bias and „misinformation‟ identified as weaknesses below.
Its ability to address project risks and to present them in an understandable way providing decision-makers with a better understanding of the likelihood of different outcomes from the project.. This is a potential strength as it requires:
The role of cost benefit analysis: a brief review
54 Frontier Economics, Atkins, ITS | June 2011
analysts to extend the CBA using QRA tools. The funding bodies can incentivise this practice by requiring it in their appraisal guidance, as an increasing number do (this includes most IFIs). The current DG REGIO CBA guidelines require this type of approach within the Risk Assessment (DG REGIO, 2008); and, expert knowledge at the project level to identify correctly the probability distributions of input variables. This is something which might be easier to obtain ex ante, whilst the project knowledge is current, and more demanding ex post, after the project „team‟ has dissipated The principal weaknesses of CBA are the following ones.
Its errors and biases, such as traffic forecast “optimism bias”, the tendency of traffic model forecasts to overestimate systematically traffic demand in transport project appraisal.
Its manipulability, as the outcome of the analysis crucially depends on assumptions made and the inputs used, especially demand forecasts.
Its lack of comprehensiveness in relation to the full set of significant impacts, including environmental pollution, and economic performance impact such as GDP and employment.
The lack of an agreed and uniform methodology to define marginal values to assess key impacts such as time savings, vehicle operating cost savings, safety benefits and other positive externalities such as reductions in pollution.
Its weak assessment of distributional impact of projects (i.e. who benefits and who loses out) whether by social group or across geographical space. Annexe 3 contains a survey which describes in more detail the strengths and weaknesses of CBA and of multi-criteria analysis (MCA), an alternative appraisal method, with references to recent academic literature. The strengths and weaknesses of CBA as an appraisal method summarised above are revealed in different ways, and to a varying degree, in the ex post evaluation of the ten transport projects which are the subject of this report and are described in Chapters 3 and 4 next.
The role of cost benefit analysis: a brief review
June 2011 | Frontier Economics, Atkins, ITS 55
4 Evaluation of selected projects
In this chapter we summarise the results of our ex post evaluation of the ten transport projects. We first start with a review of the ex ante cost-benefit analysis that was carried out to support the funding application. We then provide an overview of the findings emerging from the ex post evaluation. Finally, we present a comparison of the key results of our analysis with the results of the original ex ante appraisal. The complete case studies, included as appendices to this report, provide the detailed results of our analysis of each project. Taken together, these findings have enabled us to provide an assessment of cost- benefit analysis as a method. We present the results of this assessment in the following chapters.
4.1 Overview of selected projects As discussed in Chapter 2, we identified the ten projects subject of this study. Table 12 lists the ten selected projects, grouped by transport mode and region. We believe that, for geographic and mode coverage, this selection is representative of the initial list of projects included in the TORs.
Table 12. Summary of candidate projects
Road Rail
M1 Northern Motorway Railway line Thriassio-Pedio- (Ireland) Eleusina-Korinthos-Kiato (Greece ) Greece/Ireland/ A23 Motorway (Spain) Portugal/Spain Algarve Railway (Portugal) Agios Konstantinos Bypass (Greece) AVE Madrid - Barcelona (Spain)
IX B Corridor (Lithuania) Bratislava Railway Upgrade (Slovakia) Central and A2 Motorway (Poland) Eastern Europe M0 Budapest Ring Road (Hungary)
Source: Frontier Economics, Atkins and ITS – Projects scores are out of 100.
The combined cost of these projects subject amount to EUR 4.9bn of capital investments, of which EUR 3.1bn (63.6%) came from Cohesion Fund contributions.
Evaluation of selected projects
56 Frontier Economics, Atkins, ITS | June 2011
Table 13 shows the opening dates of the various projects. It also indicates when the most recent ex ante CBA was undertaken for each project. We have also obtained information on project capital costs and Cohesion Funds contribution from each project‟s funding applications. We note that most projects were divided into various subprojects, each with its own different completion dates and, in some cases, different ex ante analysis.
Table 13. Overview of projects considered
Capital costs - EURm Opening Date of ex ante date(s) appraisal (% Cohesion Fund contribution)
AVE Madrid - 1,719 2003 - 2008 2001 Barcelona (Spain) (61%)
A2 Motorway 476 2006 2003 (Poland) (82%)
Algarve Railway 419 2003 - 2006 1999 (Portugal) (77%)
A23 Motorway 203 2001 - 2005 1999 and 2003 (Spain) (83%)
Agios Konstantinos 441 2008 2002 Bypass (Greece) (55%)
M1 Northern 232 2003 1995 Motorway (Ireland) (66%)
Railway Thriassio – 619 2005 - 2007 2000 Kiato (Greece) (47%)
IX B Corridor 154 2006 - 2009 1999 (Lithuania) (79%)
Bratislava Railway 234 2003 - 2009 2001 Upgrade (Slovakia) (39%)
M0 Budapest Ring 367 2008 - 2010 2004 Road (Hungary) (74%)
Source: Funding applications
Table 14 lists the main objectives of each project, as stated in the funding applications. Figure 2 provides a summary which shows that safety was the main stated objective in 8 of the 10 projects, followed by the objectives of reducing travel time (6 projects) and increasing capacity (4 projects).
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 57
Table 14. Project objectives
Project Stated project objectives
AVE Madrid - Reduction in travel times on the existing corridor. Barcelona (Spain) Increase capacity and safety conditions on the rail line. A2 Motorway Reduction in congestion on existing roads. (Poland) Improve safety conditions. Algarve Railway Reduction in travel times and increase productivity. (Portugal) Reduction in operating costs on the line. Increase safety conditions. A23 Motorway Increase capacity on existing corridor. (Spain) Improve communications in low density areas. Improve safety conditions. Agios Konstantinos Reduction in travel times. Bypass (Greece) Increase safety conditions. Improve connectivity between Greece and the EU. M1 Northern Reduction in travel times. Motorway (Ireland) Facilitate growth at Dublin airport. Railway Thriassio – Reduction in travel times. Kiato (Greece) Increase safety conditions on the rail line. IX B Corridor Increase pavement strength. (Lithuania) Divert traffic away from Vilnius city centre. Improve safety conditions on the existing corridor. Bratislava Railway Reduction in travel times. Upgrade (Slovakia) Increase safety conditions on the rail line. Meet interoperability requirements. M0 Budapest Ring Reduction in transport costs (travel times and congestion). Road (Hungary) Divert traffic away from Budapest city centre Source: Funding applications
Evaluation of selected projects
58 Frontier Economics, Atkins, ITS | June 2011
Figure 2. Summary of ex ante stated objectives.
Improve safety (8)
Reduce travel time (6)
Increase capacity/reduce congestion (4)
Reduce operating costs (2)
Improve connectivity (2)
Projects 2 4 6 8 10
Source: Funding applications
4.1.1 Key features of each project Before presenting the overall results of our analysis, we present a brief summary of each of the projects we have considered. For each project, we provide a high- level overview, describing the project‟s scope, the cost and level of EU funding, its objectives and its main impacts. Annexe 1 provides a one-page summary for each project, presenting the main findings of the evaluation. Finally, the individual case studies, attached to this report as Appendices, give additional detailed information on each project.
High speed railway Madrid – Barcelona in Spain The LAV (Línea de Alta Velocidad) Madrid – Barcelona – French border is a high speed railway line connecting Madrid to the French border via Barcelona. At the moment of writing, only the section Madrid – Barcelona is operational. We have therefore focused the ex post analysis on this section rather than on the whole project. The segment between Barcelona and the French border is still under construction and it is expected to be operational in 2013. The LAV is part of the TEN-T Priority Project 3 (high-speed railway axis of south-west Europe), whose main objective is to provide high-speed rail connections between the Iberian Peninsula (Portugal and Spain) and the rest of Europe.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 59
As defined in the TORs, the project comprises 12 subprojects. These cover the construction of 72 km of rail bed and the installation of 610 km of railway tracks. The total cost of the 12 subprojects was around €1,719 million, of which €1,442 million was eligible for funding. The total Cohesion Fund contribution for these subprojects in the period 2000-2006 was around €1,042 million, equal to 72.25% of the eligible project costs. The overall capital investments for the LAV between Madrid and Barcelona amount to €7,336 million, with total Cohesion Fund contributions around €3,389 million during the period 2000 – 2006. The main objective of the LAV Madrid - Barcelona – French border is to enable rail connections between the Iberian Peninsula and the rest of Europe without the need for reloading, which is needed as a result of the historic gauge difference between the rail networks in Spain/Portugal and the rest of Europe. The new railway line between Madrid and Barcelona implies that the two biggest cities in Spain are now linked by train in two hours and half hours, creating an alternative to the busiest air corridor in the world.
A2 motorway in Poland This project involved the construction of approximately 100km of new motorway between the towns of Konin and Strykow in central Poland. The route is of national and international importance and forms part of the European Route E30 between Berlin and Belarus. This section of the A2 is part of a wider package of projects to construct a high standard road transport link between Berlin and Warsaw, as well as improving connectivity at local and regional scale. The section between Strykow (at the eastern end of the project) and Warsaw is currently under construction. The total aggregate cost of the project was €406.3m. The total amount eligible for EU funding was €401.4m. The total Cohesion Fund contribution in the period 2004-2006 was €325.2m for the entire project. This contribution covered 82% of the cost for the Konin-Emilia subsection and 75% of the costs for the Emilia-Strykow subsection. The EIB also contributed to this project, up to the maximum allowed level (90%). Prior to the construction of the A2, the main connecting route between the major settlements in the area was the single lane R72 (linking Strykow and Lowicz) resulting in traffic congestion and poor journey time reliability and high risk of accidents. The main objectives of the A2 project were therefore to improve connectivity at local and regional level, thus reducing journey times, relieving traffic congestion, and improving safety.
Algarve railway in Portugal This project covers the modernisation of the railway line between Coina (near Lisbon) to Faro (in the Algarve region) in Portugal, including the branch to Porto de Sines. The total extension of both rail segments is around 275 km and 50 km
Evaluation of selected projects
60 Frontier Economics, Atkins, ITS | June 2011
respectively. The project is part of the TEN-T Priority Project 8 (Multimodal Axis Portugal/Spain – Rest of Europe). The total aggregate cost for the 4 subprojects included in our study was €419m, with a total amount eligible for funding equal to €405m. The total Cohesion Fund contribution during the period 2000-2006 was €323m equal to about 80% of the eligible project costs. The increase in the maximum speed on the line to 220km/h, thanks to additional double track sections, the installation of a new automatic control system and the complete electrification of the line, allowed the reduction of the journey between Lisbon and Faro to around 3h. The current ex post cost benefit analysis covers the segment between Pinhal Novo and Faro, including the branch to Porto de Sines. The 15 km rail segment between Coina and Pinhal Novo was not included the ex ante analysis covering this project (it was included in another analysis focusing on the modernisation of the Lisbon interurban rail system), and we have decided to exclude it in order to allow a more precise ex ante / ex post comparison.
A23 motorway in Spain This project covers the construction of 75 km of the A23 motorway in Spain, covering two separate segments, one of 63.5 km between Teruel and Calamocha and another of 11.5 km between Huesca and Nueno. The A23 (also called Autovía Mudéjar) is a high capacity road, long 440 km (370km in service), connecting Sagunto, on the Mediterranean coast north of Valencia, and the Somport road tunnel, which connects France and Spain through the central Pyrenees. The A23 partially belongs to the European route E07 connecting Pau (France) and Zaragoza (Spain). The total aggregate cost for the 4 subprojects included in our study was €203m with a total amount eligible for funding equal to €198m. The total Cohesion Fund contribution during the period 2000-2006 was €168m, equal to about 85% of the eligible project costs. The main objective of the A23 is to facilitate the freight traffic between the Levante (Eastern Spain) region and Aragón, in Spain, and France. The new road was designed to become the natural route through the central Pyrenees, adding capacity to the existing wide road connections between Spain and France in both extremes of the Pyrenees, Irun-Hendaya to the West and La Jonquera to the East.
Agiou Konstantinou bypass in Greece This project involved the construction of a bypass around the town of Agios Konstantinos and an upgrade of an existing Bypass around Kamena Vourla, both in Greece. This is part of the PATHE (Patras-Athens-Thessaloniki-Evzoni)
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 61
corridor. The route is of national importance because it links the two largest cities in Greece (Athens and Thessaloniki), internationally the route supports an important link between Greece, Central Europe, and the Balkans. The total cost of the project was €317.8m (2000 prices, including VAT). 37% of the funding was from the Cohesion Fund and the remainder was financed by national funding. The project was required because the road through Agios Konstantinos ran between the sea and the town; therefore little opportunity existed to undertake any online improvements to the existing road. The narrow cross section of the road did not allow safe overtaking opportunities and severe congestion events were commonplace particularly in the summer months. The main objectives of the project were to reduce journey times, increase capacity and improve safety
M1 northern motorway in Ireland This project covers the construction of two sections of the M1 motorway in Ireland. The first section runs from Cloghran to Lissenhall and the second section joins this road with the Balbriggan Bypass further north. The M1 Motorway is part of the TEN-T Priority Axis 13 (Ireland – United Kingdom – Benelux). The main objective of the investments in this corridor is the reduction in journey times between Ireland, the UK and mainland Europe. The total aggregate cost for this project was €232m. The Cohesion Fund contributions (€152m) were used initially in the planning and pre-construction phases. This covered initial design, detailed design, site surveys and investigation, the Environmental Impact Assessment, a public consultation process and the preparation of contract documentation. For phase 2 cohesion fund contributions were used to for construction, including earthworks, pavement, drainage and fencing and the construction of nine bridges. Interchanges, side roads and land acquisition were financed by the Irish Government. The eligible cost for co- financing was €173m. The construction of the new sections of the M1 were required to upgrade the heavily congested existing N1 road and was seen by the Irish Government as being key to meeting European, national, regional and local objectives. The main objectives of the construction of this motorway was to Improve transit times, safety levels and level of service and to provide essential infrastructure support for economic development (particularly industry and tourism) by improving access to Dublin‟s port, airport and the main domestic markets.
Railway Thriassio – Pedio – Eleusina – Korinthos in Greece The project covers a rail upgrade between Thriasio (north west of Athens) and Kiato in Greece. The new line has replaced a former single track metric line passing through a number of urban areas. The 112km section runs parallel to the
Evaluation of selected projects
62 Frontier Economics, Atkins, ITS | June 2011
European TEN-T network Motorway Priority Axis 7 (Igoumenista-Patras- Athens-Sofia-Budapest) and forms part of longer term proposals to upgrade the Piraeus-Athens-Patra line. The route is covered by four funding applications for cohesion funding. Three of the sub-projects are related to the section of track running from Thriasio to Korinthos. An additional funding application was made for the section of track between Korinthos and Kiato. The total cost of the project was €508m. Of this, the total amount eligible for EC funding was €238m, 47% of the total spend. The old line between Thriasio and Kiato had poor geometric features, an old infrastructure, different gauge from the rest of Greece, and a lack of modern telecommunications and signalling. The line also passed through a number of built up areas and contained a number of level crossings. As a result, although track speed was generally 90km/h it dropped to as low as 25km/h in places. The objectives of the improvement project were to reduce journey times, increase revenue and decrease operating costs, improve safety, improve connectivity with the wider network and provide employment during construction.
IX B corridor in Lithuania This project covers the modernisation of one of the two main motorway routes in Lithuania. This is part of the IX B transport corridor, 315km of motorway linking the port city of Klaipeda with the capital city of Vilnius, via Kaunas. The route then links Lithuania with other destinations in Eastern Europe. The project also involved the construction of the Vilnius Southern bypass. The modernisation of the IX B Corridor was required to prepare this motorway, a key element of the Lithuanian transport network, for higher and heavier volumes of commercial traffic expected in the coming years. The main objective for the construction of the Vilnius Southern bypass was to reduce chronic delays caused by transit and local traffic being forced through the historic district of the city. The total aggregate cost for this project was €154m, while the total amount eligible for funding was €148m, or about 96 per cent of the total. The total ISPA/Cohesion Fund contribution was €121m, equal to about 82% of the eligible project costs.
Bratislava Rača – Trnava Railway Upgrade in Slovakia This project covers a rail upgrade between Bratislava Rača and Trnava in Slovakia. The 39km section forms part of the Corridor Va TEN-T rail corridor which runs in Slovakia between Bratislava, Ţilina and Košice, then onto the border with Ukraine. The section of line also forms part of Priority Axis No. 23. The modernization project includes two separate applications for funding, the first including the upgrade of the line between Bratislava Rača to Šenkvice
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 63
(FM001), and the second upgrading the line between Šenkvice and Cífer and all the stations along the section between Bratislava Rača and Trnava (FM003). The total cost for the two projects combined was €174,324,287 (2002 prices). Of this, the total amount eligible for funding was €116,859,000, or about 67% of the total spent. The total ISPA/Cohesion Fund contribution was €81,421,439, equal to about 70% of the eligible project costs. The main aims of the project were to improve the speed, comfort and attractiveness of the railway, thus making it more competitive in the transport market – particularly as the roads into central Bratislava are becoming increasingly congested. The modernized rail infrastructure would also significantly reduce the operating and maintenance costs of the line, whilst meeting international standards (AGTC) of rail track.
M0 Budapest Ring Road (eastern section) in Hungary The M0 Eastern Sector is the eastern component of the Budapest orbital motorway, whilst the M31 is a linking motorway between it and the M3 (the motorway to the north east of Hungary). The project has important international, national and local dimensions. Budapest sits on the junction of three land based Helsinki corridors, is the focus of the national transport network and is the economic centre and capital of Hungary. Congestion in the city centre, particularly in the vicinity of the Danube is severe. This project therefore performs an important function in facilitating the movement of international and national traffic while also providing some traffic relief to Budapest. The total aggregate cost of the project was €370.5m of which €284.7m has been contributed by the EC. This contribution represents 85% of the eligible costs. There have been a number of technical challenges associated with the evaluation. The definition of the counterfactual has been extremely problematic. Aside from the generic problem of identifying the impact of the transport investment against a background of economic and social change brought about by accession to the EU and the 2007/8/9 recession, the impact of the M0 Eastern Sector is confounded with that of several major transport initiatives. Additionally, the M31 only opened to traffic at the end of July 2010 making it difficult to obtain any ex post data on this road. A final complication is that a lot of the benefits of the M0 and M31 relate to decongestion benefits in Budapest. These are hard to measure with surveys. The ex post impact of the M0 has therefore been estimated with the aid of the Budapest regional transport model.
4.2 Review of ex ante CBA In this section, we present our findings from the review of the ex ante cost- benefit analysis that was prepared to support the funding applications.
Evaluation of selected projects
64 Frontier Economics, Atkins, ITS | June 2011
4.2.1 Scope of CBA analysis Figure 3 shows the types of impacts that were considered in the ex ante CBAs we have reviewed. All ex ante analyses included an assessment of the core impacts on users (time savings and reduction in vehicle operating costs) as well as the safety benefits of the project. Outside of these core areas, however, the picture is more mixed. In only four cases, the analysis included a monetary quantification of the environmental impacts of the project while in only three cases the impact of the project on other transport operators were considered. Other impacts (such as congestion benefits) were considered only in two cases.
Figure 3. Impacts considered in the ex ante CBA.
Impacts on users (Time Savings, VOCs) (10)
Impacts on safety (10)
Impacts on the environment (4)
Impacts on transport operators (3)
Other impacts (2)
Projects 2 4 6 8 10
Source: Funding applications
4.2.2 Key CBA parameters
Discount rates and appraisal period The ex ante CBAs we have reviewed are generally consistent in the use of common discount rates for both the economic and the financial analyses. Discount rates used are normally around 5-6%, except in the rail project in Slovakia (10%) and the motorway in Poland (8%). We note that higher discount rates are associated with shorter appraisal periods. The length of the appraisal period used for the evaluation shows more variability, ranging between 20 years (A2 Motorway in Poland and Bratislava Railway Upgrade in Slovakia) and 36 years (Railway Thriassio – Kiato in Greece).
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 65
Generally rail projects include a residual value for the project at the end of the appraisal period while road projects do not. This is to be expected as railway infrastructures have a longer useful life than road infrastructures. We note that the length of the appraisal period and the choice to include a residual value in the analysis have an impact on the results of the analysis. The fact that they are not harmonised across projects make their comparison more challenging. Table 15 summarises this information across all projects.
Table 15. Overview of discount rates and appraisal periods
Discout rate Discout rate Appraisal Residual (economic) (financial) period value included?
AVE Madrid - Barcelona (Spain) 6% 6% 29 years Yes
A2 Motorway (Poland) 8% N/A 20 years No
Algarve Railway (Portugal) 5% 5% 28 years Yes
6% (1999) A23 Motorway (Spain) N/A 31 years No 5% ( 2003)
Agios Konstantinos Bypass (Greece) 5% 7% 30 years Yes
M1 Northern Motorway (Ireland) 5% N/A 30 years No
Railway Thriassio – Kiato (Greece) 5% 5% 36 years Yes
IX B Corridor (Lithuania) 5% N/A 25 years No
Bratislava Railway Upgrade (Slovakia) 10% 10% 20 years Yes
M0 Budapest Ring Road (Hungary) 6% N/A 30 years No
Source: Ex ante cost-benefit analyses
Value of time As noted above, travel time savings are one of the main sources of expected economic benefits. The value of time parameter – in EUR per hour - is a key input in the calculation of these benefits. Table 16 summarises, for all projects,
Evaluation of selected projects
66 Frontier Economics, Atkins, ITS | June 2011
the values of the parameters used for this calculation. As a reference, we include the value of time parameters for working and non-working time from the HEATCO study. The value of time used may differ depending on whether it applies to passenger transport or freight transport. In addition, some ex ante CBAs used two parameters for the value of time in passenger transport, to differentiate between working and non-working time (with value of working time higher than non- working time). Overall, the range of the parameters used is wide. This is to be expected, as the value of time depends on variables such as per capita GDP. We note however that there does not to appear to be any relationship between the values used in the analysis and the HEATCO values. In some case, this is due to the fact that the ex ante analysis had been carried out before the HEATCO values became available. However, the source of those values remains unclear.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 67
Table 16. Overview of value of time assumptions used in the ex ante analyses and comparison with HEATCO values
Working time Non-working time Freight time
Case Case HEATCO HEATCO studies studies
AVE Madrid - 9.9 22.3 9.9 9.2 N/A Barcelona (Spain)
A2 Motorway 3.6 12.9 3.6 5.3 6 (Poland)
Algarve Railway 3.8 19.3 3.8 7.2 N/A (Portugal)
A23 Motorway 9.9 22.3 9.9 9.2 17.1 (Spain)
Agios Konstantinos 7.4 19.3 7.4 7.5 25.9 Bypass (Greece)
M1 Northern 11.5 29.9 2.9 9 11.5 Motorway (Ireland)
Railway Thriassio – 3.2 19.4 3.2 7.5 N/A Kiato (Greece)
IX B Corridor 3.3 – 4.5 11.6 3.3 – 4.5 4.8 3.3 – 4.5 (Lithuania)
Bratislava Railway 4.7 12.4 1.9 5 1.9 Upgrade (Slovakia)
M0 Budapest Ring 8.3 13.5 4.1 5.4 12.4 Road (Hungary)
Source: Ex ante CBAs and HEATCO Note: 1. For all projects except for Bratislava Railway Upgrade (Slovakia) and Railway Thriassio – Kiato (Greece) figures reported are in factor prices as oppose to market prices. 2. Ireland – the 1994 values in £IRL have been converted to 2002 Euros (but at 1995 values) for this comparison
Vehicle operating costs (VOC) Savings in vehicle operating costs (VOC) are another important source of benefits in the ex ante analyses reviewed. A new road generally reduces vehicle maintenance costs due to improved pavement conditions. We note that in some cases, however, it also allows higher speeds, inducing faster vehicle deterioration. A new or improved railway line may induce significant modal shift and generates
Evaluation of selected projects
68 Frontier Economics, Atkins, ITS | June 2011
VOC savings as rail transport has generally lower variable costs than road or air transport. The following Table 17 shows the VOC parameters for road transport. In two rail projects, Thriassio – Kiato in Greece and the Bratislava upgrade in Slovakia, VOC parameters for passenger and freight rail transport are also available.
Table 17. Unit costs of vehicle operating costs
Car / LGV HGV
AVE Madrid - n/a N/A Barcelona (Spain)
A2 Motorway 0.16 / vehicle / Km 0.6 / vehicle/Km (Poland)
Algarve Railway 0.11 / passenger / Km 0.027 / tonne/Km (Portugal)
A23 Motorway 0.08 - 0.09 /vehicle / Km 0.22 - 0.27 / vehicle / Km (Spain)
Agios Konstantinos (1) (1) Bypass (Greece)
M1 Northern (2) (2) Motorway (Ireland)
Railway Thriassio – 0.013 / passenger / Km 0.010 / tonne / Km Kiato (Greece)
IX B Corridor (3) (3) (Lithuania)
Bratislava Railway 0.12 / pass / Km 0.08 / tonne / Km Upgrade (Slovakia)
M0 Budapest Ring 0.086 - 0.094 /vehicle / Km 0.38 - 0.39 / vehicle / Km Road (Hungary)
Source: Ex ante CBAs (1) Value differs by speed and vehicle type. (2) Fuel and non fuel costs vehicle/Km. Value differs by speed. (3) Vehicle operating costs (VOC) calculated using the HDM sub-model. VOC calibrated for Lithuanian conditions and DA VASEMA. (*) Base year for monetary values not specified. Note: For all projects except for Bratislava Railway Upgrade (Slovakia) and Railway Thriassio – Kiato (Greece) figures reported are in factor prices as oppose to market prices. The VOC parameters used in the ten ex ante analyses are difficult to compare across projects. This is because VOC parameters are sometimes expressed in Euro/vehicle/Km and sometimes Euro/passenger (or tonne)/Km. In general,
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 69
VOC parameters are only reported for road transport, distinguishing between passenger and freight. In contrast to all other projects, with fixed VOC parameters, in Lithuania VOC parameters vary with pavement conditions.
Safety Improvements in safety conditions are another major source of benefits. Accident unit costs used in the ex ante CBA are reported except for the two Greek projects. Table 18 shows the parameter values used to assess the cost of accidents, separating rail projects and road projects. As a reference, for road projects we include the parameter values in case of a fatality from the HEATCO study. A comparison with the HEATCO values is not possible for rail projects as HEATCO does not report average accident costs per passenger kilometre.
Evaluation of selected projects
70 Frontier Economics, Atkins, ITS | June 2011
Table 18. Unit cost of accidents – comparison with HEATCO values, when available
Fatality Injury
Rail projects
AVE Madrid - Barcelona 43 / 1000passenger/Km 43 / 1000passenger/Km (Spain)
Algarve Railway (Portugal) 35 / 1000passenger/Km 35 / 1000passenger/Km
Railway Thriassio – Kiato Not reported Not reported (Greece)
Bratislava Railway 22 / 1000passenger/Km 22 / 1000passenger/Km Upgrade (Slovakia)
Road projects
Case HEATCO Case studies studies
Agios Konstantinos Not 836,000 Not reported Bypass (Greece) reported
A2 Motorway (Poland) 120,000 341,000 25,000
A23 Motorway (Spain) 165,774 1,122,000 21,882
M1 Northern Motorway 1,345,298 2,134,000 4,641 - 48,094 (Ireland)
IX B Corridor (Lithuania) 301,539 275,000 24,383
M0 Budapest Ring Road 413,502 440,000 3,797 - 28,692 (Hungary)
Source: Ex ante CBAs and HEATCO
Note: 1. For all projects except for Bratislava Railway Upgrade (Slovakia) and Railway Thriassio – Kiato (Greece) figures reported are in factor prices as oppose to market prices.
2. Ireland – the 1994 values in £IRL have been converted to 2002 Euros (but at 1995 values) for this comparison
In five cases, the ex ante analysis reports the absolute cost of an accident, distinguishing between the cost of a fatality and the cost of an injury. These costs vary significantly between countries. For example, the cost of a fatality in Ireland is more than ten times higher than the value used in Spain, and more than four times higher than in Lithuania. Injury costs seem to be slightly more aligned across countries.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 71
When comparison with the HEATCO parameters is possible, we observe that the values used in all the ex ante analyses are lower than the HEATCO ones.
4.2.3 Risk analysis The ex ante CBA analysis included various sensitivity tests as part of the risk analysis undertaken in all ten projects. The number of sensitivity tests for every project ranges from two to four, with the exception of the M1 Northern Motorway in Ireland which included 27 sensitivity tests. The capital cost of the project is the risk factor most frequently considered in the risk sensitivity analysis (eight out of ten projects include this factor). Also, different values of the key parameters used in the ex ante analysis (see section 3.3.2) are usually tested in the risk analysis (five projects). Table 19 indicates the number of sensitivity tests undertaken in each project the risk factors considered in those tests. It is noteworthy that travel demand is considered as a risk factor in only 2 of the 10 projects.
Evaluation of selected projects
72 Frontier Economics, Atkins, ITS | June 2011
Table 19. Risk analysis. Sensitivity tests run and risk factors considered
Sensitivity Risk Factors considered tests under- Travel Fuel Capital CBA para- taken demand prices costs meters
AVE Madrid - Barcelona 2 (Spain)
A2 Motorway 4 (Poland)
Algarve Railway 2 (Portugal)
A23 Motorway 2 (Spain)
Agios Konstantinos 4 Bypass (Greece)
M1 Northern Motorway 27 (Ireland)
Railway Thriassio – Kiato 2 (Greece)
IX B Corridor 3 (Lithuania)
Bratislava Railway Upgrade 3 (Slovakia)
M0 Budapest Ring Road 4 (Hungary)
Source: Ex ante CBAs
We note that the sensitivity analysis carried out in the projects we reviewed is relatively simplistic and should not be considered as a substitute for a more comprehensive and detailed risk analysis.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 73
4.2.4 The role of ex ante CBA in the decision-making process As summarised in Figure 4, we have found that the role of the ex ante CBA in the decision-making process mainly relates to complying with EC requirements. We have also found that for 6 projects, ex ante CBA is used to ensure value for money when making choices about project implementation. However, in most cases ex ante CBA is not used for strategic decision making. In only a few cases, project stakeholders stated that they had used ex ante CBA to help choose between design standards, alignments, and to prioritise elements of the national transport strategy. However, in no case ex ante CBA was used to allocate limited budgets between projects or to optimise the timing of project implementation.
Figure 4. Role played by the ex ante CBA
Compliance with EC CF application requirements (9)
Ensure value for money (6)
Choose alignment (2)
Choose design standards (2)
Prioritise elements of national transport strategy (1)
Allocate budget and optimise timing (0)
Projects 2 4 6 8 10
Source: Discussion with project stakeholders
4.3 Results of ex post evaluation This section presents a summary and comparative tables of the main results of the ex post analysis for the ten projects examined.
Overall assessment Table 20 presents the economic appraisal indicators (Present Value of Benefits, Present Value of Costs, Net Present Value, Benefit Cost Ratio, and Internal Rate of Return) for the ten transport projects. In carrying out the ex post evaluation, we have calculated a High case (optimistic) outcome and a Low case (pessimist) outcome. For each indicator, we present its values for both High and Low case.
Evaluation of selected projects
74 Frontier Economics, Atkins, ITS | June 2011
As Table 12 shows, most projects have yielded a positive net present value, indicating that the economic benefits of the projects have exceeded their costs. In the AVE Madrid – Barcelona project the NPV is negative for both the High case and the Low case due to the high upfront capital costs of the project. We note, however, that the cost-benefit analysis of this project does not account for the wider socio-economic impacts of this project. These include the impact on mobility and accessibility, socio-economic structures, urban image and spatial effects. These impacts are expected to be significant, although it is still too early to appreciate their extent. Moreover, they are difficult to quantify in monetary terms, and therefore add to the cost-benefit analysis in a consistent way, as there is no agreed methodology for doing so. The range of benefit-cost ratios of the ten project goes from 0.6-0.7 (Low-High) of the AVE Madrid-Barcelona high speed train project in Spain to 17.2-17.6 (Low-High) corresponding to the M1 Northern Motorway project in Ireland. The average benefit-cost ratios for the ten projects are 3.6 (Low) and 4.0 (High) while the average IRR are 19.1 (Low) and 20.7 (High). If we exclude the M1 Northern Motorway project in Ireland and IX B Transport corridor in Lithuania (both projects have a very high IRR), the average IRR are 10.4% (Low) and 12.3% (High). These findings confirm that overall the projects we have analysed have delivered value for money. With regards to the AVE Madrid-Barcelona high speed line, the project‟s high capital cost explains the unfavourable cost-benefit ratio. However, we found that far-reaching wider socio-economic positive impacts, which will likely become more apparent in the coming years. However, these impacts are not captured by the standard cost-benefit analysis framework, on which the results presented in the table above are based.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 75
Table 20. Ex post evaluation – economic appraisal indicators
PVB (€m) PVC (€m) NPV (€m) BCR IRR High High High High High Low Low Low Low Low
Rail projects
AVE Madrid - 5,744 7,692 -1,948 0.7 3.7% Barcelona (Spain) 4,856 7,593 -2,736 0.6 2.6%
Algarve Railway 410 331 79 1.2 7.4% (Portugal) 379 331 48 1.1 6.7%
Railway Thriassio – 583 326 258 1.8 9.3% Kiato (Greece) 358 326 32 1.1 6.1%
Bratislava Railway 443 231 98 2.0 10.4% Upgrade (Slovakia) 291 231 40 1.4 7.8%
Road projects
A2 Motorway 1,168 268 900 4.4 22.8% (Poland) 791 268 523 3.0 18.2%
A23 Motorway 253 225 28 1.1 6.3% (Spain) 198 225 -28 0.9 4.6%
Agios Konstantinos 488 206 283 2.4 13.4% Bypass (Greece) 438 206 233 2.1 12.6%
M1 Northern 4,140 235 3,905 17.6 53.0% Motorway (Ireland) 4,040 235 3,805 17.2 53.0%
IX B Corridor 300 88 212 3.4 56.0% (Lithuania) 288 88 200 3.3 55.0%
M0 Budapest Ring 1,187 213 974 5.6 24.8% Road (Hungary)
Note: For all projects except for Bratislava Railway Upgrade (Slovakia) and Railway Thriassio – Kiato (Greece) figures reported are in factor prices as oppose to market prices.
Table 13 presents the financial appraisal indicators obtained from the ex post financial analysis of the ten selected projects. For each project, table 13 presents the Net Present Value and the Internal Rate of Return of Investment, and the Net Present Value and the Internal Rate of Return of Capital, showing the results for both the High and Low case scenarios.
Evaluation of selected projects
76 Frontier Economics, Atkins, ITS | June 2011
Table 21. Ex post evaluation - Financial appraisal indicators
FNPV (I) FIRR (I) FNPV (C) FIRR (C) (€m) (€m) High High High Low High Low Low Low
AVE Madrid - -4,288 0.6% -351 4.5% Barcelona (Spain) -4,766 0.4% -919 3.7%
A2 Motorway -263 -3.5% 47 11.6% (Poland) -263 -3.5% 47 11.6%
Algarve Railway -299 -3.6% -111 0.2% (Portugal) -299 -3.6% -111 0.2%
A23 Motorway -230 -55 n/a n/a (Spain) -230 -55
Agios Konstantinos -166 - 1.3% -78 1.0% Bypass (Greece) -173 -1.7% -85 0.5%
M1 Northern -235 -81 n/a n/a Motorway (Ireland) -235 -81
Railway Thriassio – -101 3.4% 98 7.3% Kiato (Greece) -184 1.8% 10 5.3%
IX B Corridor -95 -28 n/a n/a (Lithuania) -95 -28
Bratislava Railway -11 4.4% 81 15.1 Upgrade (Slovakia) -28 3.3% 65 13.6
M0 Budapest Ring 220 n/a -49 n/a Road (Hungary)
Source: Funding applications and supporting material
In 7 projects, the financial NPV of these projects is negative. However, this is not surprising since most of these projects do not generate direct commercial revenues and the financial analysis does not take the Cohesion Fund contributions into account. Without taking those EU contributions into account, some projects have a negative financial NPV but a positive economic NPV. The Cohesion Fund contributions are therefore needed to „unlock‟ the economic benefits of these projects. Table 22 presents the main indicators from the financial appraisal of the selected projects. It provides details on the total outturn capital cost and its distribution
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 77
among EU funding and national funding. With the exception of Bratislava Railway Upgrade in Slovakia, Railway Thriassio – Kiato in Greece and the AVE Madrid - Barcelona in Spain, EU funding represents more than 50% of the total cost of the projects.
Table 22. Overview of project funding.
Outturn capital EU funding National funding costs (€m) (€m) (€m)
AVE Madrid - 2,712 5,357 8,069 Barcelona (Spain) (34%) (66%)
A2 Motorway 329 72 401 (Poland) (82%) (18%)
Algarve Railway 218 167 385 (Portugal) (57%) (43%)
A23 Motorway 175 35 210 (Spain) (83%) (17%)
Agios Konstantinos 175 143 318 Bypass (Greece) (55%) (45%)
M1 Northern 152 90 232 Motorway (Ireland) (66%) (34%)
Railway Thriassio – 218 245 463 Kiato (Greece) (47%) (53%)
IX B Corridor 67 17 84 (Lithuania) (80%) (20%)
Bratislava Railway 89 124 213 Upgrade (Slovakia) (42%) (58%)
M0 Budapest Ring 271 96 367 Road (Hungary) (74%) (26%)
Source: Project final reports
Sources of benefits Based on the ex post evaluation, we have observed that benefits from these projects come from eight categories, namely travel time saving, vehicle operating costs saving, farebox revenues, safety improvements, carbon emissions, air pollution and noise reduction and other benefits. This last category includes a
Evaluation of selected projects
78 Frontier Economics, Atkins, ITS | June 2011
variety of smaller impacts, such as the reduction in traffic congestion, for example. Figure 5 and Figure 6 present the benefit distribution for railway and road projects separately. Overall, it can be observed that most of the benefits for railway projects arise from additional revenues from fares and travel time saving. On the other hand, for road projects, most of the benefits come from travel time saving and vehicle operating cost savings. This confirms the importance for rigorous demand modelling, especially when appraising road projects.
Figure 5. Sources of benefits – railway projects
Railway projects
120%
100%
80%
60%
40%
20%
0%
-20% Railway - Spain Railway - Portugal Railway - Greece Railway - Slovakia
Travel time VOCs Transport operator fares Safety Carbon Air pollution Noise Other
* For the Bratislava Railway Upgrade (Slovakia), the analysis has identified operating cost savings for the transport operator. However, these should be considered in the analysis as cost reductions rather than benefits and therefore are not included in this chart. However, this does not have any impact on the NPV calculation.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 79
Figure 6. Sources of benefits – road projects
Road projects
140%
120%
100%
80%
60%
40%
20%
0%
-20%
-40% Road - Poland Road - Spain Road - Greece Road - Ireland Road - Lithuania Road - Hungary
Travel time VOCs Transport operator fares Safety Carbon Air pollution Noise Other
Wider socio-economic impacts All transport projects, especially major ones, tend to have significant impacts on the local, regional and national socio-economic environments. These impacts, however, cannot be quantified easily. For this reason, they are often excluded from the core cost-benefit analysis. However, they can be important and should in principle be considered in the overall assessment of a project, albeit qualitatively. Our analysis has considered these impacts from a qualitative point of view, based mainly on gathering evidence through interviews with project stakeholders at Member State level. Identifying the wider socio-economic impacts of these projects in the absence of a monitoring framework implemented from project opening onwards is particularly challenging for two main reasons: the “counterfactual” (i.e. what would have happened in the absence of the project) has not been defined, thus making it difficult to identify the incremental impacts of the project over and above its counterfactual; and, many of these projects were implemented as part of wider transport investment strategies and against a backdrop of rapid economic growth;
Evaluation of selected projects
80 Frontier Economics, Atkins, ITS | June 2011
in this situation, the attribution of wider impacts to a specific project is particularly challenging. Despite these difficulties, we have identified the following types of wider impacts: land use, supply chain, GDP or output, employment, social inclusion, environment, and others. Figure 6 summarises the presence, absence or uncertain influence that each project has had on each of the categories of wider socio- economic impact mentioned above. It is difficult for some projects to establish a direct causal link between the transport infrastructure investments and the observed effects. This is especially relevant for the GDP of the region/country impact.
Figure 7. Summary of wider impacts from the ten transport projects.
Supply GDP / Social Land Use Employment Population Environment Other Chain output Inclusion Railway - Spain ✓ ✓ ? ✓ ? ✓ ✓
Road - Poland ✓ ✓ ✓
Railway - Portugal ? ✓
Road - Spain ✓ ?
Road - Greece ✓ ✓ ✓ ✓
Road - Ireland ✓ ✓
Railway - Greece ✓ ✓
Road - Lithuania ✓
Railway - Slovakia ? ✓ ✓
Road - Hungary ✓ ✓ ? ?
Note: ✓ (Observed effect) ? (Difficult to establish a direct causality)
Unit cost analysis The ex post evaluation required a substantial analysis of the costs incurred to build the infrastructures. Unit costs have been calculated, and overall, it can be said that most of the projects presented disaggregated figures up to Level 215, with the exception of Ireland and Lithuania. On the other hand, only Hungary presented unit costs up to Level 3. The availability of unit cost information is summarised in Figure 8.
15 As defined in the unit cost methodology developed by Work Package 10 of the ex post evaluation of the ERDF in the 2000-2006 programming period.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 81
Figure 8. Level unit costs availability
Railway - Road - Railway - Road - Road - Road - Railway - Road - Railway - Road - Spain Poland Portugal Spain Greece Ireland Greece Lithuania Slovakia Hungary
Level 1 Level 2 Level 3
Table 23 compares the unit costs we have collected for each projects with the average unit costs identified in Work Package 10 of the ex post evaluation of the ERDF in the 2000-2006 programming period.
Table 23. Unit cost comparison for rail projects
Level 1 Level 2
‘All-in’ Track Stations Bridges Tunnels
(EURm/km) (EURm/km) (EURm/nr) (EURm/nr) (EURm/km)
Railway – 14.2 7.4 79.9 24.3 24.2 Spain
Railway – 1.7 0.9 4.0 3.0 n/a Portugal
Railway – 2.5 4.4 4.3 n/a 14.3 Greece
Railway – 6.3 [data not available in a format compatible with WP10] Slovakia
WP10 64.3 10.3 9.3 1.0 32.5 average
Evaluation of selected projects
82 Frontier Economics, Atkins, ITS | June 2011
The table show that the unit costs for these projects vary significantly. They also tend to differ from the average identified in Work Package 10. However, this is to be expected as each of these projects has specific characteristics that influence unit costs. Table 24 provides unit cost information for the road projects we have considered in this study. As for rail projects, there appears to be a significant variation in unit costs, due to the different project characteristics.
Table 24. Unit cost comparison for road projects
Level 1 Level 2
‘All-in’ Pavement Bridges Tunnels
(EURm/km) (EURm/km) (EURm/nr) (EURm/km)
Road – [data not available in a format compatible with 406.3 Poland WP10]
Road – 2.7 0.6 6.6 n/a Spain
Road – 15.6 2.7 n/a 19.3 Greece
Road – [data not available in a format compatible with 14.2 Ireland WP10]
Road – [data not available in a format compatible with 0.8 Lithuania WP10]
Road – 9.4 3.1 0.9 n/a Hungary
WP10 22.2 17.3 7.8 28.2 average
Overall, we found the collection of unit cost data problematic. This was due to the lack of uniformity in cost classification across Member States. In most cases, cost we available with some degree of granularity, but the categories available were not compatible with those defined in WP10, making the calculation of unit costs according to this format not possible. Unit cost databases can provide significant useful information. However, their effectiveness relies on the data being collected on the basis of a common methodology. Based on our experience with collected unit cost data for these projects, we believe that the collection of historical unit costs will generally tend to be challenging. However, if specific guidelines are put in place, over time it should be possible to compile a comprehensive unit cost library that could be
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 83
used as a guide for both the ex ante appraisal and the ex post evaluation of major transport projects.
Utilisation rates Figure 9 summarises the level of utilisation rates we have calculated for the ten transport projects. The utilisation rate is defined as the ratio between the actual current usage of the infrastructure (estimated in terms of number of passengers or vehicles) and its total capacity. There are significant differences between projects. On the one had, some projects have very high utilisation rates. The extreme case is represented by the M1 Northern Motorway in Ireland with around 100% of utilisation in the peak period. On the other extreme, the A23 motorway in Spain exhibits a utilisation rate of around 5%. Both Greek projects 23%, and the Bratislava Railway Upgrade in Slovakia around 29%. During the study we were also told that in some cases the recent recession has contributed to reducing utilisation rates, thereby freeing capacity and deferring the need for further investments.
Figure 9. Comparison of utilisation rates
100%
80%
60%
40%
20%
0% Railway - Road - Railway - Road - Road - Road - Railway - Road - Railway - Road - Spain Poland Portugal Spain Greece Ireland Greece Lithuania Slovakia Hungary
We note that, under normal circumstances, new transport infrastructures with long expected service lives (between 30 and 60 years) tend to have relatively low utilisation rates at the beginning. This is because they have been designed to accommodate increases in demand over a long period of time. As the projects we have reviewed were opened only recently, low utilisation rates should not automatically be interpreted as an indication of over-sizing.
Evaluation of selected projects
84 Frontier Economics, Atkins, ITS | June 2011
4.4 Comparison of ex ante and ex post analyses In our analysis we have found discrepancies between the results of the original ex ante analysis and the results of the ex post evaluation. In this section we first compare, using the various economic indicators, the results of the ex ante analysis and those of the ex post evaluation. We then identify the main possible causes for these discrepancies. Below, we present a graphical comparison of the main economic appraisal indicators (Net Present Value, Benefit Cost Ratio, Economic Rate of Return, and Capital costs) between the ex ante and the ex post evaluation. In the charts below, dotted lines mark the 45 degree line. Along the 45 degree line, the values on the Y-axis are equal to those on the X-axis. This should facilitate the comparison between ex ante and ex post figures.
Net present value Overall, the NPV of the majority of the projects was overestimated in the ex ante compared with the ex post evaluation. With the exception of the M1 Northern Motorway (Ireland), IX B Corridor (Lithuania) and Bratislava Railway Upgrade (Slovakia), the ex ante NPV of all of the other projects exceeded ex ante NPV. Figure 10 shows the difference between the NPV in the ex post evaluation and that of the ex ante analysis in absolute terms. The biggest negative difference are reported for the AVE Madrid – Barcelona (-€3,433 m; not shown on the graph) and for the A2 Motorway in Poland (-€1,155 m). On the other hand, the biggest positive difference corresponds to M1 Northern Motorway in Ireland (€3,321 m; not shown on the graph) and the IX B Corridor in Lithuania (€123 m)
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 85
Figure 10. Comparison of ex ante and ex post NPV
Note: Ex post figures correspond to the average for the high and low case scenarios
Figure 11 shows the same comparison as Figure 9, but without the inclusion of the more extreme observations. In most cases, the ex ante NPV tends to be higher than the ex post NPV, confirming the pattern that we have identified in most of the projects.
Evaluation of selected projects
86 Frontier Economics, Atkins, ITS | June 2011
Figure 11. NPV comparison (without more extreme observations).
Note: Ex post figures correspond to the average for the high and low case scenarios
Benefit-cost ratio Figure 12 compares ex ante and ex post benefit-cost ratios (BCR). A BCR larger than one indicates that the benefits of the project exceed its costs. Given that the M1 Northern Motorway in Ireland is an outlier, we have excluded this observation from the chart. As shown in Figure 13, generally ex ante and ex post BCRs differ. However, the evidence here is more mixed: for half of the projects, the ex post BCR is higher than the ex ante BCR. There is not a clear pattern between road and railway projects that explains this finding.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 87
Figure 12. BCR Comparison
Note: Ex post figures correspond to the average for the high and low case scenarios.
Figure 13. BCR Comparison (without outlier: Road IE)
Note: Ex post figures correspond to the average for the high and low case scenarios.
Economic Internal Rate of Return A comparison of the economic IRRs suggest that ex post figures are generally in line with ex ante values, with the exceptions of the M1 Northern Motorway project in Ireland and the IX B Corridor project in Lithuania. Apart from these two projects, the average difference between ex post and ex ante IRR is -0.6%,
Evaluation of selected projects
88 Frontier Economics, Atkins, ITS | June 2011
albeit with a wide range of variation. The extreme cases correspond to the Hungarian road project (8.7%) and the Greek road project (-4.6%). There is not a clear pattern between road and railway projects that explain this finding..
Figure 14. IRR Comparison
Note: Ex post figures correspond to the average for the high and low case scenarios.
Figure 15 shows the same comparison as that of Figure 13 but excluding extreme observations.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 89
Figure 15. IRR Comparison (without more extreme observations)
Note: Ex post figures correspond to the average for the high and low case scenarios.
Capital costs Finally, Figure 16 compares expected capital costs in the ex ante analysis and the capital costs realised ex post. With the exception of the Hungarian M0 Budapest Ring Road project, the ex post capital costs exceed ex ante figures in all projects. This indicates that most of the ex ante analysis appeared to be influenced by some degree of optimism bias. Overall, the average cost overrun for the ten projects is 13.5%. Figure 16 shows that the M1 Northern Motorway in Ireland is the project with biggest cost overrun in percentage terms (58%). In the other extreme, the M0 Budapest Ring Road in Hungary presents a cost saving of 25%.
Evaluation of selected projects
90 Frontier Economics, Atkins, ITS | June 2011
Figure 16. Capital Costs comparison.
Note: AVE Madrid Barcelona has been excluded from the graph as it is an outlier.
Factors explaining the difference between ex ante and ex post results Our analysis suggests that the differences between ex ante and ex post results can be associated with the following causes: differences between actual and forecast investment costs; difference between actual and forecast economic growth; errors in population growth forecasts; errors in travel demand forecasts; marginal values used for the analysis (for example value of time) and discount rates; and, delays. Figure 17 summarises how the above factors are responsible for the differences between ex ante and ex post results across the ten transport projects. The discount rate is a primary or secondary factor in all 10 projects. Realised investment costs are the primary factor in 5 projects and a secondary factor in another 4 projects. Travel demand is a primary factor for the discrepancy between ex ante and ex post results in 8 of the 10 projects. Interestingly, the opening year is also a factor in 9 of the 10 projects, albeit a primary one only in three projects.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 91
Figure 17. Main reasons for discrepancies between ex ante and ex post results
Outturn Economic Population Marginal Discount Opening year investment Travel demand Other growth growth values rate (delay) costs
Railway - Spain ✓✓ ✓✓ ✓ ✓✓ ✓✓
Road - Poland ✓ ✓ ✓ ✓✓ ✓✓ ✓✓ ✓
Railway - Portugal ✓✓ ✓✓ ✓✓ ✓ ✓✓ ✓✓
Road - Spain ✓✓ ✓✓ ✓
Road - Greece ✓✓ ✓✓ ✓ ✓✓ ✓ ✓ ✓
Road - Ireland ✓ ✓✓ ✓✓ ✓✓ ✓ ✓
Railway - Greece ✓ ✓ ✓ ✓✓ ✓ ✓
Road - Lithuania ✓✓ ✓ ✓✓
Railway - Slovakia ✓ ✓✓ ✓✓ ✓✓ ✓✓ ✓
Road - Hungary ✓✓ ✓✓ ✓✓ ✓ ✓
Note: ✓✓ (Primary reason) ✓(Secondary reason)
This chapter of the report has presented an overview of the findings of the ten transport projects. Further details on the analysis for each specific case study can be found in Appendices 1 to 10. In the next section, we use the findings from this review to assess cost benefit analysis as a method, thus addressing the key research questions identified in the TORs for the study.
Evaluation of selected projects
June 2011 | Frontier Economics, Atkins, ITS 93
5 Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
This chapter builds on the findings from the selected transport projects described in the previous chapter to draw out lessons on cost-benefit analysis (CBA) as a method for ex ante appraisal and ex post evaluation of transport infrastructure projects. Specifically we draw lessons on cost benefit analysis as a method, as requested by the TORs for the study , in relation to: the strengths and weaknesses of CBA methodologies as applied by the Member States; the effectiveness of CBA as a tool supporting the project generation and project decision of the Member States and the Commission; the utility of ex post CBA from the point of view of the project promoters, Member States and the Commission; and, the extent that ex post CBA is an appropriate tool for evaluation of impact assessment; The first two issues relate primarily to ex ante CBA and are addressed in section 4.1. The third and fourth issues relate to ex post CBA and are addressed in section 4.2.
5.1 Findings on ex ante CBA
5.1.1 Modelling and forecasting quality The importance of the demand analysis and forecasts (including data collection and modelling) to a CBA cannot be overstated. Central to the delivery of a robust economic appraisal, is the development of a reliable demand model. Whilst there are some examples of good practice in the ten transport projects, we find that overall the demand analysis undertaken in the ten projects is weak by today‟s standards. We set out our main findings next.
The modelling framework. Three of the ten transport projects used detailed network models to assess the impacts of the project. The remaining transport projects were based on „sketch‟ models and/or spreadsheet calculations. The geographic scope of the network model was fit for purpose whilst those of the sketch models were too limited by today‟s standards. Six of the ten transport projects distinguished between work and
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
94 Frontier Economics, Atkins, ITS | June 2011
non-work trip purposes, and four out of ten transport projects accounted for different time periods. Good practice suggests that forecasts should distinguish by time periods and by trip types. Base year traffic flows were realistic in all transport projects with one exception.
Demand forecasting methods. Regarding the modelling of how travel demand changes in response to a new project only five of the ten transport projects accounted for re-routed traffic (out of the 9 projects where re- routed/re-assigned traffic was relevant). Demand re-routeing was estimated using diversion factors in two cases and an assignment model in the other cases. Only four transport projects accounted for modal shift, two of which used a logit approach16, and two used mode diversion factors from changes in generalised cost. Only four of the ten transport projects accounted for induced traffic (or generated traffic), though none of these used an elasticity approach or a formal model of behavioural response.
External assumptions. Typically, the key external drivers to growth in demand for a transport service/road are the effect of economic growth, population change and (related to these two) changes in car ownership.17 The effect of competing or complementary projects on the demand for a transport service can be large. Six of the ten transport projects account for GDP growth, two transport projects account for changes in car ownership levels, and two transport projects account for changes in population. Only four of the ten transport projects accounted for other changes in the transport network.
Demand analysis application. There were no significant errors in the application of the demand forecasting methods in seven of the ten transport projects. In three transport projects, however, there are issues associated with either poor demand forecasting in the base case scenario (i.e. no account of natural growth) or when traffic demand would stabilise in the future. Ideally there shouldn‟t have been any errors. Table 25 sets out some specific weaknesses that were present in the demand forecasting methods in one or more of the transport projects. It should be emphasised that no case study included all or even a majority of these
16 The logit approach is a statistical technique used to model discrete choices (such as, in this case, the choice between two alternative transport methods. Using this technique, it is possible to estimate how the probability of using a transport mode varies with changes in the underlying conditions (for example, fares or travel time). 17 Tolls and public transport fares assumptions are also critical to the demand forecast, though these are in the main viewed as internal to the transport policy.
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
June 2011 | Frontier Economics, Atkins, ITS 95
weaknesses. Table 15 shows, however, that some of the weaknesses present in the transport projects are quite basic.
Table 25. Identified weaknesses in the ex ante CBA analyses
Category Errors highlighted in one or more case study
Base year situation No origin destination matrix to calculate travel patterns in (i.e. understanding the study area of existing travel demands) No boarding and alighting counts undertaken, only counts of patronage on the full section of line, at an annual scale
Public transport operations not up to date with the latest timetables and frequencies
Modelling Limited geographic scope of model yet the project has an framework influence over a wide network area
No segmentation of user classes
No segmentation of vehicle types No segmentation of demand by different time periods
Demand Travel demand patterns are assumed fixed (i.e. assumption forecasting of no induced travel) methods No inclusion of natural growth in base level demands (e.g. aligning with GDP/population/income forecasts)
Inappropriate growth in GDP levels in future years
Behavioural parameters such as the value of time are not current enough and /or do not change over time
External Benefits were set to be at 100% from year of opening – no assumptions inclusion of ramp up of effects
No inclusion of developments in the area
Future parallel transport improvements not accounted for in the assessment
5.1.2 Adequacy of the approach to network effects Policymakers are often interested in network effects both from a policy perspective but also from the perspective that these effects may be missing from the appraisal. The studies can however be quite imprecise in their definition.
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
96 Frontier Economics, Atkins, ITS | June 2011
Effectively, network effects are benefits that occur „outside‟ of the new transport infrastructure brought about by the project. They can occur on other parts of the transport network or they may occur in other markets – as changes in transport accessibility and costs affect production and distribution of goods including that of labour. One categorisation of network effects (Laird et al, 2005) uses two categories: transport network effects and transport/economy network effects: Transport network effects are effects in other parts of the transport system; and, Transport/economy network effects are effects in the economic sectors which transport connects by forward and backward linkages. Table 26 summarises the treatment of network effects in each of the ex ante studies for the 10 transport projects. Unfortunately, as can be seen from this table, none of the transport projects can be held up as a good example for the inclusion of network effects in an ex ante assessment. Looking in more detail at the table it is clear that: The ex ante studies are better at capturing transport network effects than transport/economy effects. No transport/economy network effects are in fact captured in any of the 10 ex ante studies. The ex ante studies relating to road projects have a more complete coverage of transport network effects than those relating to rail projects. To a certain extent the level of coverage reflects the availability of modelling tools and data. Road network congestion models have been standard tools in transport planning for at least 15 years. There are a number of software packages that can perform this modelling function. In contrast the available skills and data required to model network effects in rail is smaller. More effort therefore has to be made by project promoters to capture these effects in the appraisal than those associated with road congestion. Clearly – aside from Bratislava Rača – Trnava railway – this extra effort has not been made. Transport/economy network effects are even more difficult to capture (Laird et al, 2005). The evidence on them is limited, internationally, and mainly confined to agglomeration impacts which are not particularly relevant in rural areas and to inter-urban projects (Eddington, 2006; Department for Transport (UK), 2009). It is therefore much more understandable that these effects have not been included in the CBA although DG REGIO might place emphasis on the transport/economy relationship.
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
June 2011 | Frontier Economics, Atkins, ITS 97
Table 26. The treatment of network effects in the ten transport projects
Network effects included in CBA Relevant network effects omitted from CBA
Transport/economy Transport/economy Transport network effects: Transport network effects: network effects network effects
Railways
High speed None None Rail: Cross-border services (unit Agglomeration – railway Madrid – operating costs reduced) facilitated by longer Barcelona in Air: frequency unchanged but plane distance commuting to Spain size reduced (unit operating costs Barcelona increase) Algarve Railway in Road: de-congestion benefits None Rail: Lower unit operating costs (due None Portugal in Lisbon area to increased track and rolling stock utilisation rates) Railway Thriassio None None Rail: Might be some reduction in Agglomeration – – Kiato in Greece operating costs on other parts of rail anecdotal evidence of network due to increased utilisation commuting into Athens of rolling stock along improved line. Road: delays at old level crossings in towns removed Bratislava Rača – Rail: lower operating unit None Road: De-congestion benefits on Agglomeration – Trnava Railway costs along whole corridor corridor to Bratislava improved access to jobs Upgrade in Road: reduction in accident in Bratislava and Trnava Slovakia benefits on road network
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
98 Frontier Economics, Atkins, ITS | June 2011
Network effects included in CBA Relevant network effects omitted from CBA
Transport/economy Transport/economy Transport network effects: Transport network effects: network effects network effects
Motorways
A2 Motorway in De-congestion on existing None Rail: reduced use of rail for freight Knowledge spillover Poland roads Road: Maintenance costs of feeder from FDI investment in Worsening of congestion on roads to the motorway has increased economic zones R14 Freight: economies of density in Potential for Inter-relationship with freight sector due to location of large agglomeration impacts proposed new motorways distribution and warehousing sites in Lodz A23 Motorway in De-congestion on existing None None Knowledge spillover Spain route from FDI investment in technology parks Agiou De-congestion in Agiou None Reduced flooding (due to improved None Konstantinou Konstantinou drainage on road) has reduced costs bypass in Greece Inter-relationship with to businesses and households (this is proposed new motorways an environmental externality) M1 Northern De-congestion on existing None None None Motorway in route Ireland IX B transport De-congestion and re- None Mode shift from PT in Vilnius Knowledge spillover corridor in routeing in Vilnius from FDI investment in Lithuania economic zones Agglomeration impacts in Vilnius M0 Budapest Ring De-congestion and re- None Mode shift from PT in Budapest Agglomeration impacts Road in Hungary routeing in Budapest in Budapest
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
June 2011 | Frontier Economics, Atkins, ITS 99
5.1.3 Strengths and weaknesses of CBA as applied by EU Member States As mentioned in Chapter 2, ex ante CBA is a mature method regularly used worldwide for the appraisal of transport infrastructure projects and policies. As the HEATCO comparative research showed, there is a basic level of consistency in CBA methodology for transport infrastructure across EU Member States (Odgaard, Kelly and Laird, 2005). For example, all CBA methodologies include construction costs, user benefits due to travel time savings and safety benefits. There is a common understanding of the process of discounting, the microeconomics of consumer and producer surplus, and the meaning of Net Present Value (NPV).
Consistency of ex ante CBA in practice Our findings from the 10 transport projects confirm such common understanding, which is reflected in the CBA framework used to assess the ten projects and which is both consistent with DG REGIOs guidelines and good practice. All ten projects consider investment costs (including ongoing maintenance costs) and measure the user benefits and some externalities (including safety benefits). In practice, however, there are striking differences of scope in the CBA analysis applied in different EU countries in terms of the set of impacts that are included
(time savings, CO2, noise, etc), for example. Different EU countries also use different – and often inconsistent – appraisal parameters such as values of time, values of safety, base years and discount rates. Annexe 3 contains a brief survey of the experience of ex post CBA practice across selected European countries. In this section we use the findings summarised in chapter 3 to reflect on the strengths and weaknesses of CBA as a project appraisal method. Set of impacts. Figure 3 in Chapter 3 describes which impacts are valued in the ten case study CBAs. This may be compared with Table 10 in Chapter 2, which shows the set of impacts included in principle, in each country‟s ex ante CBA methodology. It is apparent that: there are substantial differences in scope between national CBA methods; whether such differences have any significant impact on project NPVs or BCRs depend on the magnitudes of the (dis)benefits in those categories, i.e. how large the noise, air pollution and climate change impacts are how large the „wider impacts‟/„indirect socio-economic effects‟; how large the impacts on operators. and how large the disruption during construction. the ten case study CBAs are broadly consistent in scope with the national CBA methods. Thus, for most of the countries covered in this
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
100 Frontier Economics, Atkins, ITS | June 2011
sample, environmental valuation is not yet standard practice, the exceptions being Spain and Portugal, Poland (air pollution only) and Greece (bundled value for all externalities). We expect this situation to progress, since valuation methods and tools are increasingly refined and available. Wider impacts are not valued in any of the 10 case study projects. This is an indication of the difficulties that are normally encountered when attempting to express wider impacts in monetary terms. Scope of analysis. Figure 6 in Chapter 3 shows how the benefits accrue in each of the 10 transport projects. It provides some insight into the ability of CBA to address very different project types – road and rail in particular. It also raises some questions about the narrow scope of many of the CBAs evaluated.
Whilst time savings are the dominant benefit item in the motorway projects18, the rail projects tend to exhibit modest time savings but substantial gains to the operator through farebox revenue increases and benefits to the environment. The manner that benefits accrue between operators and users is a function of the pricing policy, which can be linked to fare regulation. This illustrates the importance of understanding the pricing policy of the proposed project when undertaking an ex ante CBA.
Additionally, we might expect the motorway projects to: increase CO2 emissions through increased traffic and fuel consumption; have a negative impact on the fare revenues of other modes – especially rail have some impact on air pollution and noise – the net effect being driven by local factors including traffic routeing behaviour, the alignment and design of the road, and the use of mitigation measures (noise barriers, grading, etc) However these effects are often omitted – often because they are not part of the objectives of the project, and instead are unintended and undesirable externalities from it. This is at odds with the method of CBA, which states that costs and benefits to all relevant groups (users and non-users) affected by the project should be measured. Discount rates. Overall, the economic and financial discount rates used in the 10 case study projects appear to be appropriate and consistent with guidance
18 With the exception of Lithuania, where Vehicle Operating Costs make up the majority of benefits – this appears to be an inconsistency arising from the use of HDM in the Lithuanian CBA specification.
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
June 2011 | Frontier Economics, Atkins, ITS 101
(DG REGIO, 2008; Bickel et al., 2006). The exceptions are the 10% discount rate for the Slovak rail project and the 8% rate for the Poland road project, which are unexpectedly high. The rates recommended in the DG REGIO guidance are risk free rates reflecting social time preferences. With such low rates a risk analysis needs to be undertaken. Other methods are also used to determine discount rates including opportunity cost of capital (i.e. a financial rate), furthermore discount rates can include a risk premium discounting benefits a long way in the future heavily as they are „risky‟. The Slovak and Polish discount rates are more consistent with these „risky‟ discount rates and/or the opportunity cost of capital rates. There is in itself nothing wrong with using such rates as long as consistency is maintained between the projects that compete for funding (which for Cohesion Funding is at a national and not EU level). Appraisal period. Appraisal periods (see Table 15 in Chapter 3) were within the expected range of 20-40 years plus residual values for longer lived assets – primarily rail projects – although a comparison of the Polish A2 motorway and Spanish A23 motorway highlights the divergence that can occur – 20 years versus 34 years for similar projects. The reference time horizon for road projects is given as 25 years (DG REGIO, 2008:37). In part we see the divergence in appraisal periods reflecting the discount rates used. With high discount rates benefits in the future have little value and therefore appraisal periods of 20 years are acceptable in this context. That is, the high discount rates evident in the Slovak and Polish ex ante studies are associated with short appraisal periods. The other studies with discount rates closer to the level recommended by the DG REGIO (5.5%) have longer appraisal periods. Marginal cost of public funds. A marginal cost of public funds (MCPF) was included only in the Irish M1 CBA, however that is broadly consistent with the advice in the DG REGIO guidelines to assume MCPF=1.0 in the absence of specific national-level guidance on the issue (DG REGIO, 2008:54). Practice is in fact rather divided on this point: in Denmark and in Slovenia, 20% is added to the net costs financed through public funds; in Sweden, 30% is added to the resource from the General Budget (Bickel et al., 2006:47) – elsewhere no addition is made. HEATCO recommends assuming MCPF=1.0, but applying a cutoff of 1.5 to the ratio of NPV to Public Sector Support (RNPSS) when selecting projects. Value of time. With regards to the value of time, it is clear that the basis for its calculation varies across countries, although with sufficient information and assumptions these values can be compared with each other (see Table 16 in Chapter 3). We have also compared them with the HEATCO values – for time these are based on a meta-analysis of willingness to pay studies across the EU, whilst for safety these are based on a benefit transfer procedure from the leading European studies. In the main the values are low compared to the HEATCO values. There is in itself nothing wrong with this as long as the values are
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
102 Frontier Economics, Atkins, ITS | June 2011
representative of willingness to pay in the region or country. Using local values has an advantage. In particular, using local values in forecasting models ensures that predictions are based as far as possible on actual behaviour. It is then possible to conduct the appraisal using the same values. Our comparisons however with data on willingness to pay would however suggest that in a number of instances the values are too low. Adjustments were therefore made to these values before we undertook the ex post studies. It is worth noting that the Hungarian CBA was subject to Quality Assurance by a third party (COWI) who cross-checked the marginal values with other international values – a useful way of verifying the quality of the appraisal before submission. We did not make any adjustments to the Hungarian values in undertaking the ex post CBA for the M0. A particular issue with some of the ex ante CBAs we evaluated is that there are no separate values for working and non-working time, and in some cases for freight. Depending on the mix of passenger travel/freight and journey purpose changes instigated by the project, this could give a misleading estimate of the project benefits. Safety. With regards to the assessment of safety impact, the ex ante CBAs reviewed in this study used a very diverse set of definitions, units, base years and accounting conventions (Factor Prices (FP)/Market Prices (MP)) for safety impacts. While this diversity does not necessarily imply inconsistency, it does obscure the comparison and makes checking the CBA much harder. For example, the generic „externality value‟ for road traffic in Greece bundles safety and environment together, reducing transparency and auditability.
Risk and Uncertainty DG REGIO (2008) recommends undertaking risk analysis to account for uncertainty. The application form for Cohesion Funds also includes a section on the sensitivity of the economic results to input parameter values. A positive finding is that all the ex ante studies included some risk analysis. In all instances, the risk analysis was restricted to the sensitivity testing of different scenarios and in some cases had quite a limited scope. None of the studies undertook a quantitative risk analysis using Monte Carlo simulation via comprehensive software packages such as @RISK or Crystal Ball as recommended by DG REGIO (2008). Instead, all ten transport projects used basic sensitivity tests, with changes in assumptions and forecasts included within the model (see Table 19 in chapter 3). These have included the following: ▫ Cost overruns on the project – eight of ten transport projects; ▫ Under spend on the project – seven of ten transport projects; ▫ Delays and overruns on the project, or early completion – none of the ten transport projects;
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
June 2011 | Frontier Economics, Atkins, ITS 103
▫ Varying the discounting rate used to the costs over the project life – five of the ten transport projects; and ▫ The length of the appraisal period – none of the ten transport projects. Furthermore, two transport projects changed the confidence intervals on marginal values (such as the value of time), but only one case study tested the impacts of changing the marginal values within the sensitivity tests. A minority of projects reported sensitivity tests on key modelling issues, such as GDP or demand growth. The above discussion illustrate that while each case study has undertaken some form of risk analysis, there has been a limited overall approach to it. None of the transport projects has applied comprehensive risk analysis software, and instead have applied only basic sensitivity tests to the results. While the majority have included an assessment of cost overruns and under-expenditure, none of the ten transport projects have accounted for project delays (or early completion) which have proved commonplace. None of the projects have accounted for all variations in external factors that affect travel demand including GDP growth, future population growth, levels of car ownership, and the introduction of competing or complementary transport projects.
5.1.4 Effectiveness of CBA in project generation and design In most of the transport projects, the ex ante CBA was carried out for a single implementation option. This is because in many instances the project had already been defined as part of a wider national investment policy. In such circumstances, the ex ante CBA is mainly an exercise necessary for the funding application rather than a tool to choose between alternative implementation options. This finding is illustrated in Figure 4 in Chapter 3 where it can be seen that the ex ante CBA was used mainly to meet EC funding criteria and to check value for money on the proposed project. In many instances the ex ante CBAs were undertaken several years ago. Given that transport planning is an evolving practice, we investigated through interviews and at the workshop with EU Member States held in Brussels on 3rd February 2011, whether the role that CBA plays in the decision-making process is different now. The response was mixed. In some cases examples were given of CBA analyses forming part of a prioritisation strategy within road investment programmes. In other instances, the position has not changed, with CBA having only a limited role in decision-making. A number of reasons for this were advanced and these are summarised next:
The location and alignment of transport investments are often determined by environmental impacts and the political priority to reduce regional disparities – neither of which features particularly strongly in a transport CBA. There exist legislative requirements in each country regarding the
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
104 Frontier Economics, Atkins, ITS | June 2011
protection of the environment (e.g. forests, wetlands, etc.). This in itself is a huge obstacle in the planning and design of transport infrastructure, which limits the number of project options which can be feasibly subject to a CBA.
There also exists the need to create a complete network, both from a transport perspective and from political and legislative perspective. In some cases Member States are legislatively obliged to invest in the TEN-T as part of the terms of accession to the EU. It also makes little sense to only upgrade part of a route to either motorway standard or to a higher speed rail standard – as this would leave „missing‟ links.
Transport CBA is only a partial analysis. CBA is therefore strong at comparing projects which have similar types of impact but differ in scale. An example was cited at the EU Member State workshop of comparing asphalt and concrete as a design choice during the construction of a motorway. Another example was cited of prioritising the phasing of an upgrade to motorway standard of a corridor/route. With only a partial analysis, transport CBA is at its weakest when comparing projects with very different impacts – such as the types of impacts generated by the different modes. Figure 6 illustrates the differences in impacts between road and rail for the 10 transport projects analysed. Because of these differences it makes it hard to compare the NPVs and BCRs of road and rail projects objectively. Therefore, in our view, our finding that the NPVs and BCRs of rail projects are lower than that of road projects, should not be taken to imply that road projects are better than rail projects per se. This view points towards the principle that CBA forms part of a broader decision process.
A final point that was made during the workshop was that no project or application for funding had been rejected, as far as the participants were aware, for having a poor CBA result. For countries where CBA is not already part of the planning process this can reinforce the view that the CBA has little practical meaning and is just a „hoop‟ to jump over on the path to qualifying for EU funding.
5.2 Ex post CBA findings
5.2.1 Utility of ex post CBA to project stakeholders Some of the key contributions of ex post programmes are transparency and accountability, strengthening the evidence base and providing feedback on the ex ante process (see annexe 3). We use these categories to illustrate our findings on the ten transport projects in relation to the benefits of ex post work to project stakeholders, including promoters, Member States and the European Commission.
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
June 2011 | Frontier Economics, Atkins, ITS 105
Transparency and accountability Overall, the ten transport projects examined in this report demonstrate an acceptable or good value for money from the perspective of the European taxpayer. The exceptions are the two Spanish studies. The A23 road project is marginal in terms of value for money and the AVE Madrid Barcelona high speed line offers poor value for money. Whilst these general finding represent positive news, in the spirit of transparency and accountability, publication of such ex post findings, including the Spanish findings, does allow stakeholders, Member States and the Commission to ask hard and sometimes uncomfortable questions of each other and the Member State‟s planning authorities, which fosters accountability.
Strengthening of the evidence base We have collected data on unit costs of construction. Aside from one case study these were only available up to the Level 2 level of disaggregation (see Figure 8 in Chapter 3). Ideally they should also have been available at the Level 3 level. With only 10 transport projects the sample is too small in itself to demonstrate any patterns in unit costs, however, with a full ex post programme a database on unit costs can be developed and these can then aid in the planning of future transport projects.
Capital costs. We have found that the average of the difference between ex ante and ex post capital costs across the ten projects is 13.5% (see Figure 16 and accompanying text). This value is rather low and in line current experience. Nine projects had experienced cost overruns. The most significant cost over run was on the Ireland M1 motorway whilst the under- spend occurred in Hungary (M0 motorway). Interestingly there are also a handful of projects where ex ante and ex post capital costs are very close, including the A23 motorway in Spain and the A2 motorway in Poland. A number of factors seem to be at play here: Firstly the large cost overrun is associated with ex ante costs that were developed well in advance of construction. In the case of the Ireland M1 motorway the ex ante costs were estimated in 1995, when construction didn‟t finish until 2003. Project delays, alterations in scope and other unforeseen circumstances all increased capital costs. Reflecting this uncertainty large uplift factors are applied in the UK to capital costs estimates for projects at an early stage of design (see Annexe 4 which shows that in the UK an uplift factor of 44% for road and 66% for rail is applied the Stage 1 level of design).
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
106 Frontier Economics, Atkins, ITS | June 2011
Another factor at play here is that some Member States took the opportunity when applying for Cohesion Funds to update the ex ante CBA with a more up to date set of capital costs. In these situations the design of the project is at a very advanced stage and uncertainty in capital costs has been minimised. This can lead to some very close estimates of ex ante and ex post capital costs. In the Hungary case, despite an update in capital costs immediately prior to submission for Cohesion Funding, there was a large over estimate in the ex ante costs due to the uncertainty of the impact of international construction firms bidding for work in the old Eastern Block.
Utilisation rates. With long living infrastructure such as transport it is necessary to build in sufficient spare capacity to give room for growth over the project‟s lifetime, so we would never expect a project to be operating close to capacity soon after opening. Similarly, however, unless there are extenuating circumstances (such as a large uplift in demand is expected when the networks complete) we would not expect a very low levels of utilisation in the early years. As can be seen from Figure 9 in Chapter 3 in one study very low utilisation is evident (A23 motorway in Spain) while another study (M1 northern motorway in Ireland) shows very high levels of utilisation. These examples are suggestive of an imbalance between ex ante demand forecasts and actual demand.
NPV. Seven out of the 10 case study projects yielded an NPV that was lower ex post than expected ex ante. Three yielded higher NPVs than expected (M1 motorway Ireland, IX B transport corridor Lithuania and Bratislava Rača – Trnava Railway in Slovakia). We have also found that in eight out of ten transport projects differences between ex ante and ex post travel demand are a primary cause of the differences in NPV, whilst in five transport projects differences in capital costs are also a primary cause. Whilst a sample size of 10 does not provide any statistically robust conclusions it appears to us that these findings, in conjunction with some low utilisation rates and the cost overruns, is indicative of optimism bias being prevalent in the ex ante analysis. To fully develop evidence on unit costs and the presence of optimism bias in the Cohesion Fund investment programme this limited evidence base would need combining with other data or strengthened through more ex post studies.
Feedback An important attribute of CBA ex post analysis is the feedback it can provide on demand modelling, ex ante appraisal and infrastructure design. Some of the lessons learnt from our analysis of the ten transport projects are:
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
June 2011 | Frontier Economics, Atkins, ITS 107
Modelling practice. Ex ante demand modelling would benefit from improvements in study area definition, modal coverage and inclusion of induced traffic/variable demand.
Improved definition of the counterfactuals. We have identified inappropriate assumptions regarding travel demand, traffic growth and vehicle speeds in the „without‟ project scenarios. For rail and toll motorway projects it is essential that the pricing policy is defined correctly. Not only does the pricing policy affect the distribution of benefits between operators and users but it also strongly influences the demand for a project.
CBA practice. The CBA analyses would have benefited from: More consistency in scope, the inclusion of environmental externalities in the CBA and the incorporation of network effects; a disaggregation between business and non-work transport user classes; an opportunity to improve the comparability of CBAs for decision- makers, by moving towards harmonised definitions, measures (units), methods and assumptions where appropriate – so that for example the values of time, safety and environment impacts used in CBA will be on the same basis, even if they will not (rightly) be exactly the same throughout the EU.
Risk analysis. While some of the risk bearers were identified in the risk analysis, not all of them were. The main sources of difference between ex ante and ex post economic analyses were differences in capital costs and travel demand. Differences in travel demand occurred due to variances between ex ante and ex post in terms of economic growth, development impacts and whether other transport projects in the locality had been constructed.
Identification of wider impacts. The lack of project monitoring frameworks, implemented at project opening, makes the identification of wider impacts particularly challenging. This is especially the case for projects being implemented as part of a wider modernisation strategy, against a backdrop of rapid economic growth in the early 2000s and infrastructure investments. In these cases, identifying a causal link between a project and wider socio-economic impacts is challenging.
Project design. The ex post analyses did not identify any unintended consequences aside from impacts on noise that should have been mitigated against. This suggests improvements in the design of such mitigation
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
108 Frontier Economics, Atkins, ITS | June 2011
measures are needed, however, the sample size is too small to make a general statement based on this study.
Good practice guides and regulation. The desire for and need of good practice guides on economic appraisal and demand modelling was identified both in our interviews and in the Member States workshop. There was a mixed response to the need for more regulation.
5.2.2 Ex post CBA as an impact assessment tool Technical and institutional challenges to ex post analyses A number of technical challenges exist in undertaking an ex post analysis whether it is a CBA or some or other form of ex post analysis. These are:
Loss of institutional memory. In terms of planning and design, most of the projects we have considered in this study date back to the late 1990s. This was also the time in which the sponsoring organisations carried out the first comprehensive cost-benefit analyses. Due to the time elapsed, some of the institutional memory regarding the ex ante analyses has inevitably been lost. In relation to this study, for example, this implied that it was not possible in all the projects to verify all the assumptions underpinning the original CBA analysis.
Co-ordination of different groups of stakeholders. We found that the number of stakeholders involved with the project can delay the engagement of the relevant stakeholder and the provision of information. This is particularly the case for rail projects, some of which had been implemented by the national railway company before the unbundling of network and train operations. This implies that the information underpinning the ex ante analysis may be split between two different companies thereby creating additional hurdles to collect the necessary information.
Elapsed time since opening. The majority of the projects we have reviewed opened relatively recently, in one case the last phase of a project opened as late as summer 2010. The short elapsed time since project opening raises two issues with the ex post evaluation: While it is possible to obtain outturn information on project costs and existing traffic levels, there is a need to make assumptions about future traffic levels. This can led to a reasonable degree of uncertainty in the calculation of project benefits, especially in the context of the current global economic downturn; and, Wider socio-economic impacts generally take a significant period of time to emerge. For this reason, it can be difficult for an ex post analysis
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
June 2011 | Frontier Economics, Atkins, ITS 109
to consider all the effects since some of them may have not yet materialised.
Definition of the counterfactual. Any change that occurs simultaneously with the opening of the transport project makes it difficult to identify the effects of the project. In these transport projects, economic change and changes to the transport network were the two biggest confounding factors. Rapid economic growth in some EU accession countries before 2008 and the economic recession have influenced the general pattern of economic growth in the EU. All of the projects studied suffered from this in one form or another. Another problem that is most evident with the Hungarian road project is that the impacts are confounded with that of other transport projects. With respect to the M0 Eastern Sector several other transport initiatives of a similar scale to the M0 Eastern Sector affected traffic flows within weeks of the M0 opening. There clearly exists a dilemma regarding the best time to undertake an ex post CBA analysis. On the one hand, it needs to undertaken as soon as possible to minimise institutional memory loss, maximise the value of feedback into the ex ante planning, modelling and appraisal processes and to make the definition of the counterfactual as easy as possible. On the other hand, there may be a desire to wait until the transport impacts have fully fed into the wider economy and land use patterns have settled down. This was discussed at the Member State February 2011 workshop and overall, scheduling an ex post CBA 3-5 years after project opening was felt to be an appropriate compromise. Impact assessment versus ex post CBA Figure 3 in Chapter 3 (for the 10 case study projects) and Table 10 in Chapter 2 (for the EU Member States in general) highlight the gaps between good practice – a CBA which addresses the full scope of impact assessment – and current practice. In particular, this study has revealed that wider impacts require special treatment and some work has been done to develop suitable methods to address this gap. It is important that promoters, EU Member States and the Commission consider this need to bring the analysis that is carried out as close as possible to decision- makers‟ policy interests and information requirements. Additionally, for environmental impacts, methods do exist within current „good practice‟/ „state of the art‟ CBA as described by HEATCO (Bickel et al., 2006), which can be added- on to transport models. Potentially, the quantification of environmental impacts should be included more widely in ex ante cost-benefit analysis. Alternatives There are alternative ex post methods to CBA. These can range from qualitative analyses of impacts, elicited through say a survey, to complex econometric
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
110 Frontier Economics, Atkins, ITS | June 2011
exercises that use large disaggregated time series datasets. Any ex post study using an alternative ex post method will suffer from the four challenges outlined above – these issues are pervasive to all ex post analytical work. The choice whether to undertake a CBA ex post analysis or use an alternative method relates therefore to the questions the ex post study is expected to answer. The strength of ex post CBA is in its holistic approach and its ability to deliver unambiguous indicators on value for money. The fact that a transport CBA focuses on transport flows and costs means, however, that it is not as strong as some alternative methods at answering questions about changes in social inclusion for minority and disadvantaged groups, economic output, wealth and employment (disaggregated by income group and area), for example. The appropriateness of ex post CBA as a tool for the evaluation of impact assessment depends therefore on the impacts being evaluated and whether a holistic approach is required for such evaluation. Finally, we note that there may be an opportunity to combine CBA with multi-criteria analysis as a way to embed wider impacts and environmental impacts with either a CBA or the inputs to a CBA (for example, travel time savings) (see, for example, the final report of Expert Group 5, part of the 2010 TEN-T Policy Review)
Cost-benefit analysis as a project appraisal method: findings from the ten transport projects
June 2011 | Frontier Economics, Atkins, ITS 111
6 The role of CBA in predicting the macroeconomic impacts of transport investment
In this chapter we address a fifth research question included in the Terms of Reference of the study: what relevance and potential utility data obtained from CBA would have in macro-economic modelling? Chapter 2 mentioned that improving economic performance is a key motivation for investing in European transport infrastructure. The spatial pattern of GDP and employment as a result of transport investment is also important. Macroeconomic impacts of transport projects have been predicted in a number of research studies by linking transport models of accessibility change to macroeconomic models. As reviewed in Annex 1 there also exists a large body of evidence that transport projects will influence the economy. The main impact however is not on endogenous growth but in relocating economic activity from one area to a more accessible area. In our case studies we also examined the wider impacts of transport projects. In the main, due to difficulties with identifying the counterfactual and the small amount of time between project opening and the ex post analysis taking place, we had difficulty identifying impacts on GDP or employment. However, as illustrated in Figure 7 in Chapter 3 we were able to identify impacts on the supply chain (new improved distribution centres) and on land use (new business parks, trade zones, housing and retail parks). The analysis undertaken to assess the wider impacts of the transport projects is fundamentally different to that used for the CBA. Primarily this arises as CBA is very much an holistic measure of a project‟s impact whilst in the wider impact analysis we undertook the concern is with the impact in the locality or wider area of the project. Macroeconomic models (of transport projects) have the same interest and as a consequence their focus needs to be on a locality and identifying movement of economic activity to and from it. A number of other differences between the philosophy underlying CBA and that underlying macroeconomic models exist but the holistic nature of a CBA is the key one (see Annex 1). The holistic nature of a CBA means that non-work user impacts, option values are always included in a CBA. These impacts do not feature in a macroeconomic model, neither do the welfare costs associated with pain, grief and suffering (as part of safety benefits) nor the environmental costs (carbon, air pollution and noise). An approximate measure of the impacts on GDP of a transport project can be obtained by summing the business and freight related user benefits, the profitability of transport operators and, if agglomeration related network effects
The role of CBA in predicting the macroeconomic impacts of transport investment
112 Frontier Economics, Atkins, ITS | June 2011
have been calculated, the changes in productivity due to increased agglomeration. As discussed in more detail in Annex 1 this is only an approximation to the macroeconomic impact due to for example the fact that changes in output and employment are measured differently in a CBA compared to in a macroeconomic model. A CBA measures the added value of bringing new resources into use, whilst a macroeconomic model gives the gross value of the new resource. This can lead to wide variations in the impact predicted (between CBA and macroeconomic forecasts) at a small geographic level when employment levels may change19. Similarly the CBA estimate on GDP is a net effect, and there can be significant gains and losses at a local level which have a distinct spatial pattern (i.e. one area gains whilst another area loses). If more robust macroeconomic forecasts are necessary the consensus of practice is to link a transport model of accessibility change to a macroeconomic model. This can lead to some inconsistencies between the economic forecasts from the macroeconomic model and those that underpin the transport model. As discussed in more detail in Annex 1, these inconsistencies can be overcome through either the use of a Land Use Transport Interaction (LUTI) model or a Spatial Computable General Equilibrium (SCGE) model. It should of course be noted that neither of these options are trivial.
19 As the main impact of a transport project is to relocate economic activity at a national level local employment gains and losses usually net out except for the largest of transport projects.
The role of CBA in predicting the macroeconomic impacts of transport investment
June 2011 | Frontier Economics, Atkins, ITS 113
7 Conclusions and recommendations
This chapter concludes our study on the ex post evaluation of a sample of ten transport projects funded by the Cohesion Fund (and ISPA) during the period 2000-2006. This main objective of the study is to answer the following three questions:
Question 1: What were the impacts of the ten selected transport projects based on an ex post CBA evaluation?
Question 2: How can ex post cost benefit analyses (CBA) contribute to the practice of ex ante cost-benefit analysis?
Question 3: What are the potential and the limits of ex post CBA as a tool to assess the impact of transport infrastructure projects? This report has addressed these questions by: providing a summary of the results of the ex post evaluation (Question 1); and, assessing cost-benefit analysis from a methodological point of view (Questions 2 and 3). We summarise our findings in each of the areas next. We then conclude the chapter offering a set of recommendations on how to improve the practice of ex ante project appraisal in the context of current EU regional policy instrument processes and practices.
7.1 Ex post evaluation summary Rather than provide a summary review of the quantitative results presented in chapter 3, we list a few key qualitative findings related to the ex post evaluation objective of the study: ▫ There is a fair evidence base for valuing the most significant direct impacts of transport investments (e.g. time, safety) and progress on some others (e.g. noise, carbon). However, other impacts such as reliability, despite being a source of qualitative benefits, are difficult to quantify as there are no all encompassing established methodologies for doing so. ▫ Moving from the transport sector impacts to the effects of infrastructure on the economy especially at local level remains a challenge. There is a gap in this analytical area between principle and practice, which is caused primarily by the difficulties associated with collecting data, but also with challenges related to modelling such effects;
Conclusions and recommendations
114 Frontier Economics, Atkins, ITS | June 2011
▫ All forms of appraisal rely on good base data and a credible model. Ex ante appraisal is capable of providing the base data and model against which to compare ex post evidence. This however depends on good documentation of the ex ante approach and maintenance of the model. It may be up to ten years between the ex ante and ex post analysis being undertaken. ▫ All forms of appraisal such as CBA or MCA are decision support tools which inform more broadly-based forms of decision-making. We note that even in Member States with well established appraisal method, CBA or MCA are only one of the factors that are taken into account to define investment strategies. ▫ Ex post programmes add transparency to the decision making process and have provided important feedback on how to improve demand modelling (including understanding the relevance of induced traffic, economic development and the impacts of changes in competition) and the reasons for cost overruns. They have also identified the importance of defining the counterfactuals and have exposed weaknesses in risk analysis. ▫ The strength of ex post analysis relies on a critical mass of case studies. A collective analysis from a minimum set of ex post studies has far greater robustness, and adds more analytical value, than simply drawing conclusions from a small number of studies.
7.2 Cost-benefit analysis as a method As indicated in the TORs, Task 3 of this study sets out the key research questions that need to be answered to assess cost-benefit analysis as a method. Specifically, we have considered the following: the strengths and weaknesses of CBA methodologies as applied by the Member States; effectiveness of CBA as a tool supporting the project generation and project decision of the Member States and the Commission; utility of ex post CBA from the point of view of the project promoters, Member States and the Commission; the extent that ex post CBA is an appropriate tool for evaluation of impact assessment; and, relevance and potential utility of data obtained from CBA in macro- economic modelling.
Conclusions and recommendations
June 2011 | Frontier Economics, Atkins, ITS 115
In the remainder of this section we provide a summary of our assessment of these methodological issues.
Strengths and weaknesses of CBA Ex ante CBA is a mature method, which is regularly used worldwide. There is a basic level of consistency in CBA methodology for transport infrastructure across EU Member States. For example, all CBA methodologies include construction costs, user benefits due to travel time savings and safety benefits. There is also a common understanding of the process of discounting, the microeconomics of consumer and producer surplus, and the meaning of Net Present Value (NPV). Our findings from the ten transport projects confirm the following: The CBA framework used for the appraisal of these projects is both consistent with DG REGIO‟s guidelines and with good practice. All these appraisals consider investment costs (including ongoing maintenance costs) on one side and measure the user benefits as well as some externalities. For example, an assessment of the safety benefits of the project has been carried out in all cases. Future benefits are discounted correctly and some risk analysis, albeit limited to a handful of assumptions, has been undertaken. However, we have found that the scope of the ex ante analysis has often been narrower than what would be ideal. In other words, the set of impacts considered in the analysis is limited. Often, the analysis only considers times savings, vehicle operating cost savings and safety
benefits, while other impacts (for example CO2 emissions, noise, pollution) are not always taken into account. Different EU Member States use different appraisal parameters, such as values of time, values of safety, base years and discount rates. This in itself is not an issue, but there appears little similarity in some instances between the values used and other sources for these values, such as HEATCO. As a consequence, for the ex post analysis we have in some instances decided to use alternative values to those used in the ex ante analysis. We have also identified weaknesses in the demand modelling, the definitions of the counterfactuals and the modelling of network effects. We note that a robust treatment of demand and the counterfactuals is fundamental to the reliability of a cost-benefit analysis.
Effectiveness of CBA in decision-making Whilst cost-benefit analysis is regularly used worldwide, it forms only part of a broader decision-making process – even in countries with a long history of using
Conclusions and recommendations
116 Frontier Economics, Atkins, ITS | June 2011
it. For many of the projects we reviewed, we found that CBA had been used mainly to confirm that the project offered value for money and to support the application to access EU funding. Furthermore, it is clear from our discussions with stakeholders that this limited role still remains the case in some EU Member States. One of the reasons for this, aside from the time it takes to embed CBA into existing planning processes, is the need to balance environmental concerns and regional development concerns. However, we think that even within these constraints, CBA can offer relevant information and be a useful tool to decision- makers compared to the extent to which CBA is currently used.
At a technical level, CBA is particularly strong in comparing projects that have relatively similar impacts. Thus it can be used to help make design choices, alignments (depending on the environmental impact), and help prioritise different elements of a transport investment strategy.20
At a political level, CBA also adds transparency to the decision making process, and improves the accountability of the decision-makers.
Utility of ex post CBA Ex post CBA can add value to the planning process. It adds transparency, strengthens the evidence base and provides feedback on the methods and techniques used to design and appraise the infrastructure. For these reasons, active ex post CBA programmes have been in place for some time in the UK and in France, for example. Amongst their various contributions, perhaps a key one has been their contribution to developing an improved understanding of “optimism bias”: the systematic bias during the planning phases of a project, whereby the benefits are overestimated, whilst costs and completion times are underestimated. Ex post evaluation programmes can help identify instances of optimism bias by identifying cost overruns and their causes. For example, in the UK this has led to new procedures designed to counter the bias. The evidence from the ten transport projects discussed in this report shows that optimism bias appears evident, with an average cost overrun of 13.5%. In almost all transport projects, differences between forecast and actual travel demand are the primary cause for differences in the ex ante and ex post net present value of the project.
Ex post CBA as tool for evaluating project impacts Ex post CBAs main strengths are its holistic approach (i.e. the consideration of a project‟s impact in its whole life cycle) and its ability to deliver an unambiguous
20 We note that the partial nature of a CBA makes it more difficult to make comparisons between different modes, for example road and rail.
Conclusions and recommendations
June 2011 | Frontier Economics, Atkins, ITS 117
indicator of a project‟s economic worth. A further strength of ex post transport CBA is that it focuses on the direct impacts of the transport project and its associated externalities. This makes the project monitoring programme reasonably straightforward. The focus on direct impacts is also a weakness, since there can be a gap between project objectives (e.g. increased employment and wealth to a region) and the inputs to a CBA. A particular feature of transport infrastructure is that it has a long life. There is therefore a need to forecast future benefits, this creates some uncertainty regarding the project‟s worth and leads to a desire to undertake a CBA as late as possible in a project‟s life. Another feature of new transport infrastructure is that the direct transport impacts (e.g. time savings) take time to feed into the economy. This suggests leaving as long a gap as possible after the project opens to undertake an ex post CBA study. A perennial difficulty with any ex post work, whether CBA or an alternative method, is the definition of the counterfactual. A good monitoring programme is needed, potentially at the transport corridor or national transport network level, in order to identify the impact of a project against background changes. Institutional memory loss is also an issue when conducting any ex post work. This is because with relevant people moving jobs (and even reform of public institutions) it can be hard to identify the decision-makers and understand the process of the decision making ex post. Furthermore it can make it difficult to obtain data and reports on the ex ante analysis.
Relevance and potential utility of CBA for macro-economic modelling The strength of CBA is, as mentioned above, its holistic nature and its ability to deliver an unambiguous indicator of a project‟s worth. Macroeconomic models on the other hand focus on a narrow range of impacts. Transport impacts, included in a CBA, that are not relevant for macroeconomic modelling include those associated with non-work time savings, safety and environmental externalities. At best, therefore, only a fraction of transport CBA output is relevant for macroeconomic modelling (namely that associated with business and freight impacts and any productivity changes due to agglomeration benefits). This can be interpreted as an approximate indicator of GDP impacts. However, the value of transport CBA output as a GDP indicator is undermined for two reasons. Firstly it is a „net‟ benefit, that is a series of gains and losses will occur as the economy adjusts to a change in transport supply and economic activity relocates in response to changes in transport accessibility. This pattern of gains and losses is of interest to the macroeconomic model, but the transport CBA can only provide limited information on it. Additionally, it is not clear from a CBA who receives the ultimate benefit. For example, in the long term user benefits of commuters may be passed on to employers via lower wages, whilst the major
Conclusions and recommendations
118 Frontier Economics, Atkins, ITS | June 2011
beneficiaries of haulage cost savings may be international companies based outside the country where the new infrastructure has been built. The second limiting impact is that transport CBA measures the added value of bringing new resources into use (increased output and employment), while macroeconomic modelling measures the gross value. These differences between CBA and macroeconomic modelling mean that in practice the macroeconomic impacts of changes in accessibility are best modelled by linking transport accessibility models directly with macroeconomic models. This can lead to some inconsistencies between the economic forecasts from the macroeconomic model and those that underpin the transport model. These inconsistencies can be overcome through either the use of a Land Use Transport Interaction (LUTI) model or a Spatial Computable General Equilibrium (SCGE) model – neither of which are trivial modelling exercises.
7.3 Recommendations In this final section, we offer a set of recommendations on how to improve the practice of ex ante project appraisal in the context of current EU regional policy instrument processes and practices. Specifically, our recommendations focus on three key areas: quality and consistency of ex ante CBA; ability to undertake ex post evaluation; and, provision of training and guidance. We discuss each in turn below. We note that these recommendations are based on the evidence we have collected in this study. As the study considered just ten projects, this evidence is necessarily limited.
7.3.1 Improve the quality and consistency of ex ante CBA The scope of the ex ante CBA was in line with the guidelines set out in the relevant EU project appraisal guide, but the quality and consistency of the analysis submitted in funding applications show large variation across Member States. The Commission has CBA guidelines that have to be followed but their use and application by Member States reveals gaps in coverage and superficial treatment of important issues.
Recommendation Based on the findings of our study, we recommend the Commission to undertake a review of the changes that could be introduced into current regulations to improve the quality and consistency of the ex ante CBA analysis
Conclusions and recommendations
June 2011 | Frontier Economics, Atkins, ITS 119
applied to major transport infrastructure projects supporting EU funding applications. Some of the changes include the following:
A set of objective indicators regarding scope and depth of the CBA that the application of the DG REGIO CBA guidelines should adhere to. These would include issues such as, for example, appropriate network definition to account for induced and redistributed traffic; demand analysis and forecast should be carried out by appropriate models, validated and calibrated using reliable data resulting from recent traffic counts related to the area serviced by the network above; thorough assessment and quantification of environmental impacts (including climate change impacts) on the basis of the latest available guidelines available in the CBA literature; consistency of parameter values with existing good practice (e.g. HEATCO); substantial differences from these values should be duly explained and justified quantified risk and sensitivity analyses with special and equal focus on costs and benefits (e. g. construction costs, construction time, initial and forecast traffic volumes); and, compulsory inclusion of a lower-cost option in the analysis submitted as part of the funding application to better understand the relative net benefits of the selected option.
The ability for the EC to request a quality appraisal (based on previously agreed methodology and well defined benchmarks) of the CBA analysis for major infrastructure projects (over a certain cost or financing threshold to be defined). These appraisals would be paid preferably by the national authority but undertaken by an independent third party evaluator selected by the Commission.
Rejecting funding applications for major infrastructure projects or requesting resubmission of the CBA if it is found to be incomplete or of insufficient quality; etc.
7.3.2 Improve ex post project „evaluability‟ While applied CBA remains a useful analytical tool to evaluate costs and benefits of major transport infrastructure projects, including their design, it has serious limitations to assess the generation and distribution of wider socio-economic impacts of such projects (for example project impacts on employment or land
Conclusions and recommendations
120 Frontier Economics, Atkins, ITS | June 2011
use) which take time to develop (ramp up) and are not that easily evaluated with current tools. Yet, a finding of our study is the relevance of wider socio- economic benefits and the need perceived by EU Member States to take them better into account to fully assess the “value for money” implicit in allocating vast amount of resources to such projects.
Recommendation Based on the findings of our study, we recommend the Commission to examine the possibility of changing current regulations to improve the “evaluability” of major transport infrastructure projects that benefit from EU funding to better assess their “value for money” ex post. Such improvements would require, for example:
a clear definition of “value for money”, intended as way to assess the opportunity cost of the public funds used in the project, reaching from control of costs through to delivery of outcomes, including system-wide impacts and more focused outputs;
requirements to set up a monitoring framework already in the design phase of the project identifying landmarks for data collection and evaluation, including Level 3 cost data;
set up fixed intervals to monitor ramp-up and distribution effects and carry out ex post evaluations, trying to balance the need for project impacts to materialise and maintaining institutional memory. For example, the first project evaluation could take place after three to five years after project opening;
systematic data collection efforts, perhaps over a large number of projects to try and identify biases in the ex ante analysis, leading perhaps to the introduction of measures to mitigate these biases and thus improving the appraisal of future projects;
Assessment of potential role of alternative project appraisal techniques, such as multi-criteria analysis for ex post evaluation;
7.3.3 Provide guidance and organize training As part of this study, the services of the Commission organised a workshop with national representatives of EU Member States in Brussels entitled Ex Post Evaluation of Cohesion Fund (including former ISPA) 2000-2006. It was held in Brussels on the 3rd of February 2011 with the aim of sharing experience about the use and implementation of the Cohesion Fund instrument and exchanging views and reactions about the initial findings of our study. A common message shared
Conclusions and recommendations
June 2011 | Frontier Economics, Atkins, ITS 121
by all national representatives is that project promoters across the EU would benefit from an effective dissemination of cost-benefit analysis good practice beyond the existence of EC guidelines.
Recommendation Based on the findings of the study and the feedback obtained in the workshop carried out with the EU Member States in Brussels on February 2011, we recommend that the Commission creates or facilitates ways to provide guidance and train public sector officials on the use of ex ante CBA analysis and ex post project “evaluability” design. This could be achieved by, for example:
sharing examples of good practice amongst practitioners;
offering qualified and targeted training on specific aspects of the CBA analysis; and,
creating opportunities for project promoters and potential consultants to learn and share their knowledge. Publication of high quality examples of CBA (case studies) and guidance and training on good practice would lead to improvements in the applications of ex ante cost-benefit analysis, addressing the weaknesses that our research has identified such as, for example, limited scope of analysis, parameters values and their reliability, demand modelling and forecast, analysis of generation and distribution of network effect, sensitivity and risk analysis and the reporting of the base and logic of assumptions underpinning the appraisal.
Conclusions and recommendations
June 2011 | Frontier Economics, Atkins, ITS 123
Annexe 1: Project summaries
This Annexe provides a one-page summary for each of the projects considered in this evaluation. The purpose of the summary is the give a brief overview of each project‟s characteristics. Specifically, they provide the following information. description; cost and share of EU funding; main objective; results of economic ex post evaluation; main sources of benefits; and wider socio-economic impacts The summaries facilitate comparisons across projects. However, we note that a straight comparison may not always be appropriate, as each project has specific characteristics that may not be fully captured by the standard CBA methodology. These characteristics may explain some of the differences between projects.
Annexe 1: Project summaries
124 Frontier Economics, Atkins, ITS | June 2011
High Speed railway Madrid – Barcelona in Spain
Girona
Zaragoza Lleida Barcelona
Tarragona
Madrid
Subprojects considered in this analysis Madrid - Lleida (1999ES16CPT001) Lleida-Olérdola (2003ES16CPT004) Lleida-Martorell 2000ES16CPT005 Martorell-Barcelona. 2004ES16CPT003 Accesos ferroviarios de estación de Zaragoza 2000ES16CPT003 Subtramos XI-A and XI-B entre Lleida y Martorell 2001ES16CPT009 Gelida-Sant Llorenç d'Hortons-Sant Esteve Sesrovires 2003ES16CPT010 Subtramos IX-A Lleida y Martorell 2001ES16CPT005 Subtramos IX-B Lleida y Martorell 2001ES16CPT006 Subtramos XI-C Lleida y Martorell 2001ES16CPT010 Sant Esteve Sesrovires-Martorell - Río Llobregat 2003ES16CPT026 Río Llobregat - Costa Blanca - Conexión Vallés 2003ES16CPT027
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 125
Summary of evaluation
Project characteristics The LAV (Línea de Alta Velocidad) Madrid – Barcelona – French border is a high speed railway line connecting Madrid to the French border via Barcelona. The Madrid – Barcelona section is already operational. The remaining section along the coast to the French border is currently under construction; therefore we have therefore excluded it from this evaluation. The line connects the two most densely populated urban areas in Spain, Madrid and Barcelona, with intermediate connections in Guadalajara, Calatayud Zaragoza, Lleida, Tarragona (station in Camp de Tarragona, between Tarragona and Reus). The LAV is part of the TEN-T Priority Project 3 (high-speed railway axis of south-west Europe), whose main objective is to provide high-speed rail connections between the Iberian Peninsula (Portugal and Spain) and the rest of Europe. The High speed railway between Madrid and Barcelona covers 621 kilometres, and it was developed in three stages: Section Madrid – Lleida: opened in October 2003, and covering around 442 km of high speed rail. Section Lleida – Tarragona: in operation since December 2006, adding 78 km of railway line to the previous section. Section Tarragona – Barcelona (Sants station): operational since February 2008, with an additional length of 100 km. The LAV is still under construction in the section Barcelona to Figueres in Spain, with 132 km expected to be completed in 2012. The section between Figueres and Perpignan in France was completed in 2008. Project cost and EU funding The project, as defined in the TORs, comprises 12 subprojects that account for the construction of 72 km of rail bed and the installation of 610 km of railway tracks. The total cost of the 12 subprojects was around €1,719 million, of which €1,442 million was eligible for funding. The total Cohesion Fund contribution for these subprojects in the period 2000-2006 was around €1,042 million, equal to 72.25% of the eligible project costs. Project main objective Enable rail connections between the Iberian Peninsula and the rest of Europe without the need for reloading, which is needed as a result of the historic gauge difference between the rail networks in Spain/Portugal and the rest of Europe. The new railway line implies that the two biggest cities in Spain are now linked by train in two hours and half hours, creating an alternative to the busiest air corridor in the world. Results of ex-post evaluation NPV (€m) ERR (%) BCR Low case -2,736 2.6% 0.6 High case -1,948 3.7% 0.7 Main impacts Most of the benefits from this project come from passengers‟ time saving and vehicle operating cost savings. There is also benefit from positive externalities such as the reduction in noise, pollution, accidents and climate change effects Wider socio-economic impacts Most studies carried out on wider socio-economic impacts conclude the HST is not a sufficient condition to cause major transformations in the cities and regions connected by it. The HST only facilitates socio-economic changes that may be already underway. Access to HST services may provide important competitive advantages to those cities that are on the HST network compared with those that are not in the network and have therefore less train services. According to the economic literature and experience in other European countries where HST services have been introduced before suggest the main wider economic impacts of HST infrastructure and services are impact on mobility and accessibility, socio-economic structures, urban image and spatial effects. The same applies to the cities connected by the Madrid-Barcelona HST line, particularly Zaragoza and Lleida.
Annexe 1: Project summaries
126 Frontier Economics, Atkins, ITS | June 2011
A2 motorway in Poland
Subprojects considered in this analysis Construction of A2 Motorway. Section Konin-Strykow; subsection Emilia Strykow (2003PL16PPT020) Construction of A2 Motorway. Section Konin-Strykow; subsection Konin Emilia (2004PL16PPT001)
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 127
Summary of evaluation
Project characteristics This highway infrastructure project consisted of the construction of a two lane motorway (A2) between Konin and Strykow in central Poland. The project was implemented in two phases: Phase One (June 2006) – Konin to Emilia (85.8km); and Phase Two (July 2006) - Emilia to Strykow (18.1km). The A2 (European Route 30) is a major east west corridor providing a strategic road connection between Berlin, Central Poland and Belarus. The new section of motorway provides the strategic connectivity previously provided by the single carriageway R72 between Konin and Strykow. The project passes through a Special Economic Zone (SEZ) and as such is a regarded as one of the key enablers to economic growth and regeneration in this central area of Poland. The national objectives of the project were to ensure the effective road connections needed to support intensification of trade; improve the accessibility of vital economic urban centres; improve road safety and eliminate the social and economic costs of road accidents and support regional development. Project cost and EU funding The total aggregate cost for this project was €406m compared to the forecast cost of €401m. The total contribution made from EU Cohesion funds was approximately €325m, with the European Investment Bank contributing further funds up to the 90% maximum funding support permitted. Project main objective Prior to the construction of the A2, the main connecting route between the major settlements in the area was the single lane R72 (linking Strykow and Lowicz) resulting in traffic congestion and poor journey time reliability and high risk of accidents. The main objectives of the A2 project were therefore to improve connectivity at local and regional level, thus reducing journey times, relieving traffic congestion, and improving safety Results of ex-post evaluation NPV (€m) ERR (%) BCR Low case 523 18.2% 3.0 High case 900 22.8% 4.4 Main impacts The main source of benefits identified in the ex-post evaluation was from journey times and safety benefits. Journey time savings of 1h 15 minutes were achieved for the section between Konin and Stykow, this is substantially higher than the 50 minute saving forecast. Our traffic analysis has shown that a vast majority of the traffic now using the A2 has re- assigned from the R2 (north of the project) and R72 (south of the project). There has been an observed reduction in accidents of 10% on the main strategic routes affected by the project (A2; R2; R72; and R14), this compares to a forecast reduction of 13%. Wider socio-economic impacts The project was implemented against a backdrop of rapid economic growth in Poland. Therefore, it is difficult to establish a causal link between the A2 project and specific impacts. Nonetheless, it is clear from our consultation with stakeholders that the project (and future proposals to extend the A2 to Warsaw and proposals for the north south A1 route) is regarded as a key enabler of economic growth in central Poland. Key benefits of the project identified by stakeholders include: faster and more reliable connectivity between Poland and neighbouring countries; increased Foreign Direct Investment; quality of life improvements through reduced traffic on the bypassed routes; increased land values along the corridor; supported tourism access. Adverse impacts reported include increased nuisance caused by poor noise and air quality on the existing R14 (at the eastern end of the A2 motorway).
Annexe 1: Project summaries
128 Frontier Economics, Atkins, ITS | June 2011
Algarve railway in Portugal
Lisbon Pinhal Novo
Coina
PK 94
Ermidas Sines
Funcheira
Tunes
Faro
Subprojects considered in this analysis Section Pinhal Novo – Poceirão – Pinheiro – Ermidas (excluding section from Pinheiro to Km 94) and Section Ermidas to Porto de Sines (2000PT16CPT003) Section Coina – Pinhal Novo (2000PT16CPT012) Section Pinheiro to Km 94 (2001PT16CPT001) Section Ermidas-Faro and Section Funcheira-Ourique (2001PT16CPT003)
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 129
Summary of evaluation
Project characteristics The Lisbon – Algarve railway project covers a significant part of the overall modernisation of the Algarve railway line (covering the Linha do Sul and Linha do Algarve). The part included in the current study covers the segment from Coina (near Lisbon) to Faro, including a branch to Porto de Sines. The project is part of the TEN-T Priority Project 8 (Multimodal Axis Portugal/Spain – Rest of Europe). The project involved, among other works, the duplication of the track in some sections of the line, the electrification of parts of the line which were previously non-electrified, the construction of various new stations, and the installation of a new automatic control system to increase maximum speed in various sections of the line. Project cost and EU funding The total aggregate value for the project was €419,026,183, while the total amount eligible for funding was €405,043,302 or 96 per cent of the total. The total Cohesion Fund contribution in the period 2000-2006 was €323,486,946, equal to about 80% of the eligible project costs. Project main objective The main objective of this project was the reduction of journey times between Lisbon and Faro to about 3 hours. This has been achieved by increasing in the maximum speed on the line to 220km/h, thanks to additional double track sections, the installation of a new automatic control system and the complete electrification of the line. Results of ex-post evaluation NPV (€m) ERR (%) BCR Low case 48 6.7% 1.1 High case 79 7.5% 1.2 Main impacts As this project involved the upgrading of an existing railway line, rather than the construction of a new route, most of its benefits actually come from time and maintenance savings, as well as from positive externalities such as the reduction in noise, pollution, accidents and climate change effects. Wider socio-economic impacts The lack of any form of impact monitoring since the opening, makes identifying a causal link between the project and any wider effect very problematic. In this kind of situations, it is difficult to separate the impacts of the project from the impact of other ongoing changes in the economy. Nonetheless, there is some evidence that per capita income might have improved in the regions connected by the railway line. It is however problematic to attribute these changes just to this project. Following the Algarve railway project three additional routes are being considered: Lisbon – Porto, Porto – Vigo (Spain), and Lisbon – Madrid. The latter will use part of the Lisbon – Algarve railway track, specifically the Lisbon – Évora section.
Annexe 1: Project summaries
130 Frontier Economics, Atkins, ITS | June 2011
A23 motorway in Spain
Nueno
Huesca
Calamocha
Monreal del Campo
Santa Eulalia
Teruel
Subprojects considered in this analysis Section Huesca to Nueno (1999ES16CPT002) Monreal del Campo to Calamocha (Teruel) (1999ES16CPT003) Santa Eulalia del Campo to Monreal del Campo (Teruel) (1999ES16CPT004) Teruel (N) to Santa Eulalia del Campo (Teruel) (2003ES16CPT029)
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 131
Summary of evaluation
Project characteristics The A23 motorway in Spain (also called Autovía Mudéjar) is a high capacity road connecting Sagunto, on the Mediterranean coast just north of Valencia, and the Somport road tunnel. The tunnel connects Spain and France through the central Pyrenees. It was opened in 2003. Although the A23 is not part of any TEN-T corridor, it belongs to the European route E07 connecting Pau in France to Zaragoza in Spain. Currently, the A23 motorway is completed between Sagunto and Nueno. The remaining sections between Nueno and the Somport tunnel are either under construction or in advanced planning stages. At this point, it is unclear when the A23 motorway will be completed. Recent budget cuts announced by the Spanish government in July 2010 are likely to cause delays. The project under analysis comprises four subprojects, namely one 11.5 km-long segment between Huesca and Nueno, and three adjacent segments between Teruel Norte and Calamocha, for a total of 63.5 km. The two sections are 178 km apart. All four segments are located in Aragon, in north-east Spain, bordering with France to the north, in the Pyrenees. The capital city of Aragón is Zaragoza, the fifth largest Spanish city. Aragón has three provinces: Zaragoza, Huesca and Teruel. The A23 sections considered in this analysis are located in the provinces of Huesca and Terual. Both provinces cover a total extension of 30,000 km2 (about the size of Belgium) and have a total population of 370,000 people. Project cost and EU funding The total aggregate value for the project was around €205 millions and the total Cohesion Fund contribution in the period 2000-2006 accounted for 85% of the eligible project costs. Project main objective The main objective of the A23 is to facilitate the freight traffic between the Levante (Eastern Spain) region and Aragón, in Spain, and France. The new road was designed to become the natural route through the central Pyrenees, adding capacity to the existing wide road connections between Spain and France in both extremes of the Pyrenees, Irun-Hendaya to the West and La Jonquera to the East. Results of ex-post evaluation NPV (€m) ERR (%) BCR Low case -27.7 4.6% 0.9 High case 27.6 6.3% 1.1 Main impacts Most of the benefits of the project come from time savings and improvement in safety conditions. However, the new road implied higher vehicle operating costs and higher fuel consumption. Wider socio-economic impacts We note that the subprojects under analysis in this evaluation are relatively minor compared to the whole of the A23. It is therefore difficult to establish causality links between these investments and any wider impact. Moreover, the A23 is one of the many interventions that affected the region during the same period. However, for the purpose of this evaluation, we managed to isolate the impacts that are more specific to the segments considered. In relation to the three subprojects located in the province of Teruel, we obtained information about two new infrastructure projects (an industrial park and an airport) developed in the area since the opening of the first road segments. In relation to the subproject Huesca – Nueno, new infrastructures (an industrial park, a technology centre, and new university buildings) have been developed along the A23 near Huesca since the opening of the new road.
Annexe 1: Project summaries
132 Frontier Economics, Atkins, ITS | June 2011
Agiou Konstantinou bypass in Greece
Subprojects considered in this analysis PATHE section: AGIOS KONSTANTINOS by-pass - KAMENA VOURLA by-pass (2000GR16CPT007)
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 133
Summary of evaluation
Project characteristics The dual carriageway Agiou Konstantinou Bypass opened to traffic in October 2008 forms part of a wider package of improvements (most which have already been implemented) to modernise the PATHE (Patras, Athens, Thessaloniki, Evzoni) transport axis. This corridor also forms the European E75 route which is part of the TEN-T Priority Motorway Axis 7 from Igoumenister to Budapest. The strategic objectives of the improved route which bypasses a previously congested single carriageway route through the coastal towns of Kamena Vourla and Agiou Konstantinou Bypass were to: improve average speeds and journey times in general, road safety; improve highway connections between Greece and other parts of the EU and reduce load to capacity ratio to raise the level of service to standards achieved on strategic routes elsewhere in the EU and improve economic competiveness in Greece through reduced journey times.
Project cost and EU funding The total aggregate cost for this project was €317.8m, of which €117m (37%) was funded through EU Cohesion Funds. Project main objective The project was required because the road through Agios Konstantinos ran between the sea and the town. Therefore little opportunity existed to undertake any online improvements to the existing road. The narrow cross section of the road did not allow safe overtaking opportunities and severe congestion events were commonplace particularly in the summer months. The main objectives of the project were to reduce journey times, increase capacity and improve safety Results of ex-post evaluation NPV (€m) ERR (%) BCR Low case 233 12.6% 2.1 High case 283 13.4% 2.4 Main impacts The main source of benefits identified in the ex-post evaluation was from journey times and safety benefits. Journey time savings of 9mins and 13mins have been achieved for the old and new road respectively, this is marginally higher than those forecast in the CBA. The saving is partially attributable to speed violations not considered in the ex-ante CBA. A small reduction in accidents has been observed following project opening. Although representing a small proportion of the overall monetary benefit, toll revenues have been generated from the project, despite the slightly lower than forecast traffic volumes. Wider socio-economic impacts The project was implemented against a backdrop of rapid economic growth in Greece. Therefore, it is difficult to establish a causal link between the bypass and other wider impacts. That said, stakeholders felt that the project had helped unlock the following benefits: removal of traffic from bypassed settlements to enable urban re-modelling; facilitate drainage works to address historical flooding issues; improved air quality and noise conditions; and enhanced ferry port access. Collectively such impacts are helping Agiou Konstantinos to develop as a tourist location.
Annexe 1: Project summaries
134 Frontier Economics, Atkins, ITS | June 2011
M1 northern motorway in Ireland
Balbriggan Bypass
Lissenhall Interchange
Airport Interchange
Dublin
Subprojects considered in this analysis Section Cloghran (Dublin Airport) to Lissenhall (200016CPT002) Section Lissenhall to Balbriggan (200016CPT003)
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 135
Summary of evaluation
Project characteristics This project covers the construction of two sections of the M1 motorway in Ireland. The first section runs from Cloghran to Lissenhall and the second section joins this road with the Balbriggan Bypass further north. A separate application was submitted for each of the two sections (Cloghran to Lissenhall and Lissenhall to Balbriggan Bypass). The M1 Motorway is part of the TEN-T Priority Axis 13 (Ireland – United Kingdom – Benelux). The main objective of the investments in this corridor is the reduction in journey times between Ireland, the UK and mainland Europe. The M1 Northern Motorway opened in June 2003. Project cost and EU funding The total aggregate cost for this project was €232m. The Cohesion Fund contributions (€152m) were used initially in the planning and pre-construction phases. This covered initial design, detailed design, site surveys and investigation, the Environmental Impact Assessment, a public consultation process and the preparation of contract documentation. For the next phase Cohesion Fund contributions were used for the construction of earthworks, the pavement, drainage, fencing and nine bridges. Interchanges, side roads and land acquisition were financed by the Irish Government. The eligible cost for co-financing was €173m. Project main objective The construction of the new sections of the M1 were required to upgrade the heavily congested existing N1 road and was seen by the Irish Government as being key to meeting European, national, regional and local objectives. The main objectives of the construction of this motorway was to improve transit times, safety levels and level of service and to provide essential infrastructure support for economic development (particularly industry and tourism) by improving access to Dublin‟s port, airport and the main domestic markets. Results of ex-post evaluation NPV (€m) ERR (%) BCR Low case 3,805 53.0% 17.2 High case 3,905 53.0% 17.6 Main impacts The majority of the benefits from this project come from time savings. This is because the new sections of motorway relieved a very congested area. The project has also delivered vehicle operating costs and safety benefits. Wider socio-economic impacts It is difficult to establish a link between the project and wider socio-economic impacts as the project was implemented during a period of rapid growth in the Irish economy. It is however clear that the project has formed an important component of the general upgrade of the Ireland‟s transport network. The construction of the project was a key factor in facilitating the construction of the second terminal at Dublin Airport. There is also evidence of substantial growth in the towns adjacent to the project – where the congestion levels have dropped as a result of the M1 Northern Motorway.
Annexe 1: Project summaries
136 Frontier Economics, Atkins, ITS | June 2011
Railway Thriassio-Pedio-Eleusina-Korinthos in Greece
Subprojects considered in this analysis Project name: Construction of new Thriasio-Pedio-Elefsis-Corinth railway line (Phase A) (94.09.65.010 – A ) Project name: Construction of Thriasio-Elefsina-Corinth section (Phase B) (1994GR16 CPT110 – B) New Railway Track Thriasio–Elefsina–Korinthos (Phase C) (2003GR16 CPT001) New Railway Korinthos-Kiato (2000GR16CPT003 – D)
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 137
Summary of evaluation
Project characteristics The project covers a rail upgrade between Thriasio (north west of Athens) and Kiato in Greece. The new line has replaced a former single track metric line passing through a number of urban areas. The 112km section runs parallel to the European TEN-T network Motorway Priority Axis 7 (Igoumenista-Patras-Athens-Sofia-Budapest) and forms part of longer term proposals to upgrade the Piraeus-Athens-Patra line. The route is covered by four funding applications for cohesion funding. Three of the sub-projects are related to the section of track running from Thriasio to Korinthos. An additional funding application was made for the section of track between Korinthos and Kiato. Project cost and EU funding The total cost of the project was €508m Of this, the total amount eligible for EC funding was €238m, 47% of the total expenditure. Project main objective The old line between Thriasio and Kiato had poor geometric features, an old infrastructure with different gauge from the rest of Greece, and a lack of modern telecommunications and signalling. The line also passed through a number of built up areas and contained a number of level crossings. As a result, although track speed was generally 90km/h it dropped to as low as 25km/h in places. The objectives of the improvement project were to reduce journey times, increase revenue and decrease operating costs, improve safety, improve connectivity with the wider network and provide employment during construction. Results of ex-post evaluation NPV (€m) ERR (%) BCR Low case 32 6.0% 1.1 High case 258 9.3% 1.8 Main impacts The ex post analysis for this project shows that the project represents value for money having a BCR of greater than one in both high and low demand scenarios. The net present value of benefits are lower than forecast, although this reflects the smaller extent of the project considered in the outturn evaluation. The initial appraisal had considered an extension of the project as far as Patra, which is not yet completed. The main benefits for the project come through travel time savings, reduced maintenance and operating costs, accident and environmental benefits. Wider socio-economic impacts The new line is approximately 3 years into its expected economic life, whilst economic and social impacts often take many years to materialise. In addition in the years since project opening the economic downturn is likely to have offset some of the potential of the project. Consequently the wider impacts of the project are hard to isolate although the following headline messages were drawn from the ex-post analysis: A number of stations on the new line are in peripheral locations relative to their previous town centre positions, as a result they are less accessible compared to the stations located on the old routes; The out of town stations do offer a focal point for development and have been integrated into local town master planning exercises; The removal of trains from the built up areas, and associated level crossings, has offered local congestion benefits and benefits in terms of noise, poor air quality and congestion caused by traffic disruption at level crossings; The true wider economic benefits of the project are unlikely to materialise until the line completes the rail link between Patra port and Athens.
Annexe 1: Project summaries
138 Frontier Economics, Atkins, ITS | June 2011
IX B corridor in Lithuania (including Vilnius southern bypass)
Klaipeda
Kaunas
Vilnius
Subprojects considered in this analysis Widening and strengthening of 32.7 km and strengthening of 135 km of motorway. (2000/LT/16/PT/001) Widening and strengthening of 30.9 km and strengthening of 70.9 km of motorway. Reconstruction or repair of 16 bridges and viaducts and installation of 11 km of wood fencing.( 2002/LT/16/PT/007) Widening and strengthening of 37.8 km and strengthening of 20.1 km of motorway. Rehabilitation of 83.4 km of pavement, construction of 4 km of new road carriageway and implementation of various traffic safety measures (2004/LT/16/CPT/003) Construction of the Vilnius Southern Bypass (2004/LT/16/CPT/008)
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 139
Summary of evaluation
Project characteristics This project covers the modernisation of one of the two main motorway routes in Lithuania. This is part of the IX B transport corridor, 315km of motorway linking the port city of Klaipeda with the capital city of Vilnius, via Kaunas. The route then links Lithuania with other destinations in Eastern Europe. The project also involved the construction of the Vilnius Southern bypass. Project cost and EU funding The total aggregate cost for this project was €154m, while the total amount eligible for funding was €148m, or about 96 per cent of the total. The total ISPA/Cohesion Fund contribution was €121m, equal to about 82% of the eligible project costs. Project main objective The modernisation of the IX B Corridor was required to prepare this motorway, a key element of the Lithuanian transport network, for higher and heavier volumes of commercial traffic expected in the coming years. The main objective for the construction of the Vilnius Southern bypass was to reduce chronic delays caused by transit and local traffic being forced through the historic district of the city Results of ex-post evaluation NPV (€m) ERR (%) BCR Upgrading of the IX B corridor Low case 200 55% 3.3 High case 212 56% 3.4 Vilnius Southern Bypass The Vilnius Southern Bypass was declared completed in 2009. In the same year, the Vilnius Municipality prepared a Final Report to monitor the performance of the project after opening. The following table summarises the results of this analysis, which we have reviewed. Results of evaluation 142 29% n/a Main impacts Given the differences between the two projects, we have carried out the ex post CBA analysis separately for each project. The main benefits from the upgrading of the motorway are originated by savings in vehicle operating costs. The Vilnius Southern Bypass, on the other hand, generates most of its benefits as time savings. These results are in line with the findings of the original ex ante analysis. Wider socio-economic impacts The project was implemented against a backdrop of rapid economic growth. Therefore, it is difficult to establish a causal link between it and specific impacts. Nonetheless, it is clear that the project contributed significantly to the upgrading of the Lithuanian transport network. Together with other projects, it helped set up Free Economic Zones and open new logistic centres. It also contributed to the regeneration of brownfields along the motorway (for example with the set up of new shopping and residential centres).
Annexe 1: Project summaries
140 Frontier Economics, Atkins, ITS | June 2011
Bratislava Rača – Trnava Railway Upgrade in Slovakia
Subprojects considered in this analysis Upgrade of rail track between Bratislava Rača and Šenkvice (2000/SK/16/P/PT/001) Upgrade of rail track between Šenkvice and Cífer and improvements to rail stations Bratislava Rača, Svaty Jur, Pezinok, Šenkvice, Bahon, Cífer and Trnava.( 2001/SK/16/P/PT/003)
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 141
Summary of evaluation
Project characteristics This project covers a rail upgrade between Bratislava Rača and Trnava in Slovakia. The 39km section forms part of the Corridor Va TEN-T rail corridor which runs in Slovakia between Bratislava, Ţilina and Košice, then onto the border with Ukraine. The section of line also forms part of Priority Axis No. 23. The modernization project includes two separate applications for funding, the first including the upgrade of the line between Bratislava Rača to Šenkvice (2000/SK/16/P/PT/001), and the second upgrading the line between Šenkvice and Cífer and all the stations along the section between Bratislava Rača and Trnava (2001/SK/16/P/PT/003). Project cost and EU funding The total cost for the two projects combined was €213m. Of this, the total amount eligible for funding was €163m or about 77% of the total spent. The total ISPA/Cohesion Fund contribution was €89m, equal to about 55% of the eligible project costs and 42% of the total costs. Project main objective The main aims of the project were to improve the speed, comfort and attractiveness of the railway, thus making it more competitive in the transport market. The modernized rail infrastructure would also improve safety and significantly reduce the operating and maintenance costs of the line, whilst meeting international standards (AGTC) of rail track and complying with EU interoperability requirements. Results of ex-post evaluation NPV (€m) ERR (%) BCR Low case 104 9.7% 2.1 High case 215 12.4% 3.2 Main impacts The main benefits of the project come through travel time savings, particularly for passengers but also for freight. A reduction of vehicle maintenance and operating costs, and a number of externality impacts – safety and environment - also contribute toward the overall project benefits. The results are in line with the findings of the original ex ante analysis submitted in the applications for funding for the two projects. Wider socio-economic impacts The project was implemented during a period of rapid economic growth and increasing car ownership. Therefore the passenger numbers on the line have fallen relative to 2005 levels. Moreover, the continued works on other segments of the line have caused delays and cancellations further contributing to the fall in passenger numbers. Additionally, due to the global economic recession in 2008/2009, it is difficult to attribute regional development impacts directly to the project. However, the project has contributed to meeting aspirations of Slovak transport policy to increase accessibility to wider EU markets and is successful in improving communication and trade links between the two major economic centres of Bratislava and Košice. Furthermore, the improved link offers increased accessibility to jobs located in the major urban areas of Bratislava and Trnava.
Annexe 1: Project summaries
142 Frontier Economics, Atkins, ITS | June 2011
M0 Budapest ring road (eastern section) in Hungary
Subprojects considered in this analysis Eastern Section of the M0 Budapest Ring Road between National Road 4 and M3 (2004HU16CPT002)
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 143
Summary of evaluation
Project characteristics The M0 Eastern Sector is the eastern component of the Budapest orbital motorway, whilst the M31 is a linking motorway between it and the M3 (the motorway to the north east of Hungary). The project has important international, national and local dimensions. Budapest sits on the junction of three land based Helsinki corridors, is the focus of the national transport network and is the economic centre and capital of Hungary. Congestion in the city centre, particularly in the vicinity of the Danube is severe. This project therefore performs an important function in facilitating the movement of international and national traffic while also providing some traffic relief to Budapest. It also forms part of the Hungarian national transport strategy. The project comprises of 38.7 km of motorway to the east and northeast of Budapest. This motorway is Dual 2 lanes throughout, though the southern section of it has been constructed with provision for upgrading to Dual 3 lanes at a later date. In accordance with Hungarian law, no vignette or toll charges are payable on the M0 or M31. This is because they form part of an orbital motorway around an urban area. The M0 Eastern Sector opened in September 2008, whilst the M31 opened in July 2010. Project cost and EU funding The total aggregate cost of the project was €367m of which €319m is eligible for Cohesion Fund contributions. The EC will contribute €271m, which represents 85% of the eligible project costs. Project main objective The specific objectives of the project include reductions in travel times and transport costs for this traffic and a reduction in external costs (safety and environment costs) on the existing road network. Results of ex-post evaluation NPV (€m) ERR (%) BCR Central case 974 24.8% 5.6 The ‘central’ scenario reported is that corresponding to the economic, demographic and transport related plans for the Budapest region up until 2020. Taking a conservative approach traffic levels and travel times are then assumed to remain static until the end of the 25 year evaluation period. Main impacts The total benefits for this project are estimated to be €1,187m. This level of benefit is almost entirely driven by travel time savings. Accident savings are about 8% of the value of the time savings. As the project increases average trip distance by routeing traffic around Budapest vehicle operating costs increase causing a disbenefit. These „costs‟ are of the same size as the accident benefits (i.e. have a magnitude of 8% of the time savings) and therefore almost exactly cancel out the accident benefits. Environmental costs and benefits have not been monetised in the CBA. Wider socio-economic impacts The M0 Eastern Sector and M31 are youthful in terms of their age and their expected economic life. The M0 opened 2 years ago and the M31 four months ago. As such their impacts on society and the economy are limited as it takes time for direct transport benefits to feed through into other markets from the transport market. Additionally, the recession has affected the economy throughout the period the M0 has been open. Primarily as a consequence of their youthfulness quality of life benefits/costs (accessibility, air quality and noise disturbance) are those most readily observable. Accessibility benefits have been included in the CBA, though environmental benefits have not. There has, however, been some economic development along the M0 corridor (retail and logistic centres) and it is expected that more will occur, as municipalities are zoning more land in the locality of the M0.
Annexe 1: Project summaries
June 2011 | Frontier Economics, Atkins, ITS 145
Annexe 2: Strengths and weaknesses of ex ante CBA appraisal
In this annexe we draw in particular on the analyses presented in Giorgi et al. (2002) and Pearman et al. (2003), arising from the TRANS-TALK research. This research considered appropriate appraisal methods for European transport infrastructure investment, taking into account the views of a wide range of experts and stakeholders through a series of workshops at which specially- commissioned papers were presented and discussed. A tricky issue, and therefore a weakness of CBA is assessing the distributional impact of projects, whether by social group or across geographical space. Assumptions are required concerning who the ultimate beneficiaries are across groups and in space. Even if these can be made, practical issues such as partitioning trip matrices to establish benefits by zone then arise such as whether benefits stick to origin or destination zones. Knowledge of income and social characteristics at transport zone level is often problematic. Another potential strength of CBA is that much of it relies on what could be objective data. We use the qualifications „potential strength‟ and „could be objective‟ because the objectivity of ex ante CBA has been challenged several times over the history of CBA, most effectively by the recent literature on optimism bias and „misinformation‟ (Flyvbjerg, 2007 – see Section 3.4 below). A short compendium of sources of error and bias in transport appraisal is provided by Mackie and Preston (1998). These have not changed greatly over the intervening years! A study by Bain (2009) shows that while traffic forecasts on free roads are subject to error, they are unbiased; however traffic model forecasts on toll roads are on average overestimated by 30%. It is now widely recognised that the objectivity of CBA should not be taken for granted, and that it depends in particular on the institutional arrangements within which CBA is conducted. For example, demand forecasts play a pivotal role in both estimating user benefits and projecting revenues. Many funding bodies, including IFIs and various governments, have arrangements in place to audit the demand forecasts associated with a proposed project. Typically this will include an internal review of the base data, forecasting assumptions and methods or an external assessment provided by a third party. Similarly, the appraisal parameters such as values of time, asset lives and discount rates can be checked against suitable benchmarks such as evidence from the same territory, or from comparable territories. Van Wee (2007: 623) suggests that some kind of ethics qualification for forecasting specialists should be introduced to address the observed tendency for forecasts to become biased, favouring the project‟s NPV, BCR or IRR as a result. A more common approach (Pearman et al, 2003; Flyvbjerg, 2007) is to try to design the appraisal and decision-making process in such a way that the
Annexe 2: Strengths and weaknesses of ex ante CBA appraisal
146 Frontier Economics, Atkins, ITS | June 2011
incentives faced by appraisers are reasonably compatible with the public interest. One way of doing this is to establish the funding body as a „gatekeeper‟, maintaining the appraisal rules and methodology, and ensuring that the appraisals carried out by project promoters are sufficiently audited. Another tool, adopted in England, is to require an „optimism bias adjustment‟ to be included in all CBAs during each stage of project development – see Table 27 below. This is a factor which is applied to costs (and potentially to the revenues) reflecting the typical outturn performance of projects of this type. The factors are based on the evaluation data of past projects. As these standard adjustments tend to be high, project promoters have a strong incentive to make a case for a lower optimism bias adjustment by carrying out a full risk analysis (see below).
Table 27. Optimism Bias Adjustments
Source: DfT (2006b), Table 9
A further potential strength of CBA is its ability to address project risks and to present them to decision-makers in an understandable way. This is possible thanks to the development of Quantitative Risk Analysis (QRA) tools which are available as spreadsheet add-ins (e.g. @RISK and CrystalBall). Using these tools, the deterministic inputs to the model – construction cost, for example – can be replaced by their probability distributions. A simulation can then be run to estimate the probability distribution of outputs such as the NPV, the benefit-to- cost-ratio and the IRR. These results can then be used to provide the decision- maker with a better understanding of the likelihood of different outcomes from the project. In particular, the decision-maker can clearly see the % likelihood of NPV falling below the acceptable threshold of zero. The decision-maker can also see the % likelihood of certain value for money thresholds (as in Table 9) being reached. Again, this is a potential strength as it requires:
Annexe 2: Strengths and weaknesses of ex ante CBA appraisal
June 2011 | Frontier Economics, Atkins, ITS 147
analysts to extend the CBA using QRA tools. The funding bodies can incentivise this practice by requiring it in their appraisal guidance, as an increasing number do (this includes most IFIs). The current DG REGIO CBA guidelines require this type of approach within the Risk Assessment (DG REGIO, 2008);
expert knowledge at the project level to identify correctly the probability distributions of input variables. This is something which might be easier to obtain ex ante, whilst the project knowledge is current, and more demanding ex post, after the project „team‟ has dissipated. Turning to weaknesses, we also need to recognise that ex ante CBA has some outstanding issues, which it is important to discuss. We will address what the literature suggests are two possibly significant weaknesses:
the treatment of economic performance impacts, for example on GDP and employment; and,
comprehensiveness of CBA in relation to the full set of significant impacts, including environmental pollution. We will also discuss a set of minor issues which can be viewed more as requirements for „good practice‟. Improving economic performance is a key motivation for investing in European transport infrastructure (EC DG REGIO, 2008: 13; EC DG TREN, 2001: 6). Measures of economic performance which are of interest to policymakers typically include GDP and employment, as well as wider welfare measures. Furthermore, policymakers are concerned with the spatial patterns of GDP and employment, not simply the aggregate EU or even national-level impacts. The IASON project considered and further developed the available tools for this type of analysis (Tavasszy et al. 2004). Among the tools used were: analysis of spatial economic performance using a combination of the SCENES transport model with either the SASI or CGEurope models; analysis of aggregate economic performance (without spatial patterns) using a combination of SCENES and the macroeconomic model E3ME; and for comparison, a conventional ex ante CBA using the SCENES model. The scenarios tested in IASON included a Business-As-Usual baseline and a policy scenario including fast implementation of the TEN and TINA priority projects (Tavasszy et al. 2004: 31). The main issue with the conventional ex ante CBA in this context is that user benefits are measured in the transport sector, without any explicit modelling of the wider economy – theoretically this approach can be defended if transport- using sectors are all perfectly competitive, but in the presence of imperfect competition the transport benefits become only an approximation to the total
Annexe 2: Strengths and weaknesses of ex ante CBA appraisal
148 Frontier Economics, Atkins, ITS | June 2011
economic benefits. For decision-making purposes, it obviously matters how wide the divergence is between total economic benefits and the narrower measure in the transport sector: early theoretical work suggested the difference could be +30% in the UK (Venables and Gasiorek, 1998), an early critique indicated +3% (Newbery, 1998); and the UK SACTRA committee concluded that +6 to 12% was a plausible range (SACTRA, 1999). The CGEurope model is a leading example of the genre of spatial computable general equilibrium (SCGE) models, applied to EU transport infrastructure. These models can be characterised as follows:
an equilibrium model of interregional trade and location based in microeconomics;
the model includes the labour market and product markets21 (industry, services), as well as transport;
competition between firms is modelled as monopolistic competition (Dixit-Stiglitz) which allows for cross-hauls of related but differentiated products; and,
spatial clusters of activity stimulate productivity (the New Economic Geography). The SASI model is similar in purpose, although theoretically very different. It lacks the general equilibrium property and does not predict welfare changes, only GDP. The E3ME model is a multi-sectoral macro-economic model without a detailed spatial representation. The key results emerging from the tests with CGEurope in IASON were that:
the ratio of total economic benefits versus benefits measured in a conventional ex ante CBA was found to be 1.17 on average – this can be compared with the theoretical result of 1.06 to 1.12 for the UK above – hence there appear to be some additional benefits from the SCGE approach but the difference is small;
however the relationship between total economic benefit and the benefits measured in a conventional CBA was found to be less stable and predictable in more peripheral areas of the EU with less well-developed transport networks;
the rate of return on the TEN+TINA priority investments was predicted to be approximately 3-5% on capital expenditure (lower than some earlier estimates, which suggested 11% – Roy, 1996; EC, 1997);
an important caveat is that these figures are based on changes to freight and business travel – a wider welfare measure including non-working time savings, net of accidents and environmental pollution (all absent from the SCGE model) would require further model development;
the spatial pattern of welfare changes can be determined from the SCGE model – these results suggest that the TEN+TINA investments have reletively strong
21 Potentially, the land market could be included too.
Annexe 2: Strengths and weaknesses of ex ante CBA appraisal
June 2011 | Frontier Economics, Atkins, ITS 149
distributive effects in the European economy, affecting in particular the East-West growth balance and stimulating cohesion;
the impact of the TEN+TINA priority projects as a set was found to be consistent with the sum of their impacts as individual projects (i.e.there was no evidence of subadditivity or superadditivity overall); and,
on a modal point, high speed rail projects appear to be more effective in promoting regional economic activity than conventional rail, and rail projects appear to be more effective than road projects (Tavasszy et al, 2004: 6). In conclusion, IASON recommended that for major projects and policies, a good quality conventional ex ante CBA is vital. A wider economy model linked to the transport model offers a way forward in modelling the total economic benefit and the spatial GDP and welfare changes. However, caution is needed, since the land and labour markets – which are notoriously imperfect – are not yet fully incorporated into SCGE models. Nor are SCGE models as comprehensive in their coverage of impacts as the conventional ex ante CBA. Since IASON reported, there has been some further research into „wider economic benefits‟ in the UK. This has taken a rather different approach, not dependent upon a SCGE model but instead using evidence on productivity elasticities with respect to „effective density‟ of employment (Graham, 2005) to predict the agglomeration impacts of transport investment. Effective density is determined by generalised cost of separation, not only physical distance, hence transport improvements increase effective density. The impact on user benefits appears to be: a 13% increase on average due to agglomeration effects; a 3% increase on average due to the imperfect competition effect; and a 5% increase due to increased tax revenue from the shift to more productive jobs. Overall, a 22% increase in benefits on average across a sample of road, rail and public transport projects. The increase has been found to be larger for major infrastructure projects such as Crossrail in London (+56%) (DfT, 2005). It is not realistic to suggest these results could be transferred to the countries under study in this report, without significant new research in those countries to lay the groundwork. However, it does seem broadly consistent with IASON that when an area is poorly connected into the European network, and a project is proposed to improve that connection, the size and even the sign of the additional benefits (vs conventional CBA) is influenced by the economic competitive position of that area – if the area has comparative advantages as a business location it may experience larger additional benefits, or if the area has no comparative advantage it may suffer an outflow of economic activity to other centres (the „two-way road‟ effect).
Annexe 2: Strengths and weaknesses of ex ante CBA appraisal
150 Frontier Economics, Atkins, ITS | June 2011
Figure 18. Spatial pattern of economic impact (TEN+TINA priority projects)
Source: Tavasszy et al. (2004), Figure 43
Another potential weakness of conventional ex ante CBA is that a comprehensive statement of the costs and benefits of transport infrastructure investment is not usually achieved. Whether this matters is an issue we will consider shortly. First, consider three particular examples of what is omitted. HEATCO identified that in EU countries the main focus of the environmental part of CBA is on: noise; air pollution; and global warming (Odgaard, Kelly and Laird, 2005). Very few countries regularly value other environmental impacts.
Annexe 2: Strengths and weaknesses of ex ante CBA appraisal
June 2011 | Frontier Economics, Atkins, ITS 151
Figure 19. Inclusion of Other Environmental Impacts in Ex Ante CBA
Source: Odgaard, Kelly and Laird (2005), Figure 7.4. Note: QM=Quantitative Methods; QA/NC=Qualitative Assessment.
Furthermore, there is a limitation on the scope of the CBA methods for noise and local air pollution in many countries (noting that only half of the surveyed countries included these impacts at the time of the HEATCO review). Many of the values in use are based on willingness-to-pay for noise reductions or air quality improvements at home, however other locations are not covered – including workplaces, schools, shops, main streets and recreational land. Finally we should note that Laird et al. (2009) have identified significant option value and non-use value for rail projects, over and above the expected use value usually measured in CBA. These option values are not yet found in regular CBAs for infrastructure projects. Together, these represent a set of omissions which almost certainly mean that ex ante CBA is not a complete representation of the costs and benefits for transport infrastructure projects. This could be a concern if left unmitigated. However, it is standard practice across the EU to mitigate this issue by setting CBA alongside (or within) another appraisal framework which is able to accommodate quantitative (non-monetised) or qualitative assessments – see Section 3.3. The role of the CBA is then to monetise a consistent set of impacts, hopefully including the most important impacts in welfare terms, leaving another set of impacts consistently outside the CBA but within the appraisal as a whole. To conclude this brief review of weaknesses identified by the literature, we list the following minor issues which can cause inaccuracies in the results, and which are best treated with caution and consistency in practice:
Annexe 2: Strengths and weaknesses of ex ante CBA appraisal
152 Frontier Economics, Atkins, ITS | June 2011
definitions of the benefit-cost ratio (BCR) are a potential source of confusion and inconsistency – for example, England has just changed its definition (DfT, 2009) by removing indirect tax from the present value of costs (PVC) and including it in the present value of benefits (PVB) – gross versus net BCRs and the treatment of recurring costs are other issues to be managed;
switching between factor cost and market prices as the unit of account can alter the NPV by approximately 20%, so it is recommended that all appraisals be carried out in the same unit of account (HEATCO, 2006, recommends factor cost as the default);
the IRR implicitly assumes reinvestment of the benefits each year – while probably realistic for revenues, this is unrealistic for some unmarketed benefits such as non- working time savings or environmental improvements, hence the use of a Modified IRR by The World Bank (2005); and,
the study area must be widely-enough defined to capture the network effects of the project (Laird, Nellthorp and Mackie, 2005).
Multi-criteria analysis (MCA) as an alternative method to ex ante CBA The main alternative methods to CBA are multi-criteria analysis (MCA) and framework approaches. In practice, many countries seek to get the best of both worlds by using a combination of CBA with either an MCA or a framework to capture a wider set of impacts (Odgaard, Kelly and Laird, 2005). Both MCA and frameworks are structured around the objectives of the project (and potentially the wider policy objectives). Cost- effectiveness appraisal (CEA) indicators may be used at sub-budget level (e.g. savings in killed and seriously injured per unit of cost for local safety projects) but in general, transport projects have multiple outputs and so CEA indicators are not suitable. Multi-criteria methods are described in depth in CLG (2009) and were used to prioritise TEN investments by UNECE (2006), for example. The principal strengths of MCA are:
involvement of decision makers can produce greater buy-in to the ranking generated by the MCA exercise;
flexibility – MCA can be run using greater expert judgement, and less analysis, in less time than CBA;
comprehensiveness – any impact can be included (although the objectivity with which it is included still depends on the quality of data). The principal weaknesses of MCA are:
vulnerability to changes of weights and second-guessing after the MCA exercise – e.g. in the nuclear waste storage example given in CLG (2009), the planning inquiry inspector was able to overturn the ranking some years after the MCA exercise by exercising his own judgement on the weights;
the role of judgement in scoring as well as weighting is good for flexibility, negative for „robustness‟;
Annexe 2: Strengths and weaknesses of ex ante CBA appraisal
June 2011 | Frontier Economics, Atkins, ITS 153
poor treatment of time – asset lives, planning horizon, growth trends and discounting issues tend to be minimised;
evidence base / objectivity of CBA is often stronger in practice. Many countries (e.g. Austria, Czech Republic, Spain) use a CBA nested within an MCA framework in order to get the best of both approaches (Odgaard, Kelly and Laird, 2005, Annex II). „Framework‟ approaches offer:
an objectives-led approach like MCA;
without explicit weights;
the decision-maker is left to weigh up the impacts (e.g. Nellthorp and Mackie, 2000);
arguably this is as far as evidence can take us. The strengths of framework approaches are:
openness to inclusion of any impacts – hence comprehensiveness as MCA;
provides an organising framework for evidence even when scoring and weighting are not desired or not possible – for example, many countries choose to report the Environmental Impact Assessment alongside the CBA. The main weakness of framework approaches is: they do not produce and overall numerical result like BCR or Total Weighted Score, hence the burden of ranking (or other decisions) remains with the decision-maker. Whilst most EU countries are currently using a mix of CBA+MCA or CBA+Framework for transport investment appraisal, there continue to be moves towards more complete valuation in many countries and towards Value for Money measurement using the BCR in some (e.g. OEEI, 2000; DfT, 2009).
Annexe 2: Strengths and weaknesses of ex ante CBA appraisal
June 2011 | Frontier Economics, Atkins, ITS 155
Annexe 3: References
AEAT, ITS and John Bates Services (2004), Evaluation of the Multi-Modal Study Process, Report to DfT. London: DfT. Atkins (2009), Post Opening Project Evaluation Meta-Analysis Summary Report. Report to the Highways Agency, March 2009. Bain R (2009) Toll Road Traffic and Revenue Forecasts: An Interpreter’s Guide. Weald: Rob Bain Bates JJ, Mackie PJ, Nellthorp J and Forster D (2004), Evaluation of the Multi- Modal Study Process – Modelling and Appraisal. Report to DfT. ITS, University of Leeds. Bickel P et al (2006), Deliverable 5: Proposal for Harmonised Guidelines, HEATCO Project (Developing Harmonised European Approaches for Transport Costing and Project Assessment). Stuttgart: IER. Boiteux M, Baumstark L (2001), Transports: choix des investissements et coûts de nuisances. Commissariat général du plan. Paris: La Documentation française. Chapulut JN, Taroux JP and Mange E (2005), „The new ex post evaluation methods for large projects in France‟. In: Proceedings of the European Transport Conference, 3-5 October, Strasbourg. Chevroulet T ed. (2008), Deliverable 4.1 Report on the findings and outcomes of Workshop 1, Workshop Proceedings, Annex. EVA-TREN (Improved Decision- Aid Methods and Tools to Support Evaluation of Investment for Transport and Energy Networks in Europe. Commission for Integrated Transport (CfIT) (2004), High Speed Rail: International comparisons. London: CfIT. Communities and Local Government (CLG) (2009), Multi-criteria analysis: a manual. London: CLG. Court of Auditors (2008), Special Report 1/2008: concerning the procedures for the preliminary examination and evaluation of major investment projects for the 1994-1999 and 2000-2006 programming periods together with the Commission’s replies. Official Journal of the European Union, C81/1. Dasgupta P, Marglin S and Sen A (1972) Guidelines for Project Evaluation, UNIDO. New York: United Nations. Department for Transport (DfT) (2005), Transport, Wider Economic Benefits, and Impacts on GDP, Discussion Paper, July 2005. London: DfT. Department for Transport (2006a), Guidance on Value for Money. London: DfT. Department for Transport (2006b), TAG Unit 3.5.9: The Estimation and Treatment of Scheme Costs. London: DfT.
Annexe 3: References
156 Frontier Economics, Atkins, ITS | June 2011
Department for Transport (2009), NATA Refresh: Appraisal for a Sustainable Transport System. London: DfT. Department for Transport (UK) (2009) The Wider Impacts Sub-Objective. TAG Unit 3.5.14 – draft for consultation. Report dated September 2009. www.webtag.org.uk Eddington, R. (2006) The Eddington Transport Study. Norwich: Her Majesty‟s Stationery Office Eliasson J and Lundberg M (2010) Do cost-benefit analyses influence transport investment decisions? Experiences from the Swedish transport investment plan 2010-2021 European Transport Conference, Glasgow 11-13 October 2010. London: AET Transport European Commission (1997), The likely macroeconomic and employment impact of investments in Trans-European Networks, European Commission Staff Working Paper, SEC(97)10. Directorate General II – Economic and Financial Affairs, Brussels. European Commission Directorate General Regional Policy (DG REGIO) (2008), Guide to Cost Benefit Analysis of Investment Projects. Brussels: DG REGIO. European Conference of Ministers of Transport (2005), National Systems of Transport Infrastructure Planning: Round Table 128. Paris: OECD. European Commission and European Investment Bank (2005), Railway Project Appraisal Guidelines –RAILPAG. Luxembourg: EIB. Expert Group 5 (2010) Final Report of Expert Group 5 (Funding Strategy and Financing Perspectives for the TEN-T), as part of the 2010 TEN-T Policy Review. Flyvbjerg B (2007), „Policy and planning for large infrastructure projects: problems, causes, cures‟, Environment and Planning B: Planning and Design, 34(4), 578-597. Giorgi L, Pearman AD, Tsamboulas D, Reynaud R, Tandon A (eds) (2002), Policy and Project Evaluation in Transport. Aldershot: Ashgate. Graham D (2005), Wider Economic Benefits of Transport Improvements: Link Between Agglomeration and Productivity. Stage 1 Report. London: Department for Transport. Grant-Muller SM, Mackie PJ, Nellthorp J and Pearman AD (2001), „Economic Appraisal of European Transport projects: The state of the art revisited‟, Transport Reviews, 21(2), 237-261. Kjerkreit A, Odeck J and Sandvik KO (2008), „Post opening evaluation of road investment projects in Norway: how correct are the estimated future benefits?‟, In: Proceedings of the European Transport Conference, 6-8 October, Leeuwenhorst Conference Centre, The Netherlands.
Annexe 3: References
June 2011 | Frontier Economics, Atkins, ITS 157
Laird JJ, Nellthorp J and Mackie PJ (2005), „Network effects and total economic impact in transport appraisal‟, Transport Policy, 12(6), 537-544. Laird JJ, Geurs C and Nash CA (2009), „Option and non-use values and rail project appraisal‟, Transport Policy, 16, 173-182. Little, IMD, and Mirrlees JA (1974), Project Appraisal and Planning for Developing Countries. New York: Basic Books Macdonald N (2009), „The POPE programme that has been undertaken for the last 5 years on historic projects‟, In: Proceedings of the European Transport Conference, 6-8 October, Leeuwenhorst Conference Centre, The Netherlands. Mackie PJ, Grant-Muller SM, Nellthorp J and Pearman AD (1999), Socio-economic cost-benefit analysis in the context of project appraisal for developing a Trans-European transport network in accession countries. Report prepared for the TINA Secretariat and approved by The World Bank, EIB and EBRD. Vienna: TINA Secretariat. Mackie PJ and Preston JM (1998), „Twenty-one sources of error and bias in transport project appraisal‟, Transport Policy, 5, 1-7. Mackie PJ, Graham D and Laird J (forthcoming 2011) The Direct and Wider Impacts of Transport Projects- A Review. In (Eds) de Palma A, Lindsey R, Quinet E and Vickerman R, Handbook of Transport Economics, Edward Elgar Mackie PJ and Kelly CE (2007), Transport Appraisal in other countries: lessons for the NATA Refresh, Report to the Department for Transport. Leeds: Institute for Transport Studies. Nellthorp J and Mackie PJ (2000), „The UK Roads Review: a hedonic model of decision-making‟, Transport Policy, 7(2), 127-138. Nellthorp J, Bristow AL and Day B (2007), „Introducing willingness-to-pay for noise changes into transport appraisal – an application of benefit transfer‟, Transport Reviews, 27(3), 327-353. Nilsson JE (1991) Investment Decisions in a Public Bureaucracy. Journal of Transport Economics and Policy 25 (2) pp163-175 Odgaard T, Kelly CE and Laird JJ (2005), Current practice in project appraisal in Europe, Deliverable 1, HEATCO Project (Developing Harmonised European Approaches for Transport Costing and Project Assessment). Stuttgart: IER. Oxera (2005). How should the ex-post evaluation of trunk road schemes be enhanced? Report prepared for Department for Transport. Oxford: Oxera. Pearman AD, Mackie PJ and Nellthorp J (2003), Transport Projects, Programmes and Policies – Evaluation Needs and Capabilities. Aldershot: Ashgate. PIARC (2004), Economic evaluation methods for road projects in PIARC member Countries, Report by the PIARC Technical Committee on Economic and Financial Evaluation. Paris: PIARC.
Annexe 3: References
158 Frontier Economics, Atkins, ITS | June 2011
Quinet E. (2011), Cost-Benefit Indicators and Transport Programming, Fiscal Studies Review, forthcoming Roy R (1996), The macroeconomic effects of the PBKAL. Rotterdam: European Centre for Infrastructure Studies. Short J and Kopp A (2005), „Transport infrastructure: Investment and planning. Policy and research aspects‟, Transport Policy, 12, 360-367. Tavasszy LA, Burgess A and Renes G (2004), Final Publishable Report: Conclusions and recommendations for the assessment of economic impacts of transport projects and policies. IASON (Integrated Appraisal of Spatial economic and Network effects of transport investments and policies) Deliverable D10. Funded by 5th Framework RTD Programme. Delft, Netherlands: TNO Inro. van Wee B (2007), „Large infrastructure projects: a review of the quality of demand forecasts and cost estimations‟, Environment and Planning B: Planning and Design, 34, 611-625. Vickerman RW (2007), „Cost-benefit analysis and large-scale infrastructure projects: state of the art and challenges‟, Environment and Planning B: Planning and Design, 34(4), 598-610. The World Bank (2005), Economic Evaluation Notes, TRN-5 to TRN-26. Washington DC: The World Bank. http://go.worldbank.org/TEN2WQJDN0
Annexe 3: References
June 2011 | Frontier Economics, Atkins, ITS 159
Annexe 4: Ex-post CBA of transport projects – experience and lessons to date
England The Highways Agency (HA) currently evaluates all trunk road projects implemented as part of the Major Projects Programme (projects with a capital investment cost of >£10m). This process is known as POPE (Post Opening Project Evaluation) and has been contracted to Atkins since 2001. The key objectives of POPE are to identify whether the expected impacts of individual highway improvement projects actually occur, to combine the individual results in order to examine the programme as a whole and to use this information to inform the appraisal process and the HA‟s prime objectives. Building on its early origins in the 1980‟s (Project Forecast Monitoring) and in response to recommendations by Oxera in 2005, POPE has evolved to include an evaluation of project performance against each of the sub objectives underpinning the Department for Transport‟s main objectives of Economy, Safety, Accessibility, Integration and Environment. As such the POPE process evaluates both qualitative and quantitative project impacts. POPE provides the key mechanism by which the HA determines: The extent to which investment in major projects offer good value for money; The level of accuracy associated with forecasts of costs and benefits emerging from major projects; The main factors affecting the accuracy of forecasting of project costs and benefits; and How appraisal methods can be improved to help offer a greater degree of accuracy. Figure 20 summarises the cyclical process of ex post evaluation, lessons learnt, improvements to the appraisal process and new ex ante appraisal techniques resulting from POPE.
Annexe 4: Ex-post CBA of transport projects – experience and lessons to date
160 Frontier Economics, Atkins, ITS | June 2011
Figure 20. The POPE Evaluation and Appraisal Cycle
Source: Atkins (2009), Figure 1
The POPE process broadly consists of 4 key components: Stage 1 (Collection of Pre-opening Baseline Data) – Collection of baseline data for key roads within the projects area of influence e.g. Annual Average Daily Traffic; Journey Times; Accident Data; Environment al Baseline reports; ex-ante evaluation documentation; Stage 2 (One Year After Study) – The preparation of a one year after report which sets out a comparison of the forecast and out-turn impacts of a project against the 5 Department for Transport objectives one year after opening; Stage 3 (Five Year After Study) – The preparation of a five year after report expanding on the outcomes of the one year after report also based on the 5 Department for Transport Objectives; Stage 4 (Meta Reporting) – Consolidates the findings of individual project evaluations to identify common themes in the data that can be used to examine the relationship between project predicted and out-turn impacts across each of the Department for Transport Objectives.
Evaluation reports for each project are usually prepared one and five years after opening alongside lessons learnt from the overall programme of evaluation reported in the form of a biennial report, more commonly known as the Meta Report (http://www.highways.gov.uk/evaluation). To date, Atkins has evaluated over 40 major projects as part of this commission, although this is expected to increase to over 65 when the HA publishes its next meta report in 2011. The 2009 meta report provides a robust evidence base by which a number of recommendations were made of direct relevance to ex-ante evaluation process; an illustration of the headline findings summarised as follows:
Annexe 4: Ex-post CBA of transport projects – experience and lessons to date
June 2011 | Frontier Economics, Atkins, ITS 161
future modelling requirements for project appraisal need to be more clearly defined and consistently applied between projects; consideration should be given to a two tier model approach utilising Strategic National or Regional Model estimates of demand feeding into a more project specific modelling platform; risk analysis of traffic forecasts should be undertaken, similar to that usually associated with cost forecasts; for safety analysis, a sufficiently wide area should be considered to ensure that traffic re-assignment effects are adequately considered; more detailed consideration should be given to the application (relevance) of optimism bias in the preparation of project cost forecasts. The POPE of Major Projects programme is supplemented by POPE evaluation for Local Network Management Schemes (LNMS), costing less than £10m to implement. In this category, Atkins have evaluated >400 projects. As with the Major Scheme evaluation programme, the objectives of the LNMS POPE are to identify the extent to which expected benefits and project objectives materialise, whether any particular type of improvement gives best value for money, and whether any aspects of the HA‟s current appraisal (PAR) methodology could be enhanced to become more reflective of outturn impacts. A report is prepared annually; its purpose is analogous to that of the Major Schemes Meta Report. A number of broad lessons can be drawn from the English POPE methodology that should inform the evaluation process for this study – particularly highway project evaluations. Key points of particular relevance are summarised as follows.
Where feasible evaluation should be based on the use of „observed data‟ rather than re-runs of forecasting models for example: out-turn journey time benefits could be measured using a comparison of before and after journey times and network flows;
Where possible the evaluation team should undertake a early review of geographical area over which major traffic flow changes are anticipated to help define the scope of evaluation and approach to data collection;
Collective analysis of individual evaluation outcomes is likely to carry greater statistical robustness than simply drawing conclusions from individual evaluations;
Initial buy–in and continued engagement of key project stakeholders for example the project promoters and local authority representatives throughout the evaluation process for two key reasons:
Annexe 4: Ex-post CBA of transport projects – experience and lessons to date
162 Frontier Economics, Atkins, ITS | June 2011
It facilitates the efficient collection of primary and secondary data needed to support the evaluation; It enhances the quality of the evaluation by drawing upon local knowledge to interpret emerging trends and conclusions within the wider study context.
The scale of evaluation is often proportionate to the level of documentation included/available to support the original ex-ante analysis i.e. a more comprehensive set of documents supporting the ex-ante appraisal is likely to result in a more rigorous evaluation.
France In France, the Internal Transport Act 1982 (Loi d‟Orientation des Transports Intérieurs), introduced a new requirement for an ex post evaluation of the economic and social performance of any major transport infrastructure project, five years after its opening, and to make this report public. „Major‟ infrastructure was defined as costing €82m or more. It was rare in France at that time, and still is, to compare the outturn with forecasts made at the decision-making stage (Chapulut, Tarout and Mange, 2005). The Act makes the funding body responsible for the evaluation, which compromises its independence, however “this solution offers the advantage of making the knowledgeable and resourceful party responsible, and avoid[s] having to create an additional agency or administration”. There are also some safeguards: firstly, the appraisal is submitted to public scrutiny and countervaluations are made; second, the ex post evaluation is designed to check the quality of forecasts after opening. This ex post evaluation must moreover be submitted to the French Civil Engineering General Council (Conseil Général des Ponts et Chaussées- CGPC 2), which answers to the minister; and finally, it must also be made public. Nevertheless, “construction financial authorities have still little incentive for public evaluation of their own action, since there is no penalty for failing to present a report before a given deadline” (Chapulut, Tarout and Mange, 2005: 2). The deadline for production of reports was set at 3-5 years from opening, hence is some cases 15 years from the date of decision. In the early 2000s, it was found that the ex post evaluations were difficult to compare with the ex ante appraisals. In particular, monitoring committees tended to concentrate on environmental issues and any change in economic activity rather than a recalculation of the ex ante CBA. This issue was addressed by a Working Group in 2001, who recommended a dual approach:
an ex post evaluation, using outturn cost and benefit performance, but with the method and the unit values taken from the original ex ante CBA;
Annexe 4: Ex-post CBA of transport projects – experience and lessons to date
June 2011 | Frontier Economics, Atkins, ITS 163
examine how the evaluation changes when conducted to present standards using Boiteux (2001) values – this sometimes had a significant effect, e.g. increasing the appraisal period from 20 to 40 years added 2% to the IRR for TGV Nord. Another issue identified is the difficulty of establishing a reference situation in an ex post report. Expert consensus based on various trend and control corridor data is the favoured method in France. This feedback on the reference situation can be used subsequently in other appraisals. Some ex post studies in France have led to the setting-up of economic observatories to monitor and evaluate the opening of motorways and other major infrastructure. Again the lack of a counterfactual is practical issue. The 2001 Working Group embarked on 62 ex post evaluations, and usefully established that the cost was €100,000 to €500,000 for a High Speed Rail line report, versus €3,000 to €10,000 per km for motorways. Annual expenditure on these reports is €1-2million. According to Chapulut et al., the main lessons learnt from these ex post evaluations were as follows: trivially, arrange for data storage from the ex ante appraisal and preserve any measurement tools used to gather the data; costs can escalate when the construction financing authority is weak, Eurotunnel being a classic example; costs for underground works often exceed expectations; the biggest errors in demand forecasts were found to be due to unexpected competition changes (also observed by Mackie and Preston, 1998); as a result of the feedback from evaluations, more attention is being paid to developing reference scenarios and project scenarios, including infrastructure as well as service descriptions, and specifying quality and prices; also, a well-argued risk analysis, individualised for each project, should now be possible in ex ante appraisals due to the ex post findings; the evaluations enhance our expertise about important phenomena such as induced traffic, the impact on nature, economic development, etc.
Norway Kjerkreit, Odeck and Sandvik (2008) report on the post-opening evaluation of road investment projects in Norway. They focus on the accuracy of the NPVs, traffic forecasts; and divergences in accident costs, investment costs and changes in project designs. Key findings are that:
Annexe 4: Ex-post CBA of transport projects – experience and lessons to date
164 Frontier Economics, Atkins, ITS | June 2011
accounting for the severities of outturn accidents has a great influence on accident costs; national road traffic forecasts used in appraisal have been too coarse to predict demand growth at the level of an individual project (again evidence on project-level induced traffic); generally traffic growth was higher than forecast and NPV higher than predicted; and overall, deviations between forecast and actual impacts varied greatly from project to project.
EU EC DG REGIO has a programme of evaluations in progress (Evaluation Plan 2009). The recent study “Efficiency – Unit costs of major projects”, for example, evaluated the performance of large infrastructure projects and productive investments co-financed by the European Regional Development Fund (ERDF) during 2000-2006. The analysis was based on a representative sample. Work included the calculation of unit costs for key project components and the identification of potential links between project performance and project characteristics. The work also included the development of a database of unit costs and project characteristics, and provided the Commission with benchmarks for use in the appraisal of future project financing requests. The EVA-TREN 6th Framework project (http://www.eva-tren.eu/home.htm) investigated improved decision support tools for investment in transport and energy networks in Europe. One contributor (B. Scholten) cited the following issues in ex post evaluations of Cohesion Fund projects for DG Regio: Many difficulties in establishing ex post ERR (RERR) Lack of data (e.g. on output) Ex ante CBA weak or not existing (or only financial) Ex ante CBAs not always clear in methodology Changes carried out in ex post CBA: Project parameters (investments, output, timing, etc) Project period (according to CBA guide) Methodological changes (shadow rates, inclusion of externalities). Another contributor to EVA-TREN (C. Eberhard) noted that in the European Investment Bank, ex post evaluation is conducted for 15% of projects.
Annexe 4: Ex-post CBA of transport projects – experience and lessons to date
June 2011 | Frontier Economics, Atkins, ITS 165
Performance criteria are: effectiveness; efficiency; and sustainability (Chevroulet ed., 2008).
International Flyvbjerg (2007) found that misinformation about the costs and benefits of major infrastructure projects is widespread. From an international sample of projects, he obtained quantitative evidence of cost overruns (on average +44.7% for rail and +20.4% for road) and demand forecasting inaccuracies (on average +9.5% for road and -51.4% for rail), suggesting a systematic failure to learn from experience and indeed strategic behaviour by project promoters. This points to an important function of ex post evaluation – that is, the function of checking and validating ex ante appraisal. In the light of Flyvbjerg‟s findings, tools such as optimism bias adjustments and improved risk analysis methods (including QRA) are being used more widely used in ex ante CBA, and appraisal processes are being designed to offer suitable incentives to project promoters (see Section 3.2). Finally, Short and Kopp (2005) consider the link between transport infrastructure planning and decision-making. They write that “In trying to find ways to improve planning processes and decision making, it should be helpful to look back at the decisions we have taken to see what lessons we can learn from them. There are, in theory, hundreds of case studies in all modes and in all countries that we could draw on. In practice, the amount of material available from countries and institutions on projects, on their actual performance compared to their expected performance and on their impacts, is very limited” (p364-5). The authors cite the POPE evaluations as an example of a successful method with an unusually wide scope including economic and environmental impacts. They find that evaluation of light rail projects has been problematic: “It is clear that an evaluation framework for light rail needs to include broader aims: improving accessibility to cities and to particular groups, improving the attractiveness of the cities, revitalising areas of the city and attracting users from other modes in achieving reductions in congestion and pollution. Many of these can be evaluated and should be. The success of a light rail system will also depend on the supporting measures that are introduced with it, illustrating the need to evaluate the impacts of the measures that are implemented. But many did not evaluate ex post, and it is therefore not possible to say what the effects of the system have been regeneration or liveability” (p366). They conclude that: “ex post monitoring of projects and policies needs to be systematically introduced and strengthened” (p366).
Annexe 4: Ex-post CBA of transport projects – experience and lessons to date
June 2011 | Frontier Economics, Atkins, ITS 167
Annexe 5: Modelling the macroeconomic impacts of transport investment using CBA output
Introduction The purpose of this annex is to review the basis for which cost benefit analysis output can be used as an input to macroeconomic modelling. It is structured as follows. In the first section evidence on the macroeconomic impact of transport projects is reviewed, whilst in the second section the basic differences in philosophy between macroeconomic models and CBA are outlined. The third section demonstrates the extent to which CBA measures map onto GVA or GDP. The final section concludes by discussing the best method of developing macroeconomic forecasts that are consistent with the CBA.
Evidence on the macroeconomic impact of transport projects The two key macroeconomic variables of interest from a policy perspective are economic output and income. Economic output is measured as either Gross Domestic Product (GDP) or Gross Value Added (GVA), whilst income is measured as either GDP/capita or GVA/capita. Other variables of interest can include population, employment/unemployment, occupation of workers, the number of businesses and the industrial mix of businesses. Given the long standing political relevance of transport‟s impact on economic output and income evidence on it is surprisingly limited and relatively recent in nature. While economic history has plenty to say on revolutionary changes such as the effect of the railways and water transport on the American West, evidence on the incremental effect of improvements to already mature transport systems is both less abundant and less clear in its findings. In part this is due to related econometric challenges associated with attributing causality and defining the counterfactual. Since Aschaeur‟s seminal work22 on the macroeconomic impact of public capital just over 20 years ago a number of other studies have also investigated this link. Lakshmanan23 presents a brief review. In the main, these studies suggest that in recent years public capital investment offers moderate economic benefits to the economy broadly comparable to, though possibly less than, private sector returns. The evidence indicates that economic returns to public capital in
22 ASCHAUER, D.A., 1989. Is public expenditure productive? Journal of Monetary Economics 23, pp177–200. 23 LAKSHMANAN, T.R. (in press) The broader economic consequences of transport infrastructure investments. Journal of Transport Geography (in press)
Annexe 5: Modelling the macroeconomic impacts of transport investment using CBA output
168 Frontier Economics, Atkins, ITS | June 2011
affluent countries were highest in the post WWII period when, for example, the US interstate highway network was developed. Furthermore there is some evidence that rates of return have been lowest in affluent and poor countries, with the highest rates of return evident in middle income developing countries (e.g. Chile, Colombia, South Korea and the Philippines). This may point to the existence of network and threshold effects (see the earlier discussion on network effects §5.1.3). It is in these middle income countries where public investment is completing networks which support the economic transition to a manufacturing and service economy. Contemporary macroeconomic evidence is also contradictory with some studies identifying rates of return for public capital still exceeding private capital in affluent countries, whilst other studies suggesting private capital now offers the best returns. Ex-post studies of specific transport interventions are fraught with the difficulty of identifying the counterfactual. This is understandable given the passage of time between planning studies, project implementation and then the time for the full effects of the project to be felt. Partly as a consequence of this, studies of the impact of transport infrastructure on property prices are the most prevalent. A large body of evidence exists showing the positive relationship between land prices and rail station accessibility. Debrezion, Pels and Rietveld24 in a meta- analysis of 73 observations from property price studies identified that commercial properties within proximity of a rail station are 16.4% more highly valued than the average. The ability to attract higher land rents close to stations implies that such sites are economically more productive. Notwithstanding this general finding some ex-post studies (e.g. of two suburban rail/metro projects in the UK25,26) have found no impact on property prices. This identifies the danger of assuming that all transport investments will have a positive impact on particular economic indicators. In a recent study of the Frankfurt to Cologne high speed rail line Ahlfeldt and Feddersen27 found that the two „small‟ intermediate towns that suddenly experienced a very large change in accessibility to the large economic markets of Cologne and Frankfurt had a one-off increase in GDP of 2.7%. They also find that increasing market access by 1% increases GDP by 0.25%. Banerjee, Duflo
24 DEBREZION G., PELS, E. and RIETVELD, P. The Impact of Railway Stations on Residential and Commercial Property Value: A Meta-analysis. Journal of Real Estate Finance and Economics 35, pp.161– 180. 25 FORREST, D., GLEN, J., and WARD, R. (1996) The Impact of a Light Rail System on the Structure of House Prices, Journal of Transport Economics and Policy, 30 (1) pp.15-29 26 DU, H., and MULLEY, C. (2007) The Short-Term Land Value Impacts of Urban Rail Transit: Quantitative Evidence from Sunderland, UK, Land Use Policy, 24 pp.223-233. 27 AHLFELDT, G. and FEDDERSEN, A. (2009) From periphery to core: economic adjustments to high speed rail. Discussion paper (preliminary and incomplete manuscript). http://www.ieb.ub.edu/aplicacio/fitxers/WS10Ahlfeldt.pdf
Annexe 5: Modelling the macroeconomic impacts of transport investment using CBA output
June 2011 | Frontier Economics, Atkins, ITS 169
and Qian 28 find that in the case of Chinese railways that GDP per capita declines by between 0.12% and 0.18% as distance from the railway increases by 1%. These studies do not say anything about re-distribution of economic activity from elsewhere as a result of the transport investment, but we would expect that some of the observed economic growth to have arisen in this way. There is also a body of evidence on the effect of economic mass and city size on total factor productivity and labour productivity. This research has relevance to the question regarding transport‟s impact on the size of the economy if we make the small, but important, conceptual step that improving the quality of transport alters economic mass and therefore productivity ceteris paribus. Arguably the UK (and Dr Dan Graham in particular) has led the way on research in the area of transport‟s impact on agglomeration and we therefore focus on that in our discussion below. The main points we draw from this work are the following.
The principal difficulties in estimating elasticities to economic mass/density are those of causality and confounding. The issue of causality is that accessibility in large agglomerations and between large agglomerations may be high because that is where demand is the highest, rather than because the higher accessibility has created the agglomeration and its associated productivity. Confounding problems arise as the variables of interest (e.g. access to labour markets and firm to firm linkages) are often heavily and systematically correlated.
Urbanisation economies are larger than localisation economies29. Urbanisation economies relate to proximity to economic mass in general, whilst localisation economies relate to proximity to the economic mass of a particular sector (e.g. textiles). The UK‟s current appraisal guidance is centred around urbanisation elasticities. For localisation, Graham estimates a weighted average productivity elasticity of 0.03 to economic mass for manufacturing sectors and 0.01 for services. For urbanisation the respective elasticities are 0.07 and 0.19. These elasticities do not however take account of the causality issue and may therefore be biased. From a policy perspective the fact that the urbanisation elasticities are larger than the localisation elasticities indicates that it is proximity to economic mass in general that is the key driver to labour productivity.
28 ` BANERJEE, A., E. DUFLO, and N. QIAN (2009) On the road access to transportation infrastructure and economic growth, Discussion paper, MIT, available at http://igov.berkeley.edu/China09papers/On_the_Road_Access_to_Transportation_Infrastructure _and_Economic_growth.pdf 29 GRAHAM, D.J. (2009). Identifying urbanisation and localisation externalities in manufacturing and service industries. Papers in Regional Science, vol. 88, issue 1, pp. 63-84.
Annexe 5: Modelling the macroeconomic impacts of transport investment using CBA output
170 Frontier Economics, Atkins, ITS | June 2011
In more recent work that takes account of causality by using a control function approach, Graham and colleagues estimate lower range of urbanisation elasticities30. This is relevant as it confirms that causality issues can upwardly bias productivity elasticities. The elasticities estimated, which now form part of DfT guidance are: an overall agglomeration effect of 0.04 across all sectors of the economy, 0.02 for manufacturing and consumer services, 0.03 for construction, and 0.08 for business services.
Returns to agglomeration vary not only by sector but with the size of the city (agglomeration)31. With an accessibility measure based on generalised cost of travel Graham found that returns decrease with economic mass for manufacturing, construction, distribution, hotels and catering and IT. For banking, finance and insurance, business services and public services they increase with economic mass. This is relevant is it indicates that the use of an average accessibility to economic mass elasticity may disguise significant local variations – some cities will experience a higher than average increase in productivity, whilst others will experience a lower than average increase (for the same percentage change in economic mass and for the same sectoral mix).
There remain questions around the causal effect of transport infrastructure on productivity32. In his recent empirical work with van Dender, Graham found that variations in productivity between agglomerations maybe entirely attributable to differing qualities of labour between locations.
Differences in basic philosophy between macroeconomic models and CBA Clearly the previous discussion identified that whilst there remain a number of unanswered questions transport infrastructure investments have an impact on an economy both in terms of affecting the location of economic activity and also by growing the economy. On the face of it macroeconomic modelling and CBA are both concerned with the economic impact of a transport investment, therefore it would seem natural that the two forms of economic modelling could be linked.
30 GRAHAM, GIBBONS and MARTIN (2009) Transport investment and the distance decay of agglomeration benefits. Report to the Department for Transport. http://personal.lse.ac.uk/gibbons/Papers/Agglomeration%20and%20Distance%20Decay%20Jan% 202009.pdf 31 GRAHAM D.J. (2007), Variable returns to agglomeration and the effect of road traffic congestion, Journal of Urban Economics, 62 (103), pp.120. 32 GRAHAM and VAN DENDER (2009) Estimating the agglomeration benefits of transport investments: some tests for stability. Discussion paper 2009-32. December 2009. http://www.internationaltransportforum.org/jtrc/DiscussionPapers/DP200932.pdf
Annexe 5: Modelling the macroeconomic impacts of transport investment using CBA output
June 2011 | Frontier Economics, Atkins, ITS 171
This similarity in objectives, however, disguises two fundamentally different perspectives. Firstly, macroeconomic forecasts are concerned with the economic impact at a particular geographic scale – a city, a county, a region, a nation, etc. CBA on the other hand takes a supra-national perspective33. For example, for a motorway upgrade in France, the CBA does not distinguish between the international Czech freight operator transporting produce from Romania to the UK and the local French freight operator ceteris paribus. This distinction is however critical from the perspective of a macroeconomic study. This is because it is important to understand where the benefits will fall in a spatial sense. The situation becomes more complex as the re-distributive impacts of transport projects are large. Economic activity tends to centralise in response to increased accessibility. If for example 90% of the GVA gain in a conurbation is displaced from adjacent sub-regions, the local perspective becomes a significant explanation of disparities with the cost benefit analysis (CBA) approach. Secondly, a CBA is concerned with the impact of the transport investment alone, whilst macroeconomic forecasts are typically interested in „economic potential‟. Many macroeconomic models will therefore assume that other investments - e.g. in housing stock, energy supply, business premises – will follow the CBA investment. In these economic potential type studies transport investment is viewed as the catalyst to a number of structural changes to the economy and is accompanied by a whole series of other unspecified investments that contribute to driving the predicted change in wages (and GVA), population, etc. Such investments would include those into housing stock, business premises, business capital, utilities including telecommunication, human capital (including education) and health services. Land use is also often taken to be relatively unrestricted in these economic potential type studies. Transport investment is only one component of an economic package required to deliver the economic potential. Extent to which CBA measures map onto GVA or GDP If we take it that the primary interest of macroeconomic models is predictions of changes in GDP or GVA then it is interesting to know how such changes are related to welfare benefits. Unfortunately this comparison indicates that whilst there are some similarities the CBA and GVA metrics do not map particularly well and it is not possible to make generalisations about the relationship between welfare benefits (i.e. the PVB) and GVA. Primarily this arises for two reasons. Firstly the two methods (the CBA and the macroeconomic forecasts) include different impacts, and secondly they can measure the same impact differently. These points are elaborated below.
33 Subject to the size of the study area being appropriate for the appraisal.
Annexe 5: Modelling the macroeconomic impacts of transport investment using CBA output
172 Frontier Economics, Atkins, ITS | June 2011
Putting aside differences in the spatial scope between CBA and GVA type analysis and differences in the role of ancillary investments (the economic potential argument) as outlined above Table 28 sets out the impacts that each of the methods (CBA and GVA) include. As can be seen the CBA measure is more holistic than the GVA method. Measures included in a CBA but not included in a GDP type study are „quality of life‟ type benefits. These include user benefits accruing to non-work and commuting trip purposes and the preference people have for having services accessible to them even if they don‟t use them (the option value). The safety and some environmental impacts have macroeconomic consequences34 and such consequences are almost never included in a GDP type forecast of a transport infrastructure improvement. The „quality of life‟ benefits often account for a substantial component of the PVB (for example 50%).
Table 28. Transport impacts utilised by CBA and GVA modelling
CBA only CBA and GVA impacts GVA forecasts only Included in CBA Included in CBA but excluded and GVA from most GVA forecasts forecasts
User benefits(non- Safety costs User benefits None work trips: commuting, (business and Environmental retail, leisure, tourism, freight) costs etc.) Profitability of Option values transport operators Changes in productivity due to changes in agglomeration
The second major difference between CBA and macroeconomic forecast is how the impacts are measured. A cost benefit analysis only attributes the marginal benefit of using new resources as a welfare benefit. Thus the value of new employment in a cost benefit analysis is a function of the change in commuting cost (equal to the triangular element of the classic consumer surplus trapezium
34 Transport accident casualties result in a loss of consumption and production and also incur damage costs. Noise, air pollution and carbon impose costs on the economy either through damage costs or negative implications on health
Annexe 5: Modelling the macroeconomic impacts of transport investment using CBA output
June 2011 | Frontier Economics, Atkins, ITS 173
for commuting traffic)35, whilst the value of increased output is a function of business and freight user costs (again equal to the triangular element of the consumer surplus trapezium for business and freight traffic). In contrast the gross value of wages and prices are included in GVA and GDP metrics. For projects which have significant impacts on employment and output this difference is very relevant. Furthermore in a CBA willingness to pay values that exceed GDP economic type costs are used to value safety and environmental impacts. These values are derived from observations of people‟s preferences given that in the main these impacts are not traded. Another reason that leads CBAs and macroeconomic forecasts to measure the same impacts differently is that invariably a CBA uses a equity value rather than a local value for valuing the impacts. In reality marginal values vary by income and therefore there is typically a spatial and social pattern to how marginal values (e.g. of time savings) will vary in a country. To ensure investment is not concentrated in the richest parts of the country average values are typically used in CBA. Macroeconomic forecasts however need to be based on local economic values (e.g. of wages). Modelling the macroeconomic impacts of transport projects It is our view that: differences in impacts included in a CBA and macroeconomic forecasts; differences in how the impacts are measured; differences in the spatial scale; and whether the macroeconomic forecast is of economic potential mean that CBA outputs per se are of little assistance in the development of macroeconomic forecasts. Saying that, the changes in accessibility and travel costs that drive a transport CBA are also the drivers to macroeconomic forecasts. A modelling suite comprising of a transport model and an economic model is therefore required to develop both a CBA analysis and a macroeconomic forecast. If only a CBA is required then all that is needed is a transport model. In the main macroeconomic models cannot be used to calculate a CBA. This can give rise to a slight inconsistency between a CBA (calculated from a transport model) and a macroeconomic forecast of the impact of the transport infrastructure. To avoid this inconsistency two options present themselves, the development of a Land Use Transport Interaction (LUTI) model or the use of a Spatial Computable General Equilibrium (SCGE) model. LUTI models typically focus more on land uses (including forecasting employment and population) at a detailed zonal level than on forecasting economic growth but do include basic
35 If involuntary unemployment exists in locations where employment is generated a welfare benefit occurs. This is additional to the change in commuting costs, occurs. This benefit is equal to the difference between the wage and the shadow wage for each new job created. Unless the shadow wage is zero (which is extremely unlikely) the benefit attributed to new employment in a CBA will always be substantially less than that attributed in a GVA assessment.
Annexe 5: Modelling the macroeconomic impacts of transport investment using CBA output
174 Frontier Economics, Atkins, ITS | June 2011
regional economic models based on Input-Output tables. With LUTI models the macroeconomic forecast would be output by the LUTI model, whilst the CBA would be calculated from the transport model – consistency between the forecasts is achieved as the modelling process iterates between the land use (and economic) model and the transport model. SCGE models on the other hand can output both CBAs and macroeconomic forecasts. This is because whilst SCGE models produce forecasts of changes in macroeconomic variables (GDP, population, etc.) they are based on a microeconomic model of behaviour by households, businesses and government. By analysing changes in welfare at a household level a welfare benefit measure that can be used in the CBA can be generated. There is a step change in the level of resources needed between using a transport model to create a CBA and using a transport model to create a CBA and a macroeconomic forecast. The macroeconomic model fed by the transport model requires at a minimum some form of input-output table or a model of how existing transport costs impact on households and businesses. A further step change in the level of resources and in sophistication is needed should a transport model interact with a land use model or an SCGE model (i.e. land use/economic responses are to be fed back into the transport model). The scale of the task should not be underestimated. It is for this reason that, whilst growing, the number of LUTI model applications is small, and that of SCGE applications to transport projects even smaller (and confined to university research studies).
Annexe 5: Modelling the macroeconomic impacts of transport investment using CBA output
Frontier Economics Limited in Europe is a member of the Frontier Economics network, which consists of separate companies based in Europe (Brussels, Cologne, London & Madrid) and Australia (Brisbane, Melbourne & Sydney). The companies are independently owned, and legal commitments entered into by any one company do not impose any obligations on other companies in the network. All views expressed in this document are the views of Frontier Economics Limited.
FRONTIER ECONOMICS EUROPE BRUSSELS | COLOGNE | LONDON | MADRID
Frontier Economics Ltd 71 High Holborn London WC1V 6DA Tel. +44 (0)20 7031 7000 Fax. +44 (0)20 7031 7001 www.frontier-economics.com