Evaluation of Forecasting Services of the Directorate General Economic and Financial Affairs Final Report December 2007 Evaluation of Forecasting Services of the Directorate General Economic and Financial Affairs Final Report December 2007 ADE s.a. Rue de Clairvaux, 40 B-1348 Louvain-la-Neuve Belgium Tel.: +32 10 45 45 10 Fax: +32 10 45 40 99 E-mail: [email protected] Web: www.ade.be This report has been prepared by ADE and DIW at the request of the European Commission. The views expressed are those of the Consultant and do not represent the official views of the European Commission. EVALUATION OF FORECASTING SERVICES OF DG-ECFIN ADE-DIW Table of Contents LIST OF ACRONYMS EXECUTIVE SUMMARY 1. INTRODUCTION ........................................................................................................1 1.1 OBJECTIVE AND SCOPE OF THE EVALUATION ......................................................................... 1 1.2 METHODOLOGICAL APPROACH AND ORGANISATION............................................................ 2 1.3 STRUCTURE OF THE DRAFT FINAL REPORT............................................................................. 3 2. OVERALL FRAMEWORK OF THE FORECASTING ACTIVITIES OF DG ECFIN.............. 5 2.1 THE COMMISSION SERVICES' FORECASTING IN EU BUDGETARY SURVEILLANCE AND ECONOMIC POLICY COORDINATION................................................................................. 5 2.2 THE FORECASTING ACTIVITIES ................................................................................................ 10 2.3 THE USERS AND THE USAGES OF DG ECFIN FORECASTS................................................... 20 2.4 OVERVIEW OF COMPETITIVE FORECASTS............................................................................... 21 3. INTERVENTION LOGIC............................................................................................25 4 ANSWERS TO THE EVALUATION QUESTIONS ..........................................................29 4.1 EVALUATION QUESTION Q1A ................................................................................................. 29 4.2 EVALUATION QUESTION Q1B.................................................................................................. 37 4.3 EVALUATION QUESTION Q1C ................................................................................................. 44 4.4 EVALUATION QUESTION Q2A ................................................................................................. 48 4.5 EVALUATION QUESTION Q2B.................................................................................................. 51 4. 6 EVALUATION QUESTION Q2C ................................................................................................. 54 4.7 EVALUATION QUESTION 2D .................................................................................................... 56 4.8 EVALUATION Q3A ..................................................................................................................... 58 4.9 EVALUATION QUESTION 3B ..................................................................................................... 64 4.10 EVALUATION QUESTION Q4A................................................................................................... 68 4.11 EVALUATION QUESTION Q4B.................................................................................................. 71 4.12 EVALUATION QUESTION Q5A ................................................................................................. 73 4.13 EVALUATION QUESTION 5B ..................................................................................................... 78 5 5. CONCLUSIONS .....................................................................................................79 5.1 CONCLUSIONS ON THE PRODUCTS, THE PROCESS AND THE METHODS.............................. 79 5.2 CONCLUSIONS ON THE CONTRIBUTION OF THE FORECASTING ACTIVITIES TO THEIR SPECIFIC AND INTERMEDIATE OBJECTIVES................................................................. 82 5.3 CONCLUSIONS ON THE COMMUNICATION STRATEGY .......................................................... 84 6. RECOMMENDATIONS ..............................................................................................86 6.1 RECOMMENDATIONS REGARDING THE EFFICIENCY OF FORECASTING ACTIVITIES ........ 86 6.2 RECOMMENDATIONS CONCERNING THE CONTRIBUTION OF FORECASTS TO POLICY-MAKING IMPROVEMENTS ............................................................................................ 90 6.3 RECOMMENDATIONS CONCERNING THE COMMUNICATIONS STRATEGY .......................... 94 Final Report – December 2007 Table of Contents EVALUATION OF FORECASTING SERVICES OF DG-ECFIN ADE-DIW ANNEXES: ANNEX 1 - TERMS OF REFERENCE ANNEX 2 - LIST OF PERSONS MET ANNEX 3 - REFERENCES ANNEX 4 - LIST OF EVALUATION QUESTIONS ANNEX 5 - RESULTS OF THE QUESTIONNAIRE SURVEY ON THE USERS AND USES OF DG ECFFIN FORECASTS ANNEX 6 - RESULTS OF THE QUESTIONNAIRE SURVEY TO ASSESS IF PRESSURE IS EXERTED ON THE PRODUCERS OF DG ECFIN FORECASTS. ANNEX 7 - METHODOLOGY FOR EVALUATING FORECAST RESULTS ANNEX 8 - RESULTS OF THE QUANTITATIVE ANALYSIS OF THE RELIABILITY OF THE FORECASTS ANNEX 9 - ORGANISATION CHART OF DG ECFIN Final Report – December 2007 Table of Contents EVALUATION OF FORECASTING SERVICES OF DG-ECFIN ADE-DIW List of Acronyms AMECO Annual Macro Economic Database of the Commission ARIMA Autoregressive Integrated Moving Average BEPG Broad Economic Policy Guidelines CLI Composite Leading Indicators COR Committee of Regions DFM Dynamic Factor Model DG Directorate General DG ECFIN Directorate General for Economic and Financial Affairs DSGE Dynamic Stochastic General Equilibrium model EC European Commission ECB European Central Bank ECOFIN Economic and Financial Affairs Council ECON Committee on Economic and Monetary Affairs (European Parliament) EESC European Economic and Social Committee EFC Economic and Financial Committee EG Employment Guidelines EIB European Investment Bank EMU Economic and Monetary Union EO Economic Outlook (OECD) EP European Parliament EPC Economic Policy Committee ERM II European Exchange Rate Mechanism (Post 1999) EU European Union GDP Gross Domestic Product GEM Global Economy Model GFCF Gross fixed Capital Formation HICP Harmonised Index of Consumer Price IEPG Integrated Economic Policy Guidelines IGP Integrated Guidelines Package IMF International Monetary Fund JC Judgement Criterion MS Member States MP Macroeconomic Projections NAIRU Non-Accelerating Inflation Rate of Unemployment NCB National Central Banks NRP National Reform Plans OECD Organisation for Economic Co-operation and Development PEP Pre-accession Economic Programme RMSE Root Mean Square Error SGP Stability and Growth Pact SNA System of National Accounts SVAR Structural Vector Auto-Regression model VA Value-added WOE World Economic Outlook (IMF) Final Report – December 2007 List of acronyms EVALUATION OF FORECASTING SERVICES OF DG-ECFIN ADE-DIW Executive Summary ECFIN, in the framework of its mission Evaluation Objectives statement: “DG ECFIN's main role is to foster the success of Economic and Monetary Union both This evaluation of the DG ECFIN’s inside and outside the European Union, by forecasting services was commissioned by conducting economic and budgetary surveillance, the Directorate-General for Economic and providing policy assessment and advice, promoting Financial Affairs within the framework of appropriate policy action and advancing economic the DG’s Evaluation Plan for 2006. As policy coordination 2 ”. Hence, the evaluation stated in the Terms of Reference, the general addresses DG ECFIN’s forecasting activities objective of the evaluation is to promote in the broader context of the other activities understanding and lesson-learning leading, of the DG. where demonstrated as necessary, to changes The evaluation also takes into account the in the forecasting activities of the DG and fact that macroeconomic forecasts are also their resulting outputs, with consequent made available by numerous other benefits for the user communities, both 1 institutions outside the Commission, notably internal and external to the Commission . the OECD, IMF, ECB, the Member States, private businesses and scientific The evaluation addresses the following organisations. issues: the extent to which DG ECFIN’s forecasting outputs are used by various Methodology and organisation institutions and match their actual and future needs; The Evaluation Questions were imposed by the Terms of Reference and the the capacity of the forecasting process to methodological approach concentrated on meet DG ECFIN’s goals and deliver how to answer them. The evaluation high quality output; methodology included the following the capacity of the forecasting process to sequence of activities: maintain and enhance the internal Structuring the evaluation: i) elaborating knowledge and skill base of DG ECFIN; the intervention logic; ii) formulating the contribution of DG ECFIN’s judgement criteria for each Evaluation forecasts to policy formulation and Question; and iii) developing methods implementation; and instruments for collection of the the quality of the communication necessary information. strategy for forecasts. Collecting the information through the following: i) conduct of open and semi- structured interviews with strategic Evaluation Context informants; ii) elaboration of two The production of macroeconomic forecasts questionnaire surveys; iii) analysis of DG is one of