Reference guide on ex-post evaluation of competition agencies’ enforcement decisions April 2016 Governments everywhere are increasingly interested in assessing the effects of their policies and the effectiveness of public institutions, and competition agencies are no exception. This Reference Guide contains an in-depth overview of all the issues linked to ex-post assessments and contains numerous examples and references. It constitutes an excellent resource both for the competition authorities who are planning to start performing ex-post evaluations and for those who already do it but want to improve the quality of their assessment. The Reference Guide complements a Guide for assessing the impact of competition authorities' activities (2014) and a Factsheet on how competition policy affects macro-economic outcomes (2014). Find information relating to this work on the OECD website at http://oe.cd/J3 This Guide was written by Cristiana Vitale and Silvia Carrieri with support from Pedro Gonzaga from the OECD Competition Division. Valuable feedback and comments were provided by numerous competition agencies, as well as by Steve Davies, Tomaso Duso, Amelia Fletcher, João Gata, Alberto Heimler, Daniel Hosken, Damien Neven, Peter Ormosi, and Pedro Pereira. This work is published under the responsibility of the Secretary-General of the OECD. The opinions expressed and arguments employed herein do not necessarily reflect the official views of OECD member countries. This document and any map included herein are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. Table of Contents Reference guide on ex-post evaluation of competition agencies’ enforcement decisions ............................................................................................................................. 3 INTRODUCTION ................................................................................................................................ 3 Why is there so much interest in ex-post evaluations of competition enforcement decisions? ................................................................................................................................... 3 The OECD Reference Guide ........................................................................................................ 4 PART I: WHAT ARE EX-POST EVALUATIONS OF ENFORCEMENT DECISIONS AND WHY THEY ARE UNDERTAKEN ................................................................................................. 5 1. What is an ex-post evaluation?............................................................................................... 5 2. Why are ex-post evaluations of enforcement decisions undertaken? ................................... 6 PART II: HOW TO PERFORM THE EX-POST EVALUATION OF AN ENFORCEMENT DECISION ....... 15 The key steps of an ex-post evaluation ....................................................................................... 15 1. Select the decisions to assess ............................................................................................... 16 2. Choose the team who will perform the evaluation .............................................................. 19 3. Identify the appropriate counterfactual(s) ........................................................................... 21 4. Select the methodology ........................................................................................................ 22 5. Determine the variables to study ........................................................................................ 25 6. Collect data and information ................................................................................................ 27 7. Perform the analysis ............................................................................................................. 31 8. Verify the robustness of the results ...................................................................................... 32 9. Identify key lessons ............................................................................................................... 33 ANNEX A. EX-POST ASSESSMENTS OF THE EFFECTIVENESS OF MERGER REMEDIES ................... 35 ANNEX B. METHODOLOGIES FOR THE EX-POST QUANTIFICATION OF THE EFFECT OF AN ENFORCEMENT DECISION .............................................................................................................. 41 ANNEX C. LIST OF EX-POST STUDIES .............................................................................................. 69 BIBLIOGRAPHY ................................................................................................................................ 78 REFERENCE GUIDE ON EX-POST EVALUATION OF COMPETITION AGENCIES’ ENFORCEMENT DECISIONS © OECD 2016 1 BOXES Box 1. The terminology used in this Guide ................................................................................ 6 Box 2. Testing the validity of specific expectations: the New Zealand Commerce Commission’s approach ............................................................................................................. 8 Box 3. A review of merger decisions in the EU: What can we learn from ex-post evaluations? ...................................................................................................... 10 Box 4. Ex-ante decisions are inevitably probabilistic ............................................................... 13 Box 5. Seed- planting ............................................................................................................... 14 Box 6. Ex-post assessment of decisions concerning cartels .................................................... 17 Box 7. Risk of a selection bias in the selection of the decisions to assess............................... 19 Box 8. A call for permitting more ex-post studies by external advisors, including academics .. 21 Box 9. An example of an ex-post evaluation using more than one counterfactual ................ 22 Box 10. Assessing the short and long terms effects of a merger ............................................ 30 Box 11. Lessons learnt from the ex-post assessment of mergers in the US hospital sector ... 34 Box A.1 Examples of the ex-post assessment of remedies ..................................................... 37 Box A.2 Examples of lessons learnt from the ex-post assessment of remedies ..................... 39 Box B.1 Examples based on the use of simple comparator based methods ........................... 44 Box B.2. Examples of control markets ..................................................................................... 48 Box B.3 Example of use of simulations and structural models: the case of the Portuguese non-life insurance industry ............................................................................ 55 Box B.4 Examples of ex-post studies based on survey ............................................................ 61 Box B.5 Examples of event studies .......................................................................................... 64 FIGURES Figure 1.The role of ex-post evaluation in the decision making process of a CA ...................... 7 Figure 2.Outline of the steps involved in the assessment of the actual effects of an enforcement decision .............................................................................. 15 Figure 3. The DiD estimator ..................................................................................................... 46 TABLES Table 1. Some examples of quantitative assessments of the impact of mergers on non-price variables ................................................................................................ 25 Table 2. Some examples of length of time needed to assess the impact of efficiencies in mergers ......................................................................................................... 30 Table B.3. Methodologies that can be applied to different types of decisions....................... 67 Table B.4. Pros and Cons of the various methodologies ......................................................... 68 2 REFERENCE GUIDE ON EX-POST EVALUATION OF COMPETITION AGENCIES’ ENFORCEMENT DECISIONS © OECD 2016 Reference guide on ex-post evaluation of competition agencies’ enforcement decisions Introduction The enforcement challenge faced by competition agencies is enormous. With limited information and time competition agencies must evaluate the likely competitive impact of often unprecedented changes in how markets operate. Horizontal merger analysis, for example, is necessarily forward-looking. Agencies, by statute, must make predictions about how a proposed merger will affect competition. Competition agencies combine the best information available at the time of the transaction with methodologies that have been developed over time to predict how mergers will affect competition. Periodic ex post analysis of enforcement decisions are one tool that can help agencies improve future decision making. An ex-post evaluation is an examination of an enforcement decision that tries to determine the effects that the decision has had in the market sometime after the decision has been issued. Many evaluations focus on the effect caused on the prices of the goods and services traded on the market affected by the decision, but in some cases they also look at changes in other variables, such as quality, variety,