How should the ex post evaluation of trunk road schemes be enhanced? Report prepared for Department for Transport June 2005 Final report Oxera i Draft for Comment: Strictly Confidential Oxera Consulting Ltd is registered in England No. 2589629. Registered office at Blue Boar Court, Alfred Street, Oxford OX1 4EH, UK. Although every effort has been made to ensure the accuracy of the material and the integrity of the analysis presented herein, the Company accepts no liability for any actions taken on the basis of its contents. Oxera Consulting Ltd is not licensed in the conduct of investment business as defined in the Financial Services and Markets Act 2000. Anyone considering a specific investment should consult their own broker or other investment adviser. The Company accepts no liability for any specific investment decision, which must be at the investor’s own risk. © Oxera, 2005. All rights reserved. Except for the quotation of short passages for the purposes of criticism or review, no part may be used or reproduced without permission. Executive summary This report considers the feasibility of enhancing the way in which the Highways Agency and the Department for Transport (DfT) evaluate the impacts of major road schemes. It concludes that it would be possible and worthwhile to enhance existing approaches, and suggests a way forward. The study was conducted by Oxera, in conjunction with Mott MacDonald, Social Research Associates (Sra), the Tavistock Institute and the Transport Studies Unit, University of Oxford (TSU). The work was overseen by a Steering Group comprising representatives from the Highways Agency and the DfT. Background The DfT is committed to evaluating its policies and interventions, in accordance with binding cross-government requirements.1 Recent guidance and consultation published by the DfT highlights how important effective evaluation is to scheme management and ongoing policy- making: An evaluation … is an independent quantitative and qualitative assessment of the processes of implementing a scheme and its impacts. Evaluating major schemes will help the Department meet its commitment to assess the impacts of its policies, and provide the Department and authorities with valuable evidence to inform future scheme development and decision-making.2 In 2001, the Highways Agency introduced Post Opening Project Evaluation (POPE), its programme for evaluating post-implementation impacts of major road schemes. Each scheme now undergoes the same analysis one year and three to five years after opening. POPE has clear benefits in the way it has brought a systematic approach to evaluation and improved data retention to aid analysis. However: – POPE has thus far been limited to traffic volumes, travel times and accidents (although a need for it to address environmental impacts has already been recognised); – POPE has not been mapped against the needs of the users of evaluation, and evaluations are not tailored to the specific issues raised by particular schemes; – evidence is lacking on how road scheme evaluations have assisted, and could assist, policy-making; – relatively little money is spent on POPE—for example, in 2004/05, the Highways Agency evaluated around 50 schemes, at an average scheme cost of £12,000 (ie, evaluation represented 0.1% of the £507m Highways Agency major improvements budget3). Therefore, in late 2003, the DfT commissioned this study to assess the feasibility, potential extent, value and cost-effectiveness of additional evaluation activities. 1 For example, Cabinet Office (1999), ‘Modernising Government’, White Paper; HM Treasury (2003), ‘The Green Book: Appraisal and Evaluation in Central Government’; Office of the Deputy Prime Minister (2004), ‘Assessing the Impacts of Spatial Interventions: Regeneration, Renewal and Regional Development—The 3Rs Guidance’; and Office of Government Commerce (2005), ‘The OGC GatewayTM Process’. 2 DfT (2005), 'Guidance for Local Authorities Seeking DfT Funding for Major Transport Schemes' , draft guidance currently subject to consultation. 3 Source: Highways Agency Business Plan, 2003/04. Oxera i The ex post evaluation of trunk road schemes Methods The study team used evidence from past evaluations, detailed case studies, and consultation with experts and stakeholders to assess potential interest in evaluations, and to develop possible tools and techniques to address them. The approach included: – identifying and assessing over 300 transport evaluation documents; – carrying out six detailed case studies of transport scheme evaluations; – interviewing potential evaluation users from the DfT, the Highways Agency, other government departments, external organisations, and the communities in three of the case study areas; – drawing on this evidence to develop a wide selection of evaluation options across all key scheme impacts, supported by an assessment of their benefits and costs (these form a prototype ‘toolkit’ on which future evaluators could draw); – using this toolkit, revisiting the three road-based evaluation case studies, to determine how evaluation might have been carried out differently in each. Findings 1. There is unmet demand for evaluation evidence from a wide range of potential customers—appraisers want evidence that would assist the development of appraisal tools and techniques; policy-makers want to know that schemes deliver planned benefits and that roads policy is proving effective; designers and implementers want good- practice evidence on delivering roads; and communities want to know that their concerns are effectively addressed. 2. There are important issues that evaluations currently do not cover—the greatest demand is for evidence on landscape, severance, local air quality and reliability impacts, as well as on the processes involved in delivering schemes. 3. There is no mechanism for drawing together and addressing different interests, and the dissemination of findings is ineffective—at present, the Highways Agency is responsible for commissioning evaluations, subject to DfT guidance, but the range of interests represented in this process is limited to a small number of officials. Many consultees were unaware of evaluations or of their findings. It is not surprising that there is unmet demand for evaluation evidence if its customers are unaware that it is being conducted and have no opportunity to shape it into something that meets their needs. Likewise, it is not surprising that some stakeholders cannot immediately appreciate the potential benefits of evaluation if they are unaware of its existence. 4. It would be feasible to address the demand for wider evaluation evidence—the DfT/Highways Agency can learn from other transport evaluations, both in the UK and internationally, that have covered a wider range of issues, and proved more effective. The study team was specifically required to consider problems associated with transferability of evidence; timeliness of findings; boundaries; assessment of cause, effect and attribution; estimating counterfactuals (ie, what would have happened had no scheme or a different scheme been implemented); and confounding factors and unexpected events. The toolkit that has been developed demonstrates that practical difficulties can usually be overcome and that more complex issues, including the counterfactual, are not always relevant. 5. The benefits of additional evaluation would exceed additional costs—enhancing evaluation would cost more. As a guide, covering all of the top ten information needs for every scheme would cost an average of £40,000–£50,000 per scheme (not including tailored social research, the costs of which would vary according to requirements). Evaluating to this degree the 35 schemes planned for the next three years could cost around £600,000 per year. However, the study team recommends a cheaper, more flexible approach, with evaluations addressing a smaller number of key information needs in varying levels of detail, over different timescales and in proportion to scheme Oxera ii The ex post evaluation of trunk road schemes size. The costs of such an approach are difficult to predict, although it is possible that they might not increase at all if the DfT/Highways Agency choose to replace POPE's current core focus (volumes, times and accidents) with an entirely flexible approach. The study team also found that, in addition to local authorities, other bodies would be willing to contribute resources (eg, professional expertise) to evaluations that are relevant to their organisational priorities. Furthermore, costs should fall over time as the evidence base develops and information gaps, at least at the national level, are reduced. While it is not possible to place a meaningful monetary value on the tools proposed, the study team identified a number of detailed benefits of adopting them. Such benefits include improved modelling and forecasting; more effective scheme design and post-opening mitigation; the identification of unanticipated impacts; improved information on attitudes to roads; evidence on high-profile issues; and examples of how to reduce costs and speed up delivery. More generally, this analysis suggests that key outcomes from improved evaluation could include: – policy accountability—within the DfT and the Highways Agency, and externally; – improving appraisal where it is currently weak, or where there is a large degree of uncertainty about its conclusions; – closing the appraisal–decision-making–evaluation loop (eg, informing decisions on which road scheme choices are