How (In)Accurate Are Demand Forecasts in Public Works Projects? Forecasted Traffic
Total Page:16
File Type:pdf, Size:1020Kb
How (In)accurate Are This article presents results from the first statistically significant study of traffic Demand Forecasts in forecasts in transportation infrastructure projects. The sample used is the largest of its kind, covering projects in Public Works Projects? nations worth U.S.$ billion. The study shows with very high statistical signifi- cance that forecasters generally do a poor job of estimating the demand for trans- portation infrastructure projects. For The Case of Transportation out of rail projects, passenger forecasts are overestimated; the average overestima- tion is %. For half of all road projects, Bent Flyvbjerg, Mette K. Skamris Holm, and Søren L. Buhl the difference between actual and fore- casted traffic is more than ±%. The result is substantial financial risks, which are typically ignored or downplayed by planners and decision makers to the det- espite the enormous sums of money being spent on transportation riment of social and economic welfare. infrastructure, surprisingly little systematic knowledge exists about the Our data also show that forecasts have not costs, benefits, and risks involved. The literature lacks statistically valid become more accurate over the -year D answers to the central and self-evident question of whether transportation infra- period studied, despite claims to the con- structure projects perform as forecasted. When a project underperforms, this is trary by forecasters. The causes of inaccu- racy in forecasts are different for rail and often explained away as an isolated instance of unfortunate circumstance; it is road projects, with political causes playing typically not seen as the particular expression of a general pattern of underper- a larger role for rail than for road. The formance in transportation infrastructure projects. Because knowledge is wanting cure is transparency, accountability, and in this area of research, until now it has been impossible to validly refute or con- new forecasting methods. The challenge firm whether underperformance is the exception or the rule. is to change the governance structures for forecasting and project development. In three previous articles (Flyvbjerg, Holm, et al., , , ), we Our article shows how planners may help answered the question of project performance as regards costs and cost-related achieve this. risks. We found that projects do not perform as forecasted in terms of costs: almost out of projects fall victim to significant cost overrun. We also inves- Bent Flyvbjerg is a professor of planning tigated the causes and cures of such inaccurate cost projections (see Flyvbjerg, at Aalborg University, Denmark. He is Bruzelius, et al., ). In this article we focus on the benefit side of investments founder and director of the university’s and answer the question of whether projects perform as forecasted in terms of research program on large-scale infra- structure planning. His latest books are demand and revenue risks. We compare forecasted demand with actual demand Megaprojects and Risk (Cambridge Uni- for a large number of projects. Knowledge about cost risk, benefit risk, and com- versity Press, , with Nils Bruzelius pound risk is crucial to making informed decisions about projects. This is not to and Werner Rothengatter), Making Social say that costs and benefits are or should be the only basis for deciding whether to Science Matter (Cambridge University build. Clearly, forms of rationality other than economic rationality are at work Press, ), and Rationality and Power in most infrastructure projects and are balanced in the broader frame of public (University of Chicago Press, ). Mette K. Skamris Holm is a former assistant decision making. But the costs and benefits of infrastructure projects often run professor of planning at Aalborg Univer- in the hundreds of millions of dollars, with risks correspondingly high. Without sity. She now works as a planner with knowledge of such risks, decisions are likely to be flawed. Aalborg Municipality. Søren L. Buhl As pointed out by Pickrell () and Richmond (), estimates of the is an associate professor of mathematics financial viability of projects are heavily dependent on the accuracy of traffic at Aalborg University. He is associate demand forecasts. Such forecasts are also the basis for socioeconomic and envi- statistician with the university’s research program on large-scale infrastructure ronmental appraisal of transportation infrastructure projects. According to the planning. experiences gained with the accuracy of demand forecasting in the transportation sector, covering traffic volumes, spatial traffic distribution, and distribution be- Journal of the American Planning Association, Vol. , No. , Spring . tween transportation modes, there is evidence that demand forecasting—like cost © American Planning Association, Chicago, IL. forecasting, and despite all scientific progress in modeling—is a major source of Journal of the American Planning Association, Spring , Vol. , No. uncertainty and risk in the appraisal of transportation small samples used in existing studies; it does not hold for infrastructure projects. the project population. When we enlarge the sample of Traffic forecasts are routinely used to dimension the projects by a factor – to a more representative one, we construction of transportation infrastructure projects. find a different picture. Road traffic forecasts are not gen- Accuracy in such forecasts is a point of considerable im- erally overestimated, although they are often very inaccu- portance for the effective allocation of scarce funds. For rate, whereas forecasts of rail patronage are generally over- example, Bangkok’s U.S.$ billion Skytrain was hugely estimated, often dramatically so. overdimensioned because the passenger forecasts were . We follow common practice and define the inaccuracy times higher than actual traffic. As a result, station plat- of a traffic forecast as actual minus forecasted traffic in per- forms are too long for the shortened trains that now oper- centage of forecasted traffic. Traffic is measured as number ate the system, a large number of trains and cars are idly of passengers for rail, and number of vehicles for roads parked in the train garage because there is no need for them, Actual traffic is counted for the first year of operations (or terminals are too large, etc. The project company has ended the opening year). Forecasted traffic is the traffic estimate up in financial trouble, and even though urban rail is prob- for the first year of operations (or the opening year) as esti- ably a good idea for a congested and air-polluted city like mated at the time of decision to build the project. Thus Bangkok, overinvesting in idle capacity is hardly the best the forecast is the estimate available to decision makers way to use resources, and especially not in a developing when they made the decision to build the project in ques- nation where capital for investment is scarce. Conversely, tion. If no estimate was available at the time of decision to a U.K. National Audit Office () study identified a num- build, then the closest available estimate was used, typically ber of road projects that were underdimensioned because a later estimate, resulting in a conservative bias in our traffic forecasts were too low. This, too, led to multimillion- measure for inaccuracy. dollar inefficiencies, because it is much more expensive to We measured inaccuracy of traffic forecasts in a sample add capacity to existing, fully used roads than it is to build of transportation infrastructure projects with compar- the capacity up front. For these and other reasons, accuracy able data for forecasted and actual traffic. The sample in traffic forecasts matters. comprises a project portfolio worth approximately U.S.$ Nevertheless, rigorous studies of accuracy are rare. billion in actual costs ( prices). The portfolio includes Where such studies exist, they are characteristically small- rail projects and road projects completed between N research; that is, they are single-case studies or they and . The project types are urban rail, high-speed cover a sample of projects too small or too uneven to allow rail, conventional rail, bridges, tunnels, highways, and free- systematic, statistical analyses (Brooks & Trevelyan, ; ways. The projects are located in countries on conti- Fouracre et al., ; Fullerton & Openshaw, ; Kain, nents, including both developed and developing nations: ; Mackinder & Evans, ; National Audit Office, Brazil, Chile, Denmark, Egypt, France, Germany, Hong , ; Pickrell, ; Richmond, ; Walmsley & Kong, India, Mexico, South Korea, Sweden, Tunisia, the Pickett, ; Webber, : World Bank, ). Despite U.K., and the U.S. Projects were selected for the sample their value in other respects, with these and other studies, based on the availability and quality of data. As far as we it has so far been impossible to give statistically satisfying know, this is the largest sample of transportation infrastruc- answers to questions about how accurate traffic forecasts ture projects that has been established with comparable are for transportation infrastructure projects. data on forecasted and actual traffic. For a full description The objective of the present study has been to change of the sample, data, and methods of testing for inaccuracy, this state of affairs by establishing a sample of transpor- please see Flyvbjerg (). tation infrastructure projects that is sufficiently large to permit statistically valid answers to questions of accuracy. In addition, it has been a practical objective