Persistent Challenges in UK Defence Equipment Acquisition

Lucia Retter, Julia Muravska, Ben Williams, James Black For more information on this publication, visit www.rand.org/t/RRA1174-1

About RAND Europe RAND Europe is a not-for-profit research organisation that helps improve policy and decisionmaking through research and analysis. To learn more about RAND Europe, visit www.randeurope.org.

Research Integrity Our mission to help improve policy and decisionmaking through research and analysis is enabled through our core values of quality and objectivity and our unwavering commitment to the highest level of integrity and ethical behaviour. To help ensure our research and analysis are rigorous, objective, and nonpartisan, we subject our research publications to a robust and exacting quality-assurance process; avoid both the appearance and reality of financial and other conflicts of interest through staff training, project screening, and a policy of mandatory disclosure; and pursue transparency in our research engagements through our commitment to the open publication of our research findings and recommendations, disclosure of the source of funding of published research, and policies to ensure intellectual independence. For more information, visit www.rand.org/about/principles.

RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors.

Published by the RAND Corporation, Santa Monica, Calif., and Cambridge, UK © Copyright 2021 RAND Corporation R® is a registered trademark. Cover image: UK F-35 Lightning jets onboard Britain’s next generation aircraft carrier, HMS Queen Elizabeth for the first time (UK Ministry of Defence 2020)

Limited Print and Electronic Distribution Rights This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorised posting of this publication online is prohibited. Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial use. For information on reprint and linking permissions, please visit www.rand.org/pubs/permissions. 1

Preface

Equipping the armed forces to deliver In January 2021, the NAO commissioned their outputs is one of the most important, RAND Europe to prepare a short discussion challenging and complex tasks faced by the paper to help shed light on the underlying UK government. There are few, if any, public causes of equipment procurement problems sector investment programmes that involve and the reasons why they persist despite such levels of uncertainty, complexity, technical repeated efforts to learn from the past.2 Part risk and expenditure. Often, acquisition of the global RAND Corporation, RAND Europe programmes experience one or more of is a not-for-profit research institute whose the challenges associated with such large mission is to help improve policy and decision investments: cost growth, schedule slippage making through objective research and and performance shortfall. This affects value- analysis. This short paper draws on decades’ for-money, as well as the overall ability of the worth of public and non-public RAND analyses Ministry of Defence (MOD) and armed forces of US, UK and European defence acquisition to deliver the defence contribution to achieving programmes and input from senior subject the UK’s National Security Objectives (NSOs) – matter experts (SMEs). It is intended to meaning that the UK’s security, prosperity and provide food for thought for further initiatives influence suffers as a result. within the NAO. For decades, the National Audit Office The authors are very grateful to Hans Pung and (NAO) has been monitoring and evaluating Ruth Harris for their thoughtful comments and the performance of defence acquisition review. All errors and omissions are, however, programmes. The NAO’s reports and analyses, the full responsibility of the authors. such as the Major Projects Review and For more information about RAND or its Equipment Plan as well as focused studies, analytical support to the UK government, are publicly available.1 Over the years, the please contact: NAO has examined both individual equipment programmes and the overall acquisition Dr Julia Muravska system. It has identified many of the reasons Research Leader behind poor programme performance and Defence, Security and Infrastructure provided both general principles and specific RAND Europe, Westbrook Centre, recommendations that will allow the MOD to Milton Road, Cambridge, CB4 1YG improve programme delivery. e. [email protected]

1 For example: NAO (2004); NAO (2005); NAO (2021). 2 For example: Gray (2009); Levene (2011) 2 Persistent Challenges in UK Defence Equipment Acquisition

Underlying causes of equipment procurement problems

Background and scope of this be interoperable militarily but in competition discussion paper commercially. These are the unique dynamics of the defence industrial base and markets, Defence acquisition is complex, uncertain economic pressures, and resource constraints and constantly exposed to the chance of as Defence competes for an adequate failure, requiring sound risk management proportion of the budget pie with other Government departments. Development, acquisition and through-life support of military capabilities are inherently In many respects, large defence acquisition challenging. Since these capabilities are projects share characteristics with large civil often bespoke and differ greatly from their capital investment projects; by their nature, predecessors, this process involves substantial these projects are often pioneering and and sometimes unavoidable risks, some of bespoke, they are long in duration and complex which are generic to large, capital-intensive in terms of design, production, business programmes in other sectors, but many of arrangements, technology and integration. As which reflect the unique operating context a result, there are likely to be many changes of defence. Understanding the acquisition to project scope, budget and benefits as well process itself (from concept to capability), as staff turnover and corporate memory. the approach to individual projects, and the These complexities often make it difficult to wider defence environment demonstrates have a solid understanding of cost, especially why this is the case. Selecting and developing at the project inception, since parametric particular cutting-edge technologies, estimates rely on historic data and do not embedding these within platforms and system necessarily extrapolate into the future and designs, and producing complex systems bottom-up costs are difficult to estimate that are novel and integrating them within given the novelty of the programme in the first existing infrastructure, personnel and other place. This is arguably the most important aspects of capability management3 generate stage in the project’s lifecycle.4 Just as it is inherent risk at each step of the acquisition the case with large civil projects, defence process. Further challenges are presented by acquisition programmes are prone to strong uncertainty around the capabilities and intent optimism bias, which will be explored later in of potential adversaries as well as those of this paper. Finally, both sets of projects tend to some of our allies with whom we will seek to

3 These include the Defence Lines of Development (DLODs), namely: training, equipment, personnel, information, concepts and doctrine, organisation, infrastructure and logistics. 4 For example: NAO (2005); NAO (2020e). 3

BOXER ambulance vehicle crossing a temporary bridge formed by several UK and German M3 Amphibious Rigs in Minden, during a joint UK/German Bridging Operation Crown Copyright Crown

be ‘easy to start but too difficult to stop’,5 with pressure in terms of reducing the profit margin sunk costs or certain design or commercial and driving production efficiencies. Further choices made in initial phases often creating a market distortions are created by security and situation in which programmes are locked into classification requirements, export controls a trajectory, despite early signs of potentially (US’ ITAR in particular), restricting access to the non-recoverable poor performance. Political defence market to suppliers with insufficient pressure and the risk of embarrassment may resources to handle and navigate all these also add to the reduced ability to cancel poorly specificities. performing projects. Given these unique characteristics, it is Finally, it is important to remember that many perhaps unsurprising that many complex, defence acquisition programmes are run large acquisition programmes experience at in a unique market set-up where incentive least some cost growth,6 schedule slippage or structures for both customer and supplier performance shortfall – or all of the above – in are shaped by a monopoly–monopsony the course of their lifecycle. A wide range of relationship. The unique capability requirements factors contribute to historically substandard on the side of the MOD customer often result in performance of large acquisition programmes; a reduced ability to buy off-the-shelf equipment, these are examined in detail in the following foregoing potential benefits of competitive sections.

5 Flyvbjerg (2003; 2014). 6 ‘Cost growth’ is traditionally understood as the difference between actual and estimated costs (Arena 2008). This is different from ‘cost escalation’, which is the change in the price of a specific good/service over time; it can be measured as percentage change in cost between time periods (Arena 2008). 4 Persistent Challenges in UK Defence Equipment Acquisition

This paper discusses four broad drivers Under each of these headings, RAND has of cost, schedule or other performance identified from the literature and consultation problems often encountered in defence with subject matter experts a number of programmes specific factors which are examined briefly in the following sections of this short paper. In Numerous RAND, NAO and other reports identify addition to these, the discussion also captures underlying causes of problems in defence some overarching, cross-cutting factors. All equipment programmes, including MOD’s factors are summarised in Figure 1. industrial policy and strategy, commercial strategies adopted for individual programmes, poor supplier management, supply chain issues Industrial and MOD capabilities and requirements management. The scope The complexity and long duration of many of this discussion paper does not allow for defence acquisition programmes demand that covering all of them, though the bibliography both industry and the MOD have appropriate provides further readings. Rather, this paper capabilities in place at the project initiation highlights three types of factors specifically stage. At the minimum, they should have a requested by the NAO, namely: robust plan on how these capabilities will be • Industrial and MOD skills and capabilities secured and varied over time, as the demand • Supplier performance, incentives and for particular skills changes, to ensure timely contracting and cost-effective delivery. The principal • Programme management, budgeting and capabilities relevant here include a sufficient delivery. quantity of suitably qualified and experienced

Figure 1. Overview of underlying reasons for defence equipment procurement problems

Industrial and MOD Supplier performance, Programme capabilities incentives and contracting management approach

Poor requirement setting Misaligned assumptions Imbalance between Services Production inefficiencies Poor understanding of risk Frequent adjustments to programme delivery Workforce and skills management challenges Insufficient risk provision

‘Conspiracy’ of Lack of institutional Moral hazard optimism memory Cross-cutting

Source: RAND Europe analysis. 5

personnel (SQEP) and appropriate design has also been insufficient focus on iterative and production systems, processes, tools, development of capability, which can reduce materials and facilities. In the area of industrial risk and cost of the overall programme due to and MOD capabilities, two main issues have interim steps in maturing capabilities. emerged most prominently from past RAND analyses and the limited expert consultation In defence, production efficiencies are conducted for this project: poor requirement hard to achieve, hampering industry’s setting, production inefficiencies and ability to drive learning economies and challenges related to workforce and skills. maximise return on investment Several RAND reports highlight the significant Overly prescriptive or overly ambitious impact of production inefficiencies on the capability requirements can set the ability of a given programme to deliver scene for poor performance delivery within its defined performance, cost and down the track schedule envelope. Long gone are days As noted earlier, defence capability when most defence manufacturers benefited development takes place in a highly uncertain from economies of scale, driving down unit environment with many unknowns and the production cost through mass manufacturing. need to make assumptions about factors such In fact, many large equipment programmes as the future operating environment, future have relatively short production runs, with only competition, technologies, as well as MOD limited number of units produced (e.g. ships, and industrial capacity and capability. Past submarines, combat aircraft, helicopters) RAND and NAO reports noted that the MOD and there is a wider trend in recent decades response to these uncertainties has often been towards ever more complex, expensive in the form of prescriptive requirements that and ‘exquisite’ designs and a decades-long solicited an ambitious, ‘gold-plated’ proposal acquisition cycle. This means that each unit from industry in response. Where technical could almost be its own prototype and there specifications are set out in too much detail are only limited opportunities for economies of (instead of, for example, setting out the scale, reducing the productivity benefits to be broad military requirements and use cases), derived from learning or use of new technology industry has little manoeuvre in defining how over the lifetime of a production run.7 Perhaps the requirement could be delivered in a most the most notable recent exception here is efficient and effective way in terms of the key the multinational but US-led Joint Strike criteria: performance, cost and schedule. In Fighter (JSF) programme, which has started these circumstances, programmes basically seeing declining manufacturing costs due to start off trying to deliver an end product that increasing number of units being produced as may not be the best solution from a capability successful exports have secured international perspective in the first place and is likely to customers in addition to the extensive end up being more costly than necessary due orders placed by the US armed services’ joint to the ambitious nature of the design. Based procurement.8 While unit production costs on RAND literature and SME input, there generally decrease following the initial start

7 Arena et al. (2008); Rich et al. (1986); Schank et al. (2011; 2014). 8 Powers et al. (2018). 6 Persistent Challenges in UK Defence Equipment Acquisition

RAF A400M Atlas transport aircraft carrying out a series of test landings and take offs on a beach in South Wales. Crown Copyright Crown

of production, the kind of production at scale Additional costs and schedule delays may currently witnessed with the JSF is relatively be caused by changes to the production unique among Western democratic nations. drumbeat. This can happen for a range of reasons: some being the responsibility of At least since the end of the Cold War, flexible MOD (e.g. due to re-profiling to ensure in-year production facilities have been needed affordability) and some due to commercial both to maintain high utilisation rates and decisions by industry (e.g. the desire to align keep overhead rates low. Flexible, agile and production with other orders for MOD or other increasingly digitalised production facilities can customers, to smooth resourcing and minimise more easily adjust to a new production rate steep ramp-up curves for labour). Whenever and more cheaply adapt to design changes. production rates drop, per unit prices invariably These are the norm in the civil or commercial increase and vice versa as facility and overhead sector (e.g. commercial shipbuilding) but are cost for total production are amortised in the not always present in defence suppliers at the unit production costs. Past RAND analysis programme initiation stage. RAND’s work on also finds that decreases in production rates productivity challenges in UK defence supply hurt more than increases help.10 In other chains has also highlighted the lag with which words, the impact of decreasing production defence equipment manufacturers are adopting rates is greater in terms of additional cost and new and digital manufacturing technologies schedule delay than production ramp-up is in and processes relative to other sectors (e.g. cutting cost and compressing the schedule. automotive), which prevents savings and The physical constraints of facilities and the efficiencies. When there is an effort to innovate challenges of adequately resourcing any in manufacturing, it tends to coincide with the significant workforce ramp-up (as witnessed start of production, adding costs.9

9 Muravska et al. (2020). 10 Rich et al. (1986). 7

in past shipbuilding programmes in particular) especially detrimental effect on the ability mean that there are always greater limits on of industry and MOD to deliver the next the ability to ramp up production drumbeat programme efficiently.13 Defence is a niche than on the ability to slow it down. business where skills are critical and costly to rebuild, particularly in areas where unique Workforce and skills constitute critical skillsets require years of experience and enablers for effective programme delivery may only reside in a relatively small number and conversely, insufficient SQEP can (even single digits) of key individuals. Once undermine effective delivery the appropriately qualified and experienced Closely linked to production inefficiencies and workforce is diminished – whether due to similarly responsible for a large portion of demographic changes such as retirement, overall cost growth and schedule slippage on departure of employees to other industries, or a defence equipment programmes are workforce lack of sufficient demand to justify the expense and skills challenges. Both industry and MOD of new recruitment – it can be prohibitively experience challenges in relation to the quantity costly in both time and financial terms to train and quality of their respective workforces up the SQEP from a low or zero base. across all levels.11 Further challenges Similarly, long gaps between programmes experienced by large acquisition programmes mean that critical skills, particularly in the in the past have been created by rapid design and development stages of the workforce ramp-up for production. Past naval equipment lifecycle, are not sufficiently and submarine programmes demonstrate the exercised and tend to atrophy. RAND’s difficulties and delays that can arise from too analyses of the UK combat air industrial base rapid a build-up of the workforce and use of have raised these concerns time and time inexperienced workers.12 Some of these include again.14 As Figure 2 shows, in the past, aircraft costly re-work due to poor quality assurance design engineers would work on one or two and failure to realise planned production programmes at a time, with much greater efficiencies due to lower-than-planned frequency of starting new programmes. More workforce productivity or problems with the recently, there has been a 15-year gap between recruitment, training and security clearance the design phase of Typhoon (beginning 1985) pipeline. Both re-work and lower-than-assumed and the UK involvement in JSF design (about productivity naturally have an impact on 2000). Between 2000 and the early work on the programme cost and schedule. Tempest, almost twenty years have passed RAND’s frequently cited and critically since design engineers had to exercise their acclaimed analysis of lessons learned from skills on a full programme. past US, UK and Australian submarine Rebuilding, retraining, recruiting or sourcing acquisition programmes clearly shows these skills from sources external to the that long gaps between the design stages programme (or seeking to bring in external of successive programmes can have an subcontractors or partners to fill known

11 Retter et al. (2015); Silfversten et al. (2018); Galai et al. (2019); Muravska et al. (2020). 12 Schank (2011); Australian NAO (2014). 13 Schank et al. (2011). 14 Bassford et al. (2010), Jenkins et al. (2015); Black et al. (2018). 8 Persistent Challenges in UK Defence Equipment Acquisition

Figure 2. A 40-year career of a design engineer in the UK defence aerospace industry (duration of past UK combat aircraft programmes)

Gloster Javelin - 1951

Hawker Hunter - 1951

English Electric Lightning - 1954

Buccaneer UK - 1958

McDonnell Douglas Phantom - 1958

Hawker Siddeley Harrier - 1967

SEPECAT Jaguar - 1968

British Aerospace Hawk -1974

Panavia Tornado - 1974

BAe/McDonnell Douglas Harrier II - 1978

Eurofighter Typhoon - 1985

Joint Strike Fighter - 2000

1950 1960 1970 1980 1990 2000 2010 2020 2030 2040 2050

Source: Jenkins et al. (2015).

gaps) can be costly and time-consuming, low-observable materials or supersonic engine and can jeopardise the programme’s overall design), limiting transferability. performance.15 Furthermore, many of the To the authors’ best knowledge, there is no critical skills on defence programmes may periodic review of the UK defence acquisition not have any analogues in the civil sector workforce as a whole, undertaken to (e.g. there is no need for commercial provide a regular (rather than a one-off) and aviation to develop a workforce with skills in comprehensive assessment of the quantity

15 For example, such as during the Astute programme when Electric Boat engineers were brought in to help. 9

and quality of the workforce to perform all its incentives to employees. Yet it is unclear where necessary functions. This includes ensuring the most significant gaps and shortages in SQEP for contracting, design approval, test and skills are, how intractable they may be, or evaluation, project management and many what exactly the long-term risks are that these other activities vital to successful programme shortfalls introduce into individual programmes delivery. Most likely, the picture is quite or the overall Defence Equipment Plan. diverse and varies across different capability areas, with a more in-depth understanding Contracting, incentives and of the skills picture available for areas where supplier performance sovereign capability is required (e.g. complex weapons, submarines, combat aircraft). Other The commercial strategy employed for an areas are not helped by the unique nature acquisition programme has a significant of the defence workforce split between civil influence on programme performance and the servants and military personnel, who routinely effective delivery of capability. Since different rotate postings, with implications for corporate contract types suit different acquisition memory and skill atrophy. Undoubtedly, strategies and represent varied approaches the MOD faces stark competition for key to risk sharing between MOD and industry, acquisition skills against the private sector, an appropriate type of contract needs to which may be able to offer a more attractive be used. Table 1 summarises the main package of both financial and non-financial different types of contracts and the market

Table 1. Principal contract types and uses in major acquisition projects

Contract type Description Use in acquisition

FIRM PRICE Contractor is responsible for More appropriate for procuring performance with a specified price off-the-shelf systems and platforms

TARGET COST INCENTIVE FEE Contractor is responsible for More appropriate when more performance under a target price units are produced and there is with a target profit incentive an opportunity for the contractor to gain efficiency savings and the government to share in savings; more complicated to administer than a firm price contract

COST-PLUS FIXED FEE Contractor receives a More appropriate for predetermined fee developmental programmes or when costs cannot accurately COST-PLUS INCENTIVE FEE Contractor receives a smaller or be estimated; typically larger fee based on how it performs requires greater oversight and against cost or performance targets administration

COST-PLUS AWARD FEE Contractor receives an award fee for meeting specific performance targets

Source: RR-767 (2011). 10 Persistent Challenges in UK Defence Equipment Acquisition

environment in which they are most suitable; of the most critical to effective programme for example, whether they are better suited for delivery concerning: competitive purchase of off-the-shelf systems • Design maturity and integration challenges. or development of indigenous capability • Technology maturity. with domestic suppliers (i.e. in a monopoly– • Industrial base capabilities and capacity monopsony market structure). (including facilities and other delivery Problems often occur in acquisition commitments). programmes when the commercial • Workforce productivity. relationship and incentive structure between • The ability of the MOD customer to deliver the MOD and suppliers is configured required inputs, including Government with the potential to adversely impact Furnished Assets (GFX), decision making programme delivery and embedded in a and timely approvals. contract. More specifically, two main types of Misaligned assumptions often mean that at structural challenges to aligning contractual least one side may not be making programme arrangements with acquisition approaches decisions based on the best available evidence, have been identified by RAND reports and undermining the realism of cost and timeline experts, including misaligned assumptions estimates, which are then taken forward as a and inappropriate apportionment of risk. baseline for the delivery of the project. Right from the outset, therefore, the programme may While robust assumptions are inevitably be starting with erroneous assumptions. This difficult to get right, it is important that increases the likelihood of cost growth and they are accepted and shared between schedule slippage, while the communication of MOD and the suppliers unrealistic expectations to political leadership No matter how well put together, any capability and end users can undermine confidence in delivery plan and acquisition strategy can the MOD and cascade problems onto other only be based on a series of assumptions, interdependent programmes.16 as are the accompanying baseline cost and schedule estimates. These assumptions may Understanding of the causes of risk is a have different levels of certainty attached prerequisite for effective apportionment to them and may be the product of more or of responsibility for different risk types, less sophisticated modelling and estimation but it is often lacking methods. Whether they are robust or poor, it The issue of misaligned assumptions is is crucial that these assumptions are jointly linked to the inherent level of risk involved understood by both the customer and supplier in any defence equipment programme that and that they are continuously interrogated includes at least some element of design as new data and learning from experience and development work. These risks concern become available. Misaligned assumptions technical, design, integration and business occur in a wide range of areas, with some

16 RAND Europe interview for Retter et al. (2017). 11

Astute Class submarine Audacious under construction at Barrow in Furness shipyard in Cumbria. Crown Copyright Crown

elements,17 and are not always fully understood under their control (e.g. productivity, labour by the MOD or even by the supplier(s). This and material costs) and the government lack of understanding then makes risk responsible for those risks outside of the management less effective and can result in contractors’ control (e.g. requirements a mismatch between risk sharing approaches changes, legal changes, etc.). Coming to an and contractual arrangements and incentives. effective division like this, however, requires As a result, cost overruns, schedule slippages a solid understanding of the causes of risk, and quality issues may be difficult to identify, not just the consequences in terms of cost, foresee, track, quantify and address, also schedule and performance. Additionally, there because liability can be difficult to apportion may be a disconnect between the theoretical and there may be limited visibility for the MOD allocation of risk through the contract and the to see what is happening in the supply chain practical (and political) reality and the ability below the prime contractor level. of MOD to enforce this. As witnessed in the Astute programme, the hollowed out industrial Successful contracting strategies hold capability to deliver the programme meant contractors responsible for those risk factors that the MOD had to take on more risk than

17 Technical risk is the exposure to the chance that development of critical technologies does not meet programme objectives within the delivery envelope (cost, schedule, performance). Design risk is the exposure to the chance that the weapon system’s design will not result in effective operation or be easy to produce. (System) integration risk is the exposure to the chance that new and existing technologies employed in the weapon system do not work together and/or interact with operators and maintainers in way that enables effective programme delivery on schedule and on budget. Business risk is the exposure to the chance that the overall acquisition strategy and decisions about the process to select the winning bidder, standards to which the system is required to be built and schedules for design and build do not lead to effective programme delivery on budget, on schedule and meeting the performance requirements (Source: Murphy & Birkler 2009). 12 Persistent Challenges in UK Defence Equipment Acquisition

initially envisaged to ensure the programme Royal Navy (RN) and the Royal Air Force (RAF), could be delivered at all.18 In other situations, and potentially the joint Strategic Command), suppliers may be placed in a position to extract constant adjustments to programme delivery favourable terms (but potentially detrimental to management and systemically insufficient MOD) by virtue of being the preferred or indeed financial provisions for risk. remaining bidder in contract negotiations. Similarly, there may be a theoretical Imbalance between Services undermines apportionment of risk in the contract that the opportunities to use defence equipment MOD may not be able to enforce as it could budgets for modernisation efforts across result in significant financial harm to the all Services supplier, which may have a wider impact on the There is an inherent imbalance between the industrial base and industrial capability needed Services and domains in terms of weapon for any future programmes. system acquisition requirements; the RN Clarity at the contracting phase is therefore and the RAF arguably take a more platform- essential. The NAO has previously found centric approach (and have benefited from the that poor definition of responsibilities left the bulk of capital spending in recent decades), government open to compensation claims whereas the Army is centred more around the from industry. Such issues can similarly capabilities of its personnel, with equipment undermine trust and influence behaviours programmes being essentially incremental between the MOD and its suppliers on other or improvements to hull structures that see little future programmes; this means that a poorly fundamental change between generations.19 designed contract in one programme might From a technological perspective, weapon damage long-term relations to the detriment system acquisition programmes for the of other areas of delivery or even the overall Army tend to have shorter lead times and defence industrial strategy. are generally less complex than systems acquired for the RN and RAF, while being Programme management procured in larger platform numbers. Yet, the approach majority of the defence equipment budget tends to be allocated to long-term RN and RAF Much of what has been said so far concerns programmes. With costs for some high-profile problems that are experienced to some degree Army modernisation programmes remaining by any national defence acquisition system and uncommitted, while budgets for RN and RAF have been highlighted by RAND studies covering long-term programmes become already different systems in the UK, Europe, Australia, committed costs, funding for Army weapon Canada and the US. In this section, we examine system modernisation programmes may not factors that are specific to the UK’s defence materialise. There are, of course, strategic acquisition system and present underlying arguments in favour of funding Air and Sea challenges for effective delivery of equipment capabilities considering the UK’s geographic programmes. These include competition and defence posture, but the resulting between the Services (the British Army, the budgetary imbalance between the Services

18 Schank et al. (2014). 19 RAND Europe interview with SME, January 2021. 13

also stems from the different nature of settlement. This can lead to a narrow focus equipment programmes highlighted above and on in-year affordability. In practice, this the Services’ approach to pursuing funding for creates a system that is constantly ‘in flux’ them. The combined impact of these factors as frequent adjustments to programme could inhibit modernisation efforts across delivery management need to be made to all Services and make it difficult to react to a ensure affordability in line with the Annual changing threat environment. Budget Cycle. There is friction between the strategic level financial management and The adverse impact on the overall Defence equipment programme budgets, requiring Equipment Plan is further reinforced by the frequent adjustments to the latter to ensure MOD’s reluctance to cancel programmes that that overall, the UK government’s books fail to meet affordability requirements already are balanced year-on-year, and that any in early programme stages. RAND’s previous affordability issues are effectively pushed studies on cost growth and cost escalation down to subsequent years. As such, defence have noted the importance of national MODs acquisition professionals spend a significant not only identifying early which projects are amount of time on re-profiling programmes, unaffordable but also being willing and able with a knock-on impact on delivery schedule to cancel them as early as possible; citing and ultimately on overall acquisition cost. the example of the US Better Buying Power This constant adjustment and readjustment acquisition reform initiative, which mandates of programme delivery thus creates second- affordability as part of the requirement. A well- order costs and narrows down the programme known example of this policy put in practice managers’ flexibility to actively manage risk as is the case of the US Army Future Combat well as to drive long-term efficiencies. It also System, which was cancelled in 2009 after creates uncertainties for long-term programme seven years due to insufficient analysis of delivery that often translate into further costs alternatives and inability to meet affordability in the long run. To give a concrete example, criteria.20 Too often, however, defence programme managers may be restricted equipment programmes are easy to start but in their ability to commit upfront funding to too difficult to stop, given the political capital, minimise cost in the long run, for example by personal careers and business interests vested pre-ordering or bulk buying larger sets of units. in their continuation and the powerful allure of the ‘fallacy of sunk costs’. The perpetual development of options, and therefore the shifting of targets and milestones, Frequent adjustments to programme results in changes to the production drumbeat. delivery undermine its overall This disrupts the manufacturing process and effectiveness throughput, jeopardising even the limited production efficiencies that can be achieved Despite the progress made with the in defence manufacturing as discussed in a introduction of long-term equipment funding previous section. Furthermore, these frequent plans during Bernard Gray’s tenure as Chief adjustments can often result in penalty costs of Defence Materiel, the wider MOD budget for the MOD for opening up the contracts and remains vulnerable to annualisation given introducing contract variation due to changes the lack of a long-term, multi-year financial

20 Retter et al. (2017). 14 Persistent Challenges in UK Defence Equipment Acquisition

HMS Queen Elizabeth being manoeuvred by tugs as she sails from the Northern Ammunition Jetty in Glen Mallen. Crown Copyright Crown

to the delivery plan, with short-term payouts stemming from a poor understanding of for the industry but long-term disruption to the risk, and also from the cross-cutting issues overall delivery of the programme. This can described below pertaining to the optimism negate any benefits that were achieved at the bias and moral hazard. As a result, the level initial contract negotiation (for example, via a of the risk provision is often inadequate even competitive bidding process). when some is held centrally at a portfolio level. RAND’s analysis of past international Budgeting sufficient contingency for programmes suggests a pool of 10–15 per cent risk is appreciated in theory but not of the initial budget estimate should be held implemented in practice as a contingency, as that is roughly the cost burden that is attributable to risks materialising This paper has consistently emphasised the in the programme’s duration.21 importance of understanding and managing In practice, the risk as critical aspects of an effective funding allocation for the programme risk pool acquisition system. On the theoretical level, this has often been much smaller. is appreciated by MOD, as well as its principal suppliers. However, in practice, many individual Cross-cutting issues equipment programmes have insufficient The previous sections have focused on risk provision. As described above, allocating areas of most interest to the NAO for this sufficient contingency funds to enable effective discussion paper. In this final section, we cover risk management is fraught with obstacles those factors that cut across capability and

21 Schank et al. (2014). 15

skills, supplier relationship and programme Systemically, therefore, programme budgets management aspects. These cross-cutting do not appear to incorporate historically elements include: derived risk assumptions, which consistently leads to overruns and can sometimes lead • Perennial optimism bias and to underruns, which are equally disruptive to under-budgeting. programme management. These then either • Lack of institutional memory hampering need to be balanced out elsewhere (e.g. learning from the past. delaying activity into the future or delaying or • The presence of moral hazard whereby cancelling other programmes) or ‘bailed out’ consistently underperforming programmes through a boost in funding (see the section on continue to receive funding. moral hazard). This systemic underfunding These elements are both intertwined with, often leads to delays in capability delivery, and less tangible than, the various structural, which can result in capability depreciation and procedural and contractual elements discussed undermine the ability of Defence to contribute above. Nonetheless, they are just as important to the delivery of National Security Objectives. for understanding equipment programme At the same time, this paper acknowledges issues, if not more so. Tinkering with the many that cost and risk analyses require judgements ‘tactical’ problems facing programme delivery that are based on incomplete and uncertain may be insufficient if these more foundational information, making objective and robust ‘strategic’ issues are not also addressed. assessment difficult to start with. In addition, A culture of optimism permeates defence as explained by behavioural economics, those equipment programme decision making, cost estimators and decision makers expected distorting assumptions and planning to assess cost and risk do not always act outcomes ‘rationally’ but are instead subject to a variety of biases: anchoring and adjustment bias, 22 23 The Gray report, the Levene report and availability bias, ambiguity aversion, framing several NAO and RAND reports all consistently bias and others.25 As these biases cannot be identify a culture of optimism – ‘a conspiracy of eradicated, it is important to acknowledge and optimism’ in fact – that permeates the defence mitigate their impact, as discussed in the final acquisition system. Incentives exist for project section. teams to be optimistic on programme cost to increase the chance of funding, while MOD Lack of institutional memory means that customers and industry are also incentivised to lessons from the past are not learnt as assume the programme will run smoothly. This quickly and efficiently as they could be – is demonstrated by the fact that most projects or not learnt at all are budgeted for at the 50 per cent confidence Long programme cycles, combined with long interval even though historical evidence gaps between equipment programmes, staff suggests that, on average, project costs turn postings and general staff turnover have out to be at the 70 per cent confidence level.24

22 Gray (2009). 23 Levene (2011). 24 RAND Europe interview with SME, January 2021; also see Hemsley & Holden (2014). 25 Arena et al. (2006). 16 Persistent Challenges in UK Defence Equipment Acquisition

had a negative impact on the MOD – and to would be needed to robustly examine the some degree also on industry – in terms of key factors driving these cycles, overall the their ability to retain institutional memory. MOD seems to oscillate between periods with Numerous past RAND analyses have identified tighter budgeting and discipline and periods the lack of institutional memory, knowledge where budget discrepancies increase. As with capture and sharing on past programme other organisational change processes and performance as important contributors to reforms, the strength of character and firm inefficiencies on follow-on programmes. leadership play an important role in leading There is similarly a reduced incentive for by example and embedding programme managers to make decisions that might have discipline in practice. Availability, motivation a negative short-term impact (e.g. in terms of and strong performance of the workforce at their individual career prospects) but deliver all levels of the acquisition process in both substantial returns over a long-term period. MOD and industry are likely to be important While developing novel weapon systems may as well. Strong and independent contestability present unique technical challenges that the functions providing continuous challenge and MOD has not been encountered on previous validation represent another critical element of programmes, there are always valuable tackling the moral hazard.27 transferrable lessons that can be learnt from past programmes. Although implementing Areas for further consideration such learning has historically proven challenging, practical measures, processes and Sustainable, sufficiently resourced cultural changes that channel past learnings and well communicated measures are into new programmes can help. needed to address persistent acquisition challenges The UK defence acquisition system is The Gray review and the Levene report both prone to moral hazard whereby poor identified many systemic failures within delivery results in only limited negative the defence acquisition system. They also consequences identified recommendations for change. Sir Finally, there is a persistence with which Bernard Gray was appointed as the Chief of underperforming programmes continue to Defence Materiel to drive the implementation receive funding and, linked to the conspiracy of of some of these. Both reform initiatives optimism, the MOD continues to run a budget recognised the need for systemic measures deficit vis-à-vis its 10-year Equipment Plan.26 and broader cultural and organisational Only a few programmes are cancelled if they change, rather than ad hoc initiatives to are deemed unaffordable, creating a moral mitigate the challenges. Based on past RAND hazard whereby the individual equipment analysis and knowledge of the UK’s defence programmes, as well as MOD as a whole, take acquisition system, the authors have identified on more risk than it can afford because it is the following examples of measures that unlikely to have to bear the full exposure to the could help address seemingly intractable risk. Although a separate historical analysis

26 NAO (2021). 27 Cook et al. (2016). 17

performance issues on defence equipment of defence programmes. This requires programmes: specialised skillsets that do not routinely reside in the military (given the nature • Establish and embed effective challenge, of the Forces’ priorities). Therefore, an scrutiny and red teaming processes: emphasis should be placed on ensuring Acquisition best practice suggests that that a) capability costing and budgeting ‘chaos’ (i.e. red teaming and challenging with which the Financial and Military of underlying assumptions) should be Capability (FMC) organisation is tasked introduced at the early stages of the and b) MOD programme and portfolio programme set-up in order to identify delivery with which Defence Equipment & weak assumptions, appropriately assess Support (DE&S) is charged are carried out risk and prevent unrealistic estimates by a corps of programme management, from becoming contractual milestones. financial and accounting professionals. Continuous, in addition to initial, challenge This can begin with a comprehensive audit and validation is also essential, not least of acquisition skillsets and capabilities, as a way of tackling the moral hazard.28 which allows a granular view across the While such functions are already in place equipment portfolio and in individual (e.g. the Cost Assurance and Analysis programmes. This exercise should Service (CAAS), the Project Controls, the be repeated regularly, allowing also Investment Approval Committee (IAC), longitudinal analysis and measures of the NAO) their full effectiveness can the effectiveness of different initiatives be hampered by a lack of institutional to address gaps and shortfalls. Similarly, memory, frequent organisational well-functioning independent cost and risk changes and insufficient resourcing in analysis and assurance have an important terms of both funding and people. It is role to play in mitigating the adverse important to ensure that both internal and impact of optimism and other biases. external challenge and scrutiny roles are The development and growth of CAAS in established and allocated to individuals’ particular (now sitting under DG Finance), areas of responsibility – an informal or while intended for this purpose, has so far ‘part-time’ scrutiny function is unlikely to not provided a powerful enough challenge suffice. Tolerance of and responsiveness to the ‘conspiracy of optimism’. Continuous to these challenge measures, however, is reorganisation of CAAS and challenges essential but also difficult to implement; with recruiting and retaining permanent this may only be achieved with a SQEP have also undermined continuity. deliberate cultural shift and signalling from leadership. • Drive a focus on risk management and understanding: Adequately understanding • Professionalise, reinforce and enable and anticipating equipment programme programme management and cost risk is in large part a function of adequately assurance functions: This short discussion capturing and implementing lessons from paper has underscored the importance past programmes, including regarding of strategic financial risk and project realistic levels of structural risk, entrusting management to the effective delivery

28 Cook et al. (2016). 18 Persistent Challenges in UK Defence Equipment Acquisition

risk management to personnel with These steps need to then filter into future sufficient professional experience and programmes via a clear mechanism, subject matter knowledge, and acting rather than individuals’ willingness and ad upon the recommendations of those hoc arrangements. This can be done by charged with scrutiny and challenge. These conducting an ex-post analysis, capturing elements have already been discussed lessons learnt and sharing the analysis in this section. However, adequate risk widely among MOD and industry actors management is also conditional on involved in the new programme. Also, aligning the wider MOD budgeting norms senior experts and stakeholders who had and processes to the needs of equipment been involved in similar past projects can programmes, which need to be stably be brought into the project team to provide resourced, and as such are not compatible independent validation and challenge of with annual budget cycles and in-year assumptions. affordability. Finally, programme oversight This paper has attempted to highlight the should be based on risk, rather than core drivers that give rise to intractable monetary value. defence programme delivery challenges but • Embed clear processes to capture, has only been able to scratch the surface of share and feed through lessons learned: this notoriously complex issue, which sees To mitigate the impact of the lack of strategic aspects of national defence and force institutional memory and insufficient projection intersect with the democratic need risk management and scrutiny, it is to deliver value for taxpayers’ money. However, important to not only understand what it is hoped that the reflections and analysis has gone wrong in the past but also presented here will provoke further discussion develop processes and measures to guard within NAO and MOD about how to address against similar issues arising in the future. these important and pressing issues. 19

References

Arena, Mark V., Irv Blickstein, Obaid Younossi & Corporation. MG-1020-OSD. As of 2 March 2021: Clifford A. Grammich. 2006. Why Has the Cost of http://www.rand.org/pubs/monographs/MG1020 Navy Ships Risen? A Macroscopic Examination Birkler, John, Denis Rushworth, James Chiesa, of the Trends in U.S. Naval Ship Costs Over the Hans Pung, Mark V. Arena & John F. Schank. Past Several Decades. Santa Monica, Calif.: RAND 2005a. Differences Between Military and Corporation. MG-484-NAVY. As of 2 March 2021: Commercial Shipbuilding: Implications for the http://www.rand.org/pubs/monographs/MG484 United Kingdom’s Ministry of Defence. Santa Arena, Mark V., Obaid Younossi, Kevin Brancato, Monica, Calif.: RAND Corporation. MG-236-MOD. Irv Blickstein & Clifford A. Grammich. 2008. Why As of 2 March 2021: Has the Cost of Fixed-Wing Aircraft Risen? A https://www.rand.org/pubs/monographs/MG236 Macroscopic Examination of the Trends in U.S. _____. 2005b. Diversifying the Customer Base for Military Aircraft Costs over the Past Several Decades. Shipbuilding in the United Kingdom. Santa Monica, Santa Monica, Calif.: RAND Corporation. MG-696- CA: RAND Corporation. RB-9117-MOD. As of 2 NAVY/AF. As of 2 March 2021: March 2021: http://www.rand.org/pubs/monographs/MG696 https://www.rand.org/pubs/research_briefs/ Arena, Mark V., Obaid Younossi, Lionel A. Galway, RB9117 Bernard Fox, John C. Graser, Jerry M. Sollinger, Black, James. 2018. Lessons Learned from Felicia Wu & Carolyn Wong. 2006. Impossible SSBN Sustainment. Santa Monica, Calif.: RAND Certainty: Cost Risk Analysis for Air Force Corporation. Systems. Santa Monica, CA: RAND Corporation. MG-415-AF. As of 2 March 2021: Black, James, Lucia Retter, Alexandra Hall, Alice https://www.rand.org/pubs/monographs/MG415 Lynch, Antonia Ward, Bhanu Patruni & William Phillips. 2018. Future of the UK Combat Air Australian National Audit Office (NAO). 2014. ‘Air Sector, Understanding the Policy and Industrial Warfare Destroyer Program; Audit Report No. 22 Implications of UK Options for Combat Air to 2013–14’ Australian National Audit Office. 2040+. Santa Monica, Calif.: RAND Corporation. Bassford, Matt, Hans Pung, Nigel Edgington, Tony (Not publicly available.) G. Thompson-Starkey, Kristen Weed, Mark V. Black, James, Lucia Retter, Alice Lynch & Douglas Arena, James G. Kallimani, Gordon T. Lee & Obaid Barrie. 2018. Future of UK Combat Air 2040. Younossi. 2010. Sustaining Key Skills in the UK DCDC. (Not publicly available.) Military Aircraft Industry. Santa Monica, Calif.: RAND Corporation. MG-1023-MOD. As of 2 March 2021: Cook, Cynthia R., Emma Westerman, Megan https://www.rand.org/pubs/monographs/MG1023 McKernan, Badreddine Ahtchi, Gordon T. Lee, Jenny Oberholtzer, Douglas Shontz & Jerry M. Sollinger. Birkler, John, Mark V. Arena, Irv Blickstein, Jeffrey 2016. Contestability Frameworks: An International A. Drezner, Susan M. Gates, Meilinda Huang, Horizon Scan. Santa Monica, Calif.: RAND Robert E. Murphy, Charles Nemfakos & Susan Corporation. RR-1372-AUS. As of 2 March 2021: K. Woodward. 2010. From Marginal Adjustments http://www.rand.org/pubs/research_reports/ to Meaningful Change: Rethinking Weapon RR1372 System Acquisition. Santa Monica, Calif.: RAND 20 Persistent Challenges in UK Defence Equipment Acquisition

Flyvbjerg, Bent, Nils Bruzelius & Werner Murphy, Robert E., & John Birkler. 2009. Dollar Value Rothengatter. 2003. Megaprojects and Risk: An and Risk Levels: Changing How Weapon System Anatomy of Ambition. Cambridge: Cambridge Programs Are Managed. Santa Monica, Calif: RAND University Press. Corporation. OP-264-OSD. As of 2 March 2021: https://www.rand.org/pubs/occasional_papers/ Flyvbjerg, Bent. 2014. ‘What You Should Know OP264 About Megaprojects and Why: An Overview.’ Project Management Journal 45(2): 6–19. National Audit Office (NAO). 2004. ‘Ministry of Defence: Major Projects Report 2003.’ Galai, Katerina, Lucia Retter, Julia Muravska, National Audit Office. As of 2 March 2021: Marta Kepe, Alice Lynch, Anna Knack, Jacopo https://www.nao.org.uk/wp-content/ Bellasio, Antonia Ward, Sofia Meranto, Davide uploads/2004/01/0304195es.pdf Maistro, Liga Baltina & Terence Hogarth. 2019. Vision on defence related skills for Europe today _____. 2005. ‘Driving the Successful Delivery of and tomorrow. Santa Monica, Calif.: RAND Major Defence Projects: Effective Project Control Corporation. As of 2 March 2021: is a Key Factor in Successful Projects.’ National https://eu-ems.com/event_images/ Audit Office. Downloads/1%20Main%20report.pdf _____. 2015. ‘Reforming Defence Acquisition.’ Gray, Bernard. 2009. Review of Acquisition for the National Audit Office. Secretary of State for Defence. An Independent _____. 2020a. ‘The Equipment Plan 2019 to 2029.’ Report by Bernard Gray. National Audit Office. As of 2 March 2021: Hemsley, Matt, & Helen Holden. 2014. ‘Helping the https://www.nao.org.uk/wp-content/ Nation Spend Wisely.’ SCAF. As of 29 January 2021: uploads/2020/02/The-Equipment-Plan- https://scaf.org.uk/wp-content/ 2019-to-2029.pdf uploads/2019/07/2014_06-Helping-the-nation- _____. 2020b. ‘Defence capabilities – delivering spend-wisely-Matt-Hemsley-and-Helen-Holden- what was promised.’ National Audit Office. NAO.pdf National Audit Office. As of 2 March 2021: Jenkins, D., L. Retter, J. Black, L. Taggart, A. Hall, https://www.nao.org.uk/wp-content/ S. Patel & J. Downes. 2015. Industrial Implications uploads/2020/03/Defence-capabilities-delivering- of the UK Future Combat Air System (FCAS). what-was-promised.pdf Santa Monica, Calif.: RAND Corporation. _____. 2020c. ‘Environmental Sustainability Levene of Portsoken, David Allen, Björn Conway, Overview.’ National Audit Office. As of 2 March 2021: Jon Day, Gerry Grimstone, General Sir Nick https://www.nao.org.uk/wp-content/ Houghton, George Iacobescu, Raymond McKeeve uploads/2020/05/Environmental-Sustainability- & Baroness Noakes. 2011. Defence Reform: Overview.pdf An independent report into the structure and _____. 2020d. ‘Carrier Strike – Preparing for management of the Ministry of Defence. London: deployment.’ National Audit Office. As of 3 March The Stationery Office Limited. 2021: Muravska, J., A. Knack, F. Quimbre, D. Dascalu, https://www.nao.org.uk/wp-content/ P. Paille, R. Allport & B. Williams. 2020. Defence uploads/2020/06/007678-001-Carrier-Strike- Supply Chain Development: An Assessment of the preparing-for-deployment.pdf Performance of UK Defence Supply Chains and _____. 2020e. ‘Lessons learned from Major of Current Supply Chain Development Initiatives. Programmes.’ National Audit Office. As of 3 Santa Monica, Calif.: RAND Corporation. PRJ- March 2021: A117-2. Prepared for the UK Ministry of Defence. https://www.nao.org.uk/wp-content/ (Not publicly available.) uploads/2020/09/Lessons-learned-from-Major- Programmes.pdf 21

_____. 2021. ‘The Equipment Plan 2020 to 2030.’ Schank, John F., Frank W. Lacroix, Robert E. National Audit Office. As of 3 March 2021: Murphy, Mark V. Arena & Gordon T. Lee. 2011. https://www.nao.org.uk/wp-content/ Learning from Experience: Volume I: Lessons from uploads/2021/01/The-Equipment-Plan-2020- the Submarine Programs of the United States, 2030-Report.pdf United Kingdom, and Australia. Santa Monica, Calif.: RAND Corporation. MG-1128/1-NAVY. As of Powers, James D., Guy Weichenberg, Abby Doll, 2 March 2021: Thomas Goughnour, Thomas Light, Mark A. https://www.rand.org/pubs/monographs/ Lorell, Ellen M. Pint, Soumen Saha, Fred Timson, MG1128z1 Thomas C. Whitmore & Robert A. Guffey. 2018. F-35 Block Buy: An Assessment of Potential Schank, John F., Cesse Cameron Ip, Frank Savings. Santa Monica, Calif.: RAND Corporation. W. Lacroix, Robert E. Murphy, Mark V. Arena, RR-2063-AF. As of 2 March 2021: Kristy N. Kamarck & Gordon T. Lee. 2011. https://www.rand.org/pubs/research_reports/ Learning from Experience: Volume II: Lessons RR2063 from the U.S. Navy’s Ohio, Seawolf, and Virginia Submarine Programs. Santa Monica, Calif.: RAND Pung, Hans. 2008. Sustaining Key Skills in the Corporation. MG-1128/2-NAVY. As of 2 March UK Naval Industry. Santa Monica, Calif.: RAND 2021: Corporation. RB-9349-RE https://www.rand.org/pubs/monographs/ Retter, L., L. Taggart & J. Freeman. 2015. Key MG1128z2 skills and competences for defence: Executive Schank, John F., Frank W. Lacroix, Robert E. Summary. Santa Monica, Calif.: RAND Murphy, Cesse Cameron Ip, Mark V. Arena & Corporation. RR-1226-EDA. As of 2 March 2021: Gordon T. Lee. 2011 Learning from Experience: https://www.rand.org/pubs/research_reports/ Volume III: Lessons from the United Kingdom’s RR1226 Astute Submarine Program. Santa Monica, Calif.: Retter, L., E. Silfversten & D. Jenkins. 2017. RAND Corporation. MG-1128/3-NAVY. As of 2 Response strategies to the cost escalation of March 2021: defence equipment. Santa Monica, Calif.: RAND https://www.rand.org/pubs/monographs/ Corporation. MG1128z3 Rich, Michael D., Edmund Dews & C. L. Batten. 1986. Schank, John F., Cesse Cameron Ip, Kristy N. Improving the Military Acquisition Process: Lessons Kamarck, Robert E. Murphy, Mark V. Arena, Frank from Rand Research. Santa Monica, Calif.: RAND W. Lacroix & Gordon T. Lee. 2011. Learning from Corporation. R-3373-AF/RC. As of 2 March 2021: Experience: Volume IV: Lessons from Australia’s https://www.rand.org/pubs/reports/R3373 Collins Submarine Program. Santa Monica, Calif.: Schank, John F., Jessie Riposo, John Birkler RAND Corporation. MG-1128/4-NAVY. As of 2 & James Chiesa. 2005. The United Kingdom’s March 2021: Nuclear Submarine Industrial Base, Volume 1: https://www.rand.org/pubs/monographs/ Sustaining Design and Production Resources. MG1128z4 Santa Monica, Calif.: RAND Corporation. Silfversten, Erik, Alex Hall, Marta Kepe & Richard MG-326/1-MOD. As of 2 March 2021: Flint. 2018. Key Skills and Competences for http://www.rand.org/pubs/monographs/MG326z1 Defence: Governmental Domain. Santa Monica, Schank, John F., Mark V. Arena, Kristy Kamarck, Calif.: RAND Corporation. EP-67674. As of 2 Gordon T. Lee, Robert E. Murphy & Roger March 2021: Lough. 2014. Keeping Major Naval Acquisitions https://www.rand.org/pubs/ on Course: Key Considerations for Managing external_publications/EP67674 Australia’s SEA 5000 Future Frigate Programme. Santa Monica, Calif.: RAND Corporation. RR-767. 22 Persistent Challenges in UK Defence Equipment Acquisition

Younossi, Obaid, Mark A. Lorell, Kevin Brancato, Younossi, Obaid, Mark V. Arena, Robert S. Cynthia R. Cook, Mel Eisman, Bernard Fox, John Leonard, Charles Robert Roll, Arvind Jain & C. Graser, Yool Kim, Robert S. Leonard, Shari Jerry M. Sollinger. 2007. Is Weapon System Cost Lawrence Pfleeger & Jerry M. Sollinger. 2008. Growth Increasing? A Quantitative Assessment of Improving the Cost Estimation of Space Systems: Completed and Ongoing Programs. Santa Monica, Past Lessons and Future Recommendations. Calif.: RAND Corporation, MG-588-AF. As of 2 Santa Monica, Calif.: RAND Corporation. March 2021: MG-690-AF. As of 2 March 2021: http://www.rand.org/pubs/monographs/MG588 http://www.rand.org/pubs/monographs/MG690