Defence Fire and Rescue Project Independent Review and Lessons Learned September 2019 © Crown copyright 2019 Contents
Background, Conclusions and Recommendations 1 Background 1 Conclusions on Key Findings 2 Recommendations on Lessons to be Learned 4
Fuller Description and Assessment 7 Methodology for the Evaluation 7 Evaluator Training 9 Did MOD Follow the Stated Evaluation Process? 9 Discrete Risk and Pricing Adjustments 11 Resourcing of the Evaluation 13 Moderation Process 13 Summarising and Explaining Evaluation Results 14 Use of the Financial Model 14 Adjustments made by Bidders in the Final Stage to Address Affordability 15 Governance, Assurance and Approvals 16 Use of External Legal and Financial Advisers 21 Issuing of Award Letters and Related Information Provided to Bidders 22
Annex A: Terms Of Reference - Summary 25
Independent Review and Lessons Learned 3 Defence Fire and Rescue Project Background, Conclusions and Recommendations
Background 4. The competition was run under the Defence and Security Public Contracts Regulations 1. Fire and rescue services for the Ministry (DSPCR) 2011. Thirty companies expressed of Defence (MOD, or the Department) are an initial interest, of which eight submitted a currently provided both on permanent bases pre-qualification questionnaire. In December and on operations by Defence Fire and 2014, four passed pre-qualification. They Rescue (DFR, formerly called DFRMO – the submitted outline solutions with indicative Defence Fire Risk Management Organisation). costs in April 2015. Alongside the outsourcing Approximately 1,900 personnel are drawn option, MOD also developed an in-house from the military, MOD civil servants, locally value for money benchmark against which to employed civilians overseas and contractors. compare proposals.
2. The Defence Fire and Rescue Project (DFRP 5. Following three rounds of dialogue, in or the project) arose out of the Strategic September 2016 the IAC considered the Defence and Security Review in 2010, when project’s Main Gate Business Case.2 It gave a decision was made to seek efficiencies approval in March 2017 to call for final in a number of areas. One was to consider tenders for a 12-year contract once the outsourcing the DFR service. The project to bids were sufficiently mature.3 By this time, do so was initially led by a central MOD team. two bidders had withdrawn, leaving two In 2013, when the project received its initial companies bidding: Capita Plc (Capita) and formal approval, responsibility was transferred Serco Group Plc (Serco). The competition to the Army. A project team was established was important to both bidders, both because under a director-level civil servant who of its size and 12-year duration and because worked for the Adjutant-General and to whom the winner would become the dominant the role as the Senior Responsible Owner provider for Fire and Rescue services across (SRO)1 was later delegated. MOD as a whole.
3. In 2014, the Department’s official-level 6. The invitation to submit final tenders was Investment Approvals Committee (IAC) issued in July 2017 and the companies agreed a procurement process using submitted final bids in September 2017. Final competitive dialogue. This involves a series of tender evaluation took place from September structured bilateral discussions with bidders 2017 to January 2018. to shape a solution before holding a final tendering round, and gives flexibility to clarify 7. On 15 January 2018, the project sought elements of the requirement and propose approval to award the contract to Capita. solutions during the procurement. At that However, on the same day Carillion, a major stage, the team estimated that contract award contractor for outsourced government would take place in February 2017. services, went into liquidation. As a result,
1 The senior responsible owner (SRO) is accountable for ensuring that a programme or project meets its objectives, delivers the projected outcomes and realises the required benefits. He or she also typically chairs the project board and is responsible for ensuring that the right expertise and governance is in place. 2 The document that sought approval to issue an invitation to submit final tenders and for the expected time, cost and performance parameters for the project. 3 IAC approval dated 3 March 2017.
Independent Review and Lessons Learned 1 under arrangements coordinated by the that should be learned, focusing on the final Cabinet Office covering all major contracts, stage of the process between September the Department undertook a further 2017 and the issue of Serco’s claim in July thorough review of aspects of the project. 2018. Our terms of reference are at Annex A. This meant there was an extended period before contract award. 12. We have not carried out a review of the process before September 2017 but have 8. Approval to award the contract to Capita referred to some earlier events where was eventually given within Government on relevant. Nor have we carried out a forensic 14 June 2018.4 The projected value of the investigation or audit; or considered the contract to Capita over the 12-year contract comparison of the bids to the public sector period was then around £500m at outturn option, the due diligence carried out by MOD prices, while the project as a whole was into the financial sustainability and technical assessed to cost in total some £1.4bn at expertise of the bidders, or MOD’s dealings outturn prices. This total included continuing with Serco and Capita since July 2018. Our MOD costs - mainly from military and locally work was carried out between June and employed overseas personnel who would August 2019, based on documents provided remain MOD employees. by MOD and interviews with a range of those involved from MOD, Serco, Capita and 9. On 18 June 2018 a written ministerial Deloitte. This timetable allowed meetings statement was laid before Parliament to with Serco and Capita that could not be held announce the award and the two bidders until after the contract was awarded in July. were sent letters notifying them of the result and of the overall scores and costs of both Conclusions On Key Findings tenders. On 22 June Serco sent a pre-action protocol letter to MOD alleging, amongst 13. Our conclusions are: other things, breaches in the evaluation of both bids and lack of transparency in cost C1 We have found nothing to show that there evaluation. They sought an extension of was actual bias in the treatment of the final the standstill period ahead of the award two bidders to make the procurement open of contract and disclosure of further to challenge. information. Serco were dissatisfied with the C2 The evaluation methodology was fit for response received, and on 17 July issued purpose. We see no fundamental flaws in formal legal proceedings challenging the it, or in the financial model used for the decision. This automatically suspended the evaluation. award of the contract. C3. However, there were several deficiencies 10. The suspension was lifted in May 2019 in the conduct and recording of the final following an MOD application to the court. evaluation that did make the procurement On 18 July 2019 MOD announced that it open to challenge: had completed arrangements to award the contract to Capita at an outturn price of a. The criteria for evaluating MOD’s £525m and that the litigation with Serco had confidence in the technical elements of been settled out of court. the bids were described too generically. Technical subject matter experts should 11. As a result of the challenge to the have been more involved in preparing the procurement, in June 2019 the Permanent evaluation criteria and the application of Secretary of MOD asked a small team from the criteria should have been tested more outside the Department to carry out this fully before the process went live; independent review to establish the lessons
4 DFRP Treasury Clearance Note approval 14 June 2018.
2 Defence Fire and Rescue Project b. Evaluation records often used language C4. MOD ultimately decided to award the that did not properly reflect the scoring contract to Capita because it had higher criteria. Many referred to concerns about confidence in Capita’s solution and Capita’s bidder responses that nonetheless scored whole life cost was some 0.8% lower than ‘Good Confidence’ or ‘High Confidence’. Serco’s. We have found no evidence that On the face of it this should not have this position would have been reversed by been possible. It arose with both bidders, any of the individual deficiencies we have but there were more examples in Capita’s identified, and we consider it unlikely that scoring than in Serco’s; a more robust process would have led to a different outcome to the competition. But in c. In scoring some of Capita’s responses, our view the deficiencies were sufficient to moderation notes referred to reliance on justify MOD’s decision to reach a settlement the presence of “commercial levers” (i.e. with Serco, given the resource and cost that contractual measures to increase the would have been involved in defending the likelihood of things being done), which claim and the significant risk that the claim should not have been taken into account. would have been successful. We heard that this did not affect the scores awarded, but references to an C5. Lessons should also be learned from certain extraneous factor do not give confidence underlying factors in DFRP that helped to about the process or the way it was cause the deficiencies in the conduct and recorded; recording of the evaluation and reduced the d. The evaluation did not entirely follow chance that the risk of legal challenge would the process that had been set out for be fully recognised and addressed: explaining risk adjustments to each bid: a. At the time of the evaluation there some adjustments were not fully and was only limited central guidance transparently communicated to bidders; in Government on best practice in e. There was no consolidated evaluation outsourcing contracts and bid evaluation; report prepared at the end of the process. b. DFRP was not treated as a high priority This made it difficult for the project team in the Army or by customers more widely. to articulate the outcome confidently or The SRO and the project team lacked convincingly; the degree of support and challenge that f. MOD was further exposed to litigation might have been expected for a project of because the decision letter to Serco this importance; did not contain sufficient information c. The SRO attached a high level of to explain why they did not win the importance to adhering to the agreed bid; the record keeping and absence timetable for the final stage evaluation. of a summary evaluation report then This was understandable given the made it difficult to assemble information impact of continued uncertainty on DFR to respond to Serco’s requests for personnel and the projected financial disclosure; and examination of the and other benefits to be derived from evaluation records themselves revealed concluding a contract that was by then some of the deficiencies referred to running behind schedule. However, above, which led to delays in responding. the timetable and resourcing for the final stage evaluation required very intensive working from the evaluators and consensus leads, which seems to have left insufficient time for quality control of the process and record-keeping;
Independent Review and Lessons Learned 3 d. There was inadequate governance and Recommendations on Lessons oversight of the project and no route to to be Learned escalate concerns, for example about resourcing or risks. The board responsible 14. We recommend that the following actions for governing the project did not meet are taken to reduce the risk of similar issues often enough, does not appear to have arising on current and future projects in considered key risks in depth, and did Defence and across wider Government: not have representation from outside the project team to consider legal risks; R1. Cabinet Office should continue its current work to provide further central guidance to e. The assurance of the arrangements for departments on outsourcing contracts and and outputs from bid evaluation was bid evaluation, building on the Outsourcing insufficient; Playbook published in February 2019.5 This f. The Investment Approvals Committee should cover best practice for scoring and did not give much consideration to the recording evaluations, and appropriate adequacy of the governance or assurance communication with bidders. arrangements; R2. MOD should provide projects with fuller g. The legal advisers could have had a guidance and training on best practice for wider role to contribute to the design conducting and recording evaluations. In of the evaluation, advise on its conduct particular: and consider the risk of legal challenge. a. Evaluator and consensus lead training This should have helped to identify the should emphasise the importance of need for better record keeping and risks capturing reasons and evidence to support of inconsistency in the conduct of the evaluation scores, and using language evaluation; consistent with evaluation criteria; h. The financial advisers should have been b. Consensus leads should not also be used more extensively. evaluators and should be independent of C6. The size of the contract and extended the project; timetable for the competition together c. The evaluation process should created a risk that the award would be incorporate quality assurance to ensure challenged. A lot was at stake and after a that the stated process is followed and long competitive dialogue process both that records are fully consistent with bidders were likely to gain confidence scoring criteria; that there were no major concerns with their proposals. This was exacerbated d. There should be appropriate second- and by the period of over nine months third-line assurance of evaluation outputs, between submission of final tenders including financial modelling; and announcement of the result, which e. It should be a requirement to have a itself gave rise to concerns on the part consolidated evaluation report; and of the bidders that there was something there should be one storage place questionable in the process. for all records that may be needed to support the report, including emails and documents prepared by other government departments or external advisers;
5 The Outsourcing Playbook: Central Government Guidance on Outsourcing Decisions and Contracting, February 2019.
4 Defence Fire and Rescue Project f. When preparing award letters and bidder R6. Cabinet Office should reflect debriefs, more emphasis should be given recommendations 2 to 5 in its guidance and to transparency and timely and clear ask other departments to assess whether explanation of the evaluation outcomes. the lessons learned here are being applied to their own procurements. R3. MOD should ensure that all major contracting exercises have in place a 15. In our view all these recommendations are governance and assurance plan that important and merit an early response from considers the factors highlighted in the MOD. We recognise that recommendations 2 Infrastructure and Project Authority’s and 3 will take time to implement across the Government Functional Standard for Project Department and that there will be other more Delivery.6 These plans should establish: urgent priorities; but we believe it would be reasonable for the Permanent Secretary to a. responsibility for governance oversight; require significant actions to be taken on b. the role of governance in assessing risk, all the recommendations by the end of the including resourcing and timetable, and second quarter of 2020. the need for assurance; c. the presence on governance boards of independent expertise not only in finance and commercial but also in legal matters and procurement risk; and d. appropriate assurance arrangements in line with the guidance on three lines of defence.
R4. MOD should consider whether the IAC’s remit, as well as giving approvals, is to review the adequacy of governance and assurance. If it is, then this part of its role should be clarified and reinforced.
R5. MOD should ensure that: a. Major projects have legal and financial advisers, whether internal or external, with a remit and sufficient resource not only to advise on specific matters as asked but also to help design the procurement, comment on how it is conducted and advise on the risk of legal challenge; b. Legal support is available to cover all substantive dialogue, clarification or negotiation meetings with bidders where issues are likely to arise which could have legal implications. This should particularly be considered where the bidders are using legal representation.
6 Government Functional Standard GovS 002: Project Delivery Version 1.2 dated 1 August 2018.
Independent Review and Lessons Learned 5 6 Defence Fire and Rescue Project Fuller Description and Assessment
Methodology for the Evaluation (scored as 100), ‘Good Confidence’ (70), ‘Minor Concerns’ (40), ‘Major Concerns’ 16. MOD published its Evaluation Strategy (20) or ‘Critical Concerns’ (0); and Criteria (ESC) for the final stage of the c. Once responses for each RoR had competition on 25 July 2017 as part of its been evaluated, ‘consensus’ leads led Invitation to Submit Final Tenders (ISFT). This moderation meetings to reach a single set out the following stages once tenders had consensus confidence level for them. been submitted: a. Review to check that bids are compliant; The outcome of this evaluation was an aggregate confidence score for each bidder. b. Apply a confidence score to the proposed solutions for the 39 ‘Requirements of 18. For the whole life cost evaluation at (c) Response’ (or RoRs) with a predetermined above, the financial aspects of bidders’ weighting for each, producing an overall tenders were put into a financial model to confidence score; calculate the cost of each bid. The figures were then subject to two forms of risk c. Determine the risk-adjusted whole life cost7 adjustment referred to as a discrete risk of each bidder’s solution in Net Present adjustment and a pricing risk adjustment: Value terms; and a. For the discrete risk adjustment, the d. Combine b and c to determine the Most financial evaluators, working with Economically Advantageous Tender (MEAT). commercial and technical colleagues, 17. In relation to the confidence score at (b): determined the likelihood of the bid creating risk or leaving risk with MOD and a. Requirements were in three categories: the expected impact of this on MOD’s technical, weighted at 78%; commercial, own costs,8 and made an adjustment weighted at 20%, and financial accordingly. MOD costs represented two management, weighted at 2%. Individual thirds of total project costs. The discrete RoRs in each category were also weighted. risk adjustments themselves differed For example, a bidder’s mobilisation and between the bidders by about 0.1% of migration plan (given the largest weighting) total project costs; carried 10% of the total technical marks, while the proposals on Cyprus locally b. For the pricing risk adjustments, financial engaged civilian employees carried only 1%; evaluators considered the projected impact on price of three scenarios: costs b. Evaluators assigned a confidence level for as estimated, a 30% cost over-run, and each RoR based on their expert judgement a 10% cost under-run. The mean of the of the evidence the bidder had provided to three scenarios was then used to calculate support its proposed solution. Confidence an adjusted projected cost to MOD. scores were divided into ‘High Confidence’
7 The total cost of DFRP over the length of the contract. 8 The majority of the risks related to the future cost of activities carried out by personnel, principally firefighters, still employed by and paid for by the Department but tasked by the contractor, the few fire services provided under separate contracts and the short-term cost of the training college at Manston before its closure.
Independent Review and Lessons Learned 7 19. Once the discrete risk adjustments and the 22. Results were then summarised by plotting pricing risk adjustments had been applied, them on a chart in which confidence scores the NPV of each bidder’s whole life cost to were plotted on the y axis and cost scores MOD was calculated. on the x axis, as shown in figure 1. Lines (‘indifference curves’) show the points at 20. The final step - combining the confidence which, given MOD’s decision about the scores and the projected whole life cost relative weight it would give to cost and - was then made in a separate part of the confidence at the stated confidence levels, financial model that attached predetermined different combinations of the two would weightings to each score to make the be assessed as equally advantageous. 9 combined calculation of the MEAT. In this illustration we have shown four 21. The weighting scale in this competition hypothetical bids. The Most Economically was on a sliding scale that gave very Advantageous Tender would be the bid on heavy weight to achieving at least a 70% or above the indifference curve nearest the confidence score, much less weight to top left-hand corner (in this case, the blue achieving a higher confidence score dot). The bids represented by the grey and (between 70% and 82%) and no further value purple dots would be assessed as equal to at all to confidence above that level. In order each other but inferior to the blue. The bid to create a figure that could be combined indicated by the green dot would be the with the confidence score arithmetically, least advantageous. the whole life cost was converted to a percentage by dividing the risk adjusted whole life cost into MOD’s own projected cost of the existing DFR arrangements.
Figure 1: Most Economically Advantageous Tender (MEAT) Graph
9 20180111-Cost Model used for TCN v193-OSC
8 Defence Fire and Rescue Project 23. This approach to evaluation is provided for in helping evaluators understand the concept both the Public Contracts Regulations 2015 of moderation rather than actually training (PCR) and DSPCR 2011, and is a recognised the consensus leads. way to obtain best value for money in public procurement. It is designed to incentivise 27. In summary, while the training appears to and select the bid which delivers the most have been sensible and relevant, in our view economically attractive combination of there were significant gaps, particularly on satisfactory - and preferably superior - record keeping. This proved to be significant technical quality with an efficient price. It and should be addressed for future avoids the risk of incentivising and paying procurements of this size and complexity. for unnecessarily high levels of delivery confidence which would not provide value Did MOD Follow the Stated for money. Evaluation Process? 24. In our view, the DFRP evaluation 28. The project team conducted the final stage methodology was fit for purpose and the evaluation largely in accordance with the explanatory materials provided to bidders published documentation. However, there were generally clear. were certain limited exceptions. Evaluator Training 29. First and most important, there were apparent discrepancies in the way some 25. Evaluator briefing and training for the Final of the consensus confidence scores were Stage evaluation took place in June and assessed: August 2017. The June briefing materials a. In scoring four out of 39 of the Capita included a 130-page evaluator pack, which responses, reference was made in included a briefing on the competition, an moderation notes to the presence explanation of the RoRs and the scoring of “commercial levers”, which were system for each, and a two page “DFRP thought to mitigate certain concerns Evaluation Do’s and Don’ts”. This guidance and so to help justify a score of Good was sensible and covered most of the key Confidence. We were told these referred points for evaluators. However: to contractual remedies or incentives. a. It did not emphasise the importance According to the scoring criteria, of evaluators taking a full note of their contractual remedies or incentives process or being precise and consistent for performance failures should not in language to reflect the scoring criteria; have been evaluated in a technical for example, that it would be inconsistent response. We heard that scores of Good to record that the evaluator had minor Confidence would have been given even concerns and give a confidence score of without the contractual levers. However, Good Confidence without explaining why; the inclusion of these matters in the moderation notes raises a question as b. It did not include any guidance for to whether these four responses were consensus leads on how to quality control evaluated fairly. No Serco responses had individual evaluator reports, or on how to been recorded as benefitting from the chair and record moderation discussions. presence of such “commercial levers”. 26. The August training session included further However, concerns were also noted short guidance for evaluators on do’s and against each of their equivalent four don’ts (again correct in content but very responses, all of which also received a high level). Guidance was provided on the score of Good Confidence despite such moderation process but was aimed largely at issues. This suggests to us that there was some leniency in the assessment of both bids;
Independent Review and Lessons Learned 9 b. The methodology specified a confidence confidence as to the proposed solution”. score of Minor Concerns where a Any further statements on what would bidder’s solution provided MOD “with constitute “a high level of confidence” were minor concerns”. However, there were also written generically. This left evaluators several instances where moderation without clear and specific guidance to help reports referred to “concerns”, but them record accurately why they had scored evaluators awarded an overall score any concern as major, minor, or simply as an of Good Confidence rather than Minor issue to be noted. Concerns. This arose for both bidders; but caused Serco to question whether all 31. The criteria for scoring should have been of the Good Confidence scores awarded better defined. Both the definitions and the to Capita were justified, because quality of recording evaluation judgements the scoring criteria suggested that a could have been improved if technical Good Confidence score could only be subject matter experts had been more awarded where there were no concerns involved in preparing the evaluation criteria, of any kind. This is not how the scoring and if the application of the criteria had been methodology was actually applied; tested with evaluators before the process went live. c. In recording a score of Good Confidence many of the moderation reports referred 32. However: to the relevant bidder having provided a. based on a sample which we have an “adequate” level of confidence that reviewed, the generic criteria appear to the response met the MOD’s aims. have been applied equally to both bidders; From interviews, it was clear that these responses were considered to meet a b. both bidders received scores of Good Good Confidence threshold. However, Confidence where the evaluators noted anyone considering a challenge to some remaining “concerns”. On the face the decision could clearly argue that of it this should not have been possible. “adequate” is less than “Good” and Our analysis suggests that this occurred that scores of Good Confidence more often with Capita than with Serco, should not have been awarded in these which could give the impression of bias. circumstances. Better quality control and However, having interviewed members review of the moderation notes should of the project team, and given our other have caught this discrepancy. findings about the evaluation process, we believe it is more likely to reflect the poor 30. Members of the project team who acted recording of moderation decisions; as evaluators told us that the way the c. While the references to “commercial confidence scoring criteria had been levers” should not have been made, drafted proved unhelpful when recording we see no evidence that these affected consensus decisions, in particular when the scores awarded. We have reviewed choosing between Good Confidence the moderation notes for the four RoRs and Minor Concerns. After reviewing the for which this occurred. In each case, documentation, we have concluded that both Capita and Serco’s responses had these discrepancies arose partly because concerns noted against them, and in each the marking methodology for the technical case a consensus confidence score of RoRs was written generically and not Good Confidence was awarded. Even if specifically tailored RoR by RoR. The Capita had been awarded scores of Good descriptions of each confidence score Confidence when it should have received were also circular. For example, for a score scores of Minor Concerns, we conclude of High Confidence the criteria said that the same would probably have been a bidder had to “provide a solution that: equally true of Serco. … provides the MOD with a high level of
10 Defence Fire and Rescue Project 33. Accordingly, we think it is unlikely that the comments that issues with the Falkland factors described here disadvantaged one Islands solution were discussed, there bidder relative to the other. is no record of how the concern was satisfactorily addressed and why 34. Secondly, as noted above, there were the process resulted in a consensus deficiencies in the recording of the confidence score of Good Confidence. moderation processes. Paragraph 18.1.4 of the ESC stated that consensus leads would 35. In our view, as a result of the issues “Maintain a full audit trail of all documentation described in paragraphs 28-34, the conduct pertaining to the evaluation.” In fact, in a of the evaluation and the way it was number of responses strong concerns were recorded left significant scope for challenge. raised at evaluation stage but apparently This could have been avoided or mitigated resolved at consensus stage, leading to an by more specific definitions of scoring overall consensus confidence score of Good criteria, more use of technical subject matter Confidence but without adequate supporting experts in finalising the criteria and testing evidence of how concerns had been the scoring methodologies, and more quality addressed. Examples include: control and review of completed evaluation and moderation notes. a. For RoR 2.1.2 (“Management & Maintenance of Vehicle at Overseas Locations and on Operations and Discrete Risk and Pricing Exercises”) there was initially a Major Adjustments Concern about a specific element of Serco’s solution relating to the provision of 36. The adjustments explained in paragraph 18 appropriate vehicles in Cyprus, although were small compared to the difference in overall the evaluator then scored Serco confidence scores for each bid. Our review as Minor Concerns. In moderation, confirms that their size was not sufficient to Serco then received a score of Good outweigh the difference in confidence scores Confidence for the RoR, but with a arising from the confidence evaluation comment that there was a concern about and we have seen nothing to suggest that vehicle sufficiency in Cyprus. While it the process used to determine them was seems from this that the concern was anything other than fair and professional. discussed during moderation, we have not 37. However, a number of the adjustments been shown any record of how that final involved a degree of discretion and bidders consensus confidence score was reached; were given no written explanation of b. For RoR 1.4.1 (Defence Airfields), Capita’s them. Paragraph 12.1.3 of the ESC stated response scored Good Confidence, that adjustments would “be exposed despite the moderation report stating to the Bidder at the Detailed stage of that the evaluators were “of the opinion Dialogue” and that bidders would “have that the Bidder did not understand the the opportunity to challenge”. In fact, while requirement of the Falkland Islands” the project team made clear to bidders and that Capita’s “proposed solution during dialogue that their bid prices would would not meet the requirement…”. In be subject to adjustments and alerted them the underlying evaluation reports, one where specific points were likely to attract evaluator stated “there appears to be an adjustment, bidders were not proactively no appreciation for the actual scale informed about the size of adjustments. of operations that take place in the 38. In addition, 39.9% of the adjustments by Falklands…” and that the proposals value were not communicated at all to Serco were “doomed to fail”; yet that evaluator because the project team considered them eventually gave an overall confidence irrelevant to evaluation on the ground that score of Minor Concerns. Again, while they were largely identical for both bidders. it seems from the moderation report
Independent Review and Lessons Learned 11 12 Defence Fire and Rescue Project However, in fact there was a £0.87m deadlines. This included regular weekend difference between the bidders. This was working and repeated late nights over a not sufficient in itself to make a decisive nine-week period by a number of people, difference to the outcome of the competition some of whom were the only specialist given the difference between the bidders’ for an aspect of the assessment. While confidence scores. However, as the project evaluators were able to meet deadlines, they team did not disclose them during the have said they were not able to record clear process, and the financial aspects of the notes of their decision-making processes. evaluation were in a tight range, the lack of The papers we have reviewed show explanation increased the risk of bidders evidence that they did not do so.10 not fully understanding the position, and therefore the risk of challenge. 44. Some members of the project team said that concerns were raised over the lack 39. It would have been preferable to give of sufficient time for evaluating tenders, bidders an indication of the amount recording outcomes and holding moderation assigned to each risk, to explain how it had meetings; but that they were told that the been determined and to consider any points timetable had to be met. the bidder then raised. 45. Nothing we have seen suggests that lack of Resourcing of the Evaluation time impacted the fairness of the evaluation process or the outcome. However, in our 40. The number of RoRs and the requirement view it affected the standard of record for bidders to provide method statements keeping. This made it much harder for MOD as part of their responses created a large to evidence the robustness of its process volume of material to be evaluated. Each and much easier for an unsuccessful bidder tender comprised approximately 9,000 to challenge apparent anomalies. pages of A4. Moderation Process 41. In planning resources, an average of five minutes per page was allowed for the 46. Ten of the forty-nine evaluators also acted as evaluation process. This took account of consensus leads for moderation meetings. the fact that different people read different The current view across Government is that sections, so that every evaluator did not read evaluators should not also act as consensus every page. It also reflected the fact that, by leads. This is partly because a consensus the time of the final assessment, the team lead’s role includes quality assurance of were familiar with much of the content from evaluations and an evaluator cannot quality the successive stages of dialogue. assure their own work; partly because it risks the consensus lead’s evaluation score having 42. Although five minutes is probably an undue weight in moderation discussions; and adequate time to read a page of method partly due to the workload of balancing both statement or technical response, several evaluation and consensus lead roles. members of the project team reported during our interviews that there was not time 47. Guidance on these matters could not have to cross-check references where supporting been known to MOD or the project team evidence was spread across multiple pages at the time, because it is only now being or where bidders referred to other parts of prepared by the Cabinet Office. However, in their tender. our view, the dual role of consensus leads and the recording of the moderation process 43. As significant cross-referencing was required may have increased the risk of successful in evaluation, some of the evaluators worked legal challenge. extremely long hours to meet evaluation
10 See paragraphs 28-34.
Independent Review and Lessons Learned 13 Summarising and Explaining Use of the Financial Model Evaluation Results 52. The project team used a central financial 12 48. The evaluation process as a whole was model, also referred to as a cost model, to largely managed through the AWARD® collate and analyse all information received e-procurement system.11 This system is used from the bidders. Bidders were provided with by MOD for a wide range of procurements. a template that aligned with the model in It allows the issue of tender documentation which to submit their financial responses. The to bidders, covers the clarification process data in the templates was then transferred between bidders and the MOD and records into the model. Once populated, the model evaluator and consensus confidence scores. was used to produce the total projected It therefore provides the basis for recording whole life cost for each bid. It also included all the large majority of the process. other calculations, including the incorporation of discrete risk and pricing risk adjustments, 49. However, the project team did not prepare the confidence scores transferred into a consolidated evaluation report at the the financial model from AWARD, and the end of the process to record and explain calculation of the MEAT scores. the outcome. This would have been done as a matter of course under Regulation 53. The financial model was principally an 84 of PCR 2015, under which most UK established management accountancy tool public procurement takes place, and which used in the Army. Finance team members sets out record keeping requirements were either familiar with the model from for procurement. However, DFRP was using it on previous projects or had time to conducted under the DSPCR 2011 familiarise themselves with it before the ISFT regulations, which do not contain an stage. The part that combined confidence equivalent obligation. and cost scores into a combined MEAT score was added by the project team with 50. We have heard that all material that would particular input from Deloitte. have been included in an evaluation report was available either in AWARD or in 54. We have reviewed the arrangements for other documents such as the investment assuring the model. In January 2018, appraisal, just not in a consolidated form. the project team engaged MOD’s Cost Even if this is the case, the lack of a single Assurance and Analysis Service (CAAS) consolidated report created difficulties in to carry out assurance of the input of the providing a clear and simple explanation bidders’ templates into the model. Line to Serco of why their bid had not been by line cost information and assumptions successful and seems to have been one were extracted from the templates by of the factors that caused delay and lack the project team and given to CAAS in a of confidence in responding to Serco’s separate spread sheet. CAAS checked 576 disclosure requests in June and July 2018. cost lines to assure that they calculated correctly and were based on the correct 51. In our view, it is good practice to have a inputs. Three of them were given a status of consolidated evaluation report as an audit Red. Of the three, two related to Capita (total trail and to provide the basis for decision value £0.67m which represented 0.1% of letters and explanations to bidders. output cost) and one related to Serco (total value £23.37m which represented 1.8% of output cost). In all cases the concern was about insufficient evidence for the input numbers rather than errors in calculation.
11 The AWARD® suite is a collaborative web-based solution designed specifically to support complex and strategic procurement projects. 12 20180111-Cost Model used for TCN v193-OSC (1)
14 Defence Fire and Rescue Project CAAS stated that “the project team where November 2016 showing the level of cost not available to give a comment due to reduction needed in the early years of the the very tight timescales involved”. With contract and requesting that they make 99.7% and 98% respectively of all the lines proposals for addressing the issue. deemed valid across the two bidders, this satisfied CAAS’ criteria, so they concluded 58. Capita’s proposal was to use what was that the “data in the model is of appropriate referred to as a “receivables funding” (RF) maturity and sourcing to be deemed as fit product. This involved raising bank debt for purpose”. The project team did not seek to finance some costs in the early years so further evidence to address the concerns that contract charges in those years could because they were satisfied they understood be reduced to remain within MOD’s budget. the source of the numbers. This finance would be repaid from cashflows later in the project. In order to establish 55. CAAS did not assure the risk adjustments. security for this bank financing, Capita The scrutiny report said ‘there is evidence proposed that MOD should agree to ring- to indicate that a detailed risk adjustment fence a fixed part of its payment obligation has been made to the bids. No independent under the contract and recognise this as validation of this risk adjustment appears a firm commitment that MOD would be to have been undertaken’.13 A validation of obliged to meet in the event of termination of these adjustments by people external to the the contract, regardless of cause. project team would have been best practice, even though the adjustments were small. 59. The short-term affordability issue in Serco’s own bid was significantly smaller than 56. The model was not subjected to any Capita’s because of a different phasing of final assurance once all the inputs and cashflows. Perhaps for this reason, Serco’s adjustments were made to test that the response to MOD’s invitation did not MEAT result was calculated correctly. propose a third-party debt-financed solution. We have looked at a sample of the MEAT Instead it re-phased certain cashflows to a calculations and found no inaccuracies. more limited degree than Capita and sought However, it would have been best practice analogous but lesser protections under the to have had assurance external to the contract for its additional early funding. project team to provide confidence that this additional element of the model and MEAT 60. The project team decided that it would not be result was correct. appropriate to tell Serco that a debt-financed solution was an option, as this would in their view have meant sharing confidential Capita Adjustments made by Bidders bid information with Serco. in the Final Stage to Address 61. MOD, having consulted HM Treasury, Affordability initially indicated that it was content with 57. Towards the end of 2016, the project team the Capita proposal, because it addressed had identified that both bidders’ proposals the affordability issue in years 1-3 and presented affordability challenges because provided a better value for money option – while the bids were judged to represent than Capita funding the rephasing of spend value for money against the public sector itself at a higher cost of capital. As a result, benchmark in Net Present Value terms over Capita were informed in principle of the the contract as a whole – there were some acceptability of the RF product and they early years in which the bids exceeded included it in their final bid in September MOD’s current budgets. To address this, 2017. The project team did not ask the they issued a paper to each bidder in Treasury for a view on Serco’s proposal.
13 Scrutiny report dated 19 January 2018
Independent Review and Lessons Learned 15 62. In the event, contrary to their previous could be proposed as part of their view and as result of discussions across submission, in our view raising debt Government and with the NAO about the to smooth cashflows is a common definition of contingent liabilities across arrangement that either bidder could have Government, in May 2018 Treasury expressed proposed. There was no need specifically concern that the RF arrangement proposed to request consideration of it; by Capita might constitute a contingent b. We agree with the view reached by the liability. This was then confirmed to be the project team that it would not have been case. A contingent liability for the ring-fenced appropriate to inform Serco of the nature amount was therefore reported to Parliament of Capita’s proposal after it had been at the time of contract award on 18 June submitted in March 2017 or during the 2018. Although it was not considered at the final evaluation; time, we understand that had Serco’s tender won then it would probably also have given c. The potential whole life cost impact of rise to a contingent liability. the contractual amendments proposed by the bidders were understood and 63. While Serco had received the same letter as appropriately taken into account. This was Capita in November 2016, they only became not done through an arithmetic calculation aware that consideration was being given to of changes to potential termination debt finance as a result of the statement to liabilities, because evaluation was carried Parliament on the date they were notified of out on the basis that services would the results of the competition. continue to be delivered for the full life of the contract. However, we are satisfied 64. The project team have shown us that they that this was a reasonable approach. analysed what the difference would have been between the bids if, instead of using the RF proposal, Capita had internally Governance, Assurance funded the change in bid profile using their and Approvals own capital. The analysis showed that Capita’s whole life cost would have been 66. The current Government Functional Standard higher, which would have narrowed the cost for Project Delivery sets out guidance on gap between the Capita and Serco bid. commercial standards for the direction and However, Capita would still have had the management of programmes.14 Although lower whole life cost; and because of the the standard dates from 2018, the concepts significant weighting given to the bidders’ it describes are of long standing. It covers confidence scores and their higher rating for governance, assurance and approvals: confidence, Capita would have still have had a. Governance includes authorising, a higher MEAT score than Serco. directing, empowering and overseeing 65. The disclosure of the RF arrangement at the management; time of contract award was one fact among b. Assurance means checking and several that led Serco to believe they had confirming that work is controlled, been treated unequally. We have therefore on track and aligned with policy and considered whether the handing of this strategy; and matter raises issues of fairness. We do not believe that it does: c. Approvals should make timely decisions by evaluating proposed approaches and a. While it might arguably have been alternative choices against agreed criteria. preferable for MOD to indicate to both bidders in 2016 that external finance
14 Government Functional Standard GovS 002: Project Delivery Version 1.2 dated 1 August 2018.
16 Defence Fire and Rescue Project These three elements of project delivery for who would have been well placed to take a DFRP are described below. wider view on the approach the project was taking and the risk of legal challenge. Governance 70. It had been intended that the Operations 67. MOD guidance15 on governance is largely Board would meet around the time of key directed at equipment procurement projects. project milestones and at least quarterly. It says that a project should have a sponsor In practice, it met only once in 2017, in supported by a sponsoring group. March soon after Main Gate approval and before issue of the invitation to submit 68. In March 2014 the project team prepared a final tenders; and once in 2018, in May governance strategy, which was approved while the project was in the process of by the project team leader. Governance obtaining final clearance to proceed to was to be provided by a Project Board contract award. Minutes of the two meetings chaired by the SRO. Subordinate to the suggest that, while there were some Project Board was an Operations Board, substantive discussions, for example on an chaired by the Project Director. The role and organisational safety assessment, much of membership of both boards was to review the agenda involved briefings on progress key documents, risks, assumptions and from the project team without the Board benefits, agree trade-offs and resolve issues. reviewing decisions, giving guidance or The IAC scrutineers were broadly content considering major risks. with this proposal although they questioned why the strategy had not been approved 71. The project team have explained that by the Project Board.16 The governance meetings were taking place outside this strategy was then revised in 2015 so that the forum with scrutineers and other experts, Operations Board had day-to-day delegation and that Board members were regularly to act as the Project Board, which had not kept informed of progress outside meetings. met since 2013. However, in a period in which important final evaluations were being conducted, in our 69. The MOD guidance recommends that view it would have been best practice for members of governance boards should the Board to have met more frequently to include experts from the subject areas discuss progress and risks. that make up a project, such as training, infrastructure or equipment, together with 72. At the Operations Board meeting in May representatives from finance, commercial 2018, delay due to legal challenge was and risk “as required”. The membership shown in papers as a high risk. At that of the DFRP Operations Board comprised time, the project team was confident that mainly senior members of the fire and adherence to the evaluation strategy meant rescue community and their customers. that any challenge could be successfully Experts external to the project team included defended.17 Risk registers in the Main Gate the Head of Army Commercial, the lead Business Case in September 2016 and scrutineer to the Investment Approvals the investment appraisal also highlighted Committee and representatives from HR the risk of a legal challenge caused by and finance. In our view, best practice inadequate communication with bidders would have been to include more people or failure to conduct the competition in a independent of the project procurement, fair and transparent manner. The mitigation including some with legal and risk expertise was shown as being to demonstrably
15 Capability Management Practitioners’ Guide Release 2.1 dated 23 September 2013/Policy (Direction &Guidance for Staff Undertaking SRO Roles in MOD) Version 6 dated 28 September 2015. 16 Email from scrutineers to project team dated 24 April 2014. 17 Record of Decisions of the DFRP Operations Board held 4 May 2018.
Independent Review and Lessons Learned 17 follow the stated process, with effective 76. The second mechanism for providing communication and transparency, governance in MOD would have been via the engagement with MOD’s Central Legal portfolio system operated within individual Services (CLS), effective information TLBs. Projects in this system, of which there management and appropriate feedback are currently 38 in the Army, are overseen being put onto the AWARD system. by the TLB holder’s Portfolio Office which reports quarterly on progress to a committee 73. With hindsight, it is clear that more attention chaired by the Deputy Chief of the General should have been paid to the management Staff who programme SROs also meet every of these risks. It appears this did not happen six months for a project review. DFRP was because the Board did not have a clear role already in existence when this system was and did not give sufficient consideration to established, and we have been advised that, risk as a whole. as it was a legacy project, the decision was taken not to include it in the portfolio. 74. There are two routes through which DFRP could have received greater governance 77. The mechanism for governance and oversight and support. The first was the oversight of DFRP in the Army therefore Defence Major Programmes Portfolio consisted only of quarterly performance (DMPP). Projects are included in DMPP reporting against the Corporate Plan and according to their size, complexity and risk. arrangements set up by the project itself. DMPP projects make a quarterly return to Under the former, DFR reported performance the centre of the Department on a range of each quarter against a range of activities issues such as dependencies, resources, related to maintaining its service including risk and performance against milestones. a “RAYG”19 chart showing the direction of The SRO of a DMPP project also has a six- travel on DFRP with a short commentary. monthly challenge and support meeting with This is reported through the chain of senior officials to review progress and issues. command with significant issues ultimately raised to the centre of the Department. 75. A 2017 review of DMPP’s scope identified DFRP as a possible candidate for the 78. In our view, DFRP’s governance Portfolio but did not recommend its arrangements left the project team inclusion. It was not considered as high risk with inadequate oversight or routes for as some other projects. In our view, this was escalation of issues. The Operations Board probably the wrong decision. Consideration for DFRP was the sole potential source of should have been given to the potentially governance for the project team and had contentious nature of a major outsourcing only limited effectiveness. involving safety risks, and attention should have been paid to the need for governance Assurance for a project that did not have strong natural ownership at the centre of the Department 79. Government central guidance20 is that or in any single Top Level Budget (TLB).18 assurance should be provided in three Inclusion in the portfolio would probably Lines of Defence (LODs). This is also have provided stronger governance of the consistent with corporate best practice. project and identified and addressed the The first line consists of the project factors which contributed to the procurement team carrying out its own checks that process being open to challenge. appropriate standards are being used.
18 For budgeting and administrative purposes, the MOD is divided into a hierarchy of budget areas of which the Top Level Budget is the highest. The Army, Navy and Air Force are all Top Level Budgets as are organisations such as the Defence Infrastructure Organisation. 19 Red/Amber/Yellow/Green – A widely used method of indicating the status of a project in terms of it meeting its expected progress and other measurement criteria. 20 Government Functional Standard GovS 002: Project Delivery Version 1.2 dated 1 August 2018.
18 Defence Fire and Rescue Project The second consists of consideration by in paragraphs 28 to 34. or to its own those with no first line responsibilities to internal assurance of the records; and it ensure that first line defence is properly did not seek additional assurance from designed and operating. The third is LODs 2 and 3; normally undertaken by internal audit or an b. LOD2: MOD scrutineers were mainly independent external body. Guidance is also content with the Main Gate Business that assurance reviews should be scheduled Case requesting approval for contract prior to significant decisions. award, while noting that there appeared not to have been an independent 80. MOD guidance is that SROs should seek validation of the risk adjustments made appropriate assurance through Gateway to the bids. They could have raised more reviews and from Defence Internal Audit. questions on the risk of legal challenge These are intended to provide the third line of and the steps taken to mitigate it. In defence. We note that another source would addition, the CAAS assurance of both the have been review from the Infrastructure and financial model and the risk adjustments Projects Authority (IPA). should have been more extensive, even 81. For DFRP the lines of defence were as follows: though this does not appear to have been a critical factor; a. LOD 1 assurance within the project team itself, which included the role of c. No LOD3 assurance was called for considering the evaluation process; from Defence Internal Audit or external advisers in the period we are reviewing. b. LOD 2 assurance was shaped largely by The Gateway 3 review report suggested the scrutineers supporting the Investment that there could be value in a further Approvals Committee. The project was Gateway report before contract award, provided with a list of the evidence but this was not carried out. needed to support its Main Gate Business Case. This was then considered by 83. In our view, the assurance activity was not the scrutineers. Further assurance was sufficient and falls short of best practice. The provided by the reports from MOD’s Cost SRO or governance arrangements should Assurance and Analysis Service (CAAS) on have increased the level of assurance of the the financial model (see paragraphs 54-55); bid evaluation. c. LOD3 consisted of a Gateway 3 review by independent experts within MOD Approvals considered the project in 2016.21 It made 84. As a Category A project, the highest level recommendations about the provision of of project in the Ministry of Defence, DFRP additional commercial, legal and project was considered by the Department’s management staff and assessed all these Investment Approvals Committee. The as critical. Subsequently one project IAC has responsibility for considering manager was recruited but, following major investment proposals on behalf discussions with Army Commercial and of the Defence Board and making CLS, it was decided not to obtain further recommendations to Ministers as resource in other areas. appropriate. Its terms of reference require it 82. Our view on the effectiveness of the three to establish whether the projects it considers, LODs for DFRP is: among other things, meet the requirement, are value for money and are deliverable a. LOD1: The project team did not pay through sound project management. sufficient attention to the importance of recording evaluation results, as described
21 MOD Integrated Assurance Review -Gateway 3: Investment Decision dated 17 March 2016.
Independent Review and Lessons Learned 19 20 Defence Fire and Rescue Project 85. DFRP was considered by the Committee 90. The legal advisers were not integrated fully at several points and required IAC approval into the project. They were used as an to move from one major stage to the next. expert service and instructed on an “advice During the period covered by this review, the and review” basis - meaning that they were SRO sought approval to award the contract instructed by the project team on discrete to the winning bidder. topics as and when required; for example, on TUPE matters arising for transfers of 86. The IAC did not appear to think that its role employees. They provided input into the included consideration of the governance evaluation strategy but did not advise on the and assurance environment for the project, evaluation process and the criteria to be used. which in our view should be part of considering the ‘sound project management’ 91. The allocation of legal resources to MOD as referred to in its terms of reference. a whole and to their main projects is decided annually as part of the MOD budget setting 87. We also note that, before the period covered process. CLS concluded at the outset of the by the review, the original SRO - the then DFRP procurement process that it had the Adjutant-General - had delegated the role to capability and capacity itself to advise the the Project Director because he believed the project without appointing a private firm of project did not require an owner at his level lawyers. and the Project Director was better qualified for the task and would have more time to 92. The Gateway report of March 2016 said that devote to it. There was no approval required the review team “considers… dedicated for the role to be delegated. MOD guidance commercial legal support to the project to be does not prescribe who should appoint necessary in order to capture the negotiation SROs or their equivalents to projects that are discussion in a timely manner across all three neither part of the DMPP nor part of a TLB’s concurrent dialogues”. The legal resource portfolio. It says they can be appointed available was therefore reviewed at the by a senior member of the TLB with the request of the project team.22 CLS confirmed delegated authority of the TLB holder. Had their view that there was sufficient resource approval been required from the IAC or the available in the legal team to provide the Permanent Secretary it could have given an required support without the need to have additional opportunity to consider the need recourse to private sector lawyers, although for better governance. they considered that the project team’s own resources could benefit from being reinforced.
Use of External Legal and 93. Later in the procurement process, when Financial Advisers detailed dialogue was underway with both bidders, MOD chose to attend some 88. The team had legal advice and support from meetings without legal support, even the Department’s Central Legal Services though the bidders themselves were legally - “CLS”, which is part of the Government represented. Serco commented to us that Legal Department (“GLD”); and financial they believed the presence of lawyers would advice and support from Deloitte. have facilitated the procurement process by maintaining consistency of approach and Legal Advisers perhaps identifying the departures from the agreed process. 89. The project was advised by a team of three lawyers supported by a Deputy Director 94. In our view it is normal practice for parties (“DD”) from CLS. Their expertise included to have legal representation where legal procurement, contract, pensions, TUPE and issues are expected to arise. Lack of legal employment law.
22 See paragraph 81c.
Independent Review and Lessons Learned 21 representation at commercial negotiations assessing the robustness of recording of the or other substantive dialogue meetings may evaluation process or giving any assurance have been a mistake if legal issues did in on the matter. fact arise. 99. Our conclusion is that there was appropriate 95. The project team have said to us that they financial expertise available to serve the would have welcomed more comprehensive project properly, but it could have been used legal support. They have also said that in more extensively in the final stage. their view appointment of a private firm would probably have made it easier to Issuing of Award Letters and access this resource quickly at times of peak demand. Related Information Provided to Bidders 96. Our conclusion is that legal advisers should have been given broader roles to consider 100. The letters to the two bidders notifying the design of the tender evaluation, how it them of the outcome of the competition was conducted and how to minimise the (the Contract Award Decision Notification associated risk of legal challenge. Additional or “CADN letters”) were prepared by the legal resource appropriately deployed project team with input from CLS. could also have helped maintain a focus on 101. The data used to inform the content of record keeping where project team resource the letters was drawn from the AWARD was stretched. There is some evidence to platform, but in the absence of a suggest that this level of resourcing might consolidated evaluation report it was not in have been easier to achieve with an external a form which could readily be used to draft firm of legal advisers, but we cannot reach a the letters to the bidders. The project team conclusion on this. therefore collated the relevant information and contacted CLS with a first draft of the Financial Advisers letters on 16 March 2018. From then until 8 May there were regular e-mail exchanges 97. Deloitte were initially involved in helping to as well as meetings and telephone calls design the bid evaluation process. Latterly with the lawyer conducting the review of they were used largely on a task driven the drafting. The focus was on the content basis, with the level of support provided of the letter to Serco. designed to reduce over time. It was felt that in the earlier stages of the project there was 102. CLS advised that the risk of legal challenge a greater need for structuring and strategic was high and that it was important the support, while later only more limited letter to Serco clearly set out why the support was needed assisting with the losing bid had not won. They questioned financial model and certain discrete tasks. whether the content of the proposed CADN letter provided a sufficiently detailed 98. By the time of evaluation there was only explanation. The lawyer did not think that one Deloitte consultant regularly engaged the information provided, both generally with the project team. Although we and particularly about the discrete risk heard from Deloitte and others that the adjustments, was sufficiently transparent to requirement for their input at this stage had enable the losing bidder to satisfy itself why indeed significantly reduced, this seems the bid had not been selected and to have unusually light. We would typically expect confidence in the figures provided. a project team working on a procurement of this size and complexity to benefit from 103. The project team concluded however that greater support. In particular, Deloitte were sufficient information was being provided not asked to give advice or support in because there had been a very lengthy
22 Defence Fire and Rescue Project dialogue process, as a result of which both bidder since the date of submission of bidders understood how the costs were its final tender. Further information was calculated and the way in which the risk not supplied by MOD, who stated that adjustment had been approached. On this the request was too wide ranging, that assumption, the CADN letter to the losing it would be disproportionate to conduct bidder was adjudged ready to send. such a search (much of the information requested was stored on a number of 104. In our view, notwithstanding the earlier different IT systems some of which were engagement with bidders, a clearer no longer easily accessible or in use, explanation was required in the CADN letter making recovery difficult and costly), to enable the bidders to understand and and that some of the information was have confidence in the derivation of the confidential to the winning bidder. final figures. 108. The response and additional information 105. On 18 June, letters were sent to both supplied by GLD did not satisfy Serco bidders. The content of both included a and nor did the feedback meeting held by graph of the type shown in paragraph 22 MOD on 16 July to explain the reasons for plotting the confidence score and whole the award decision. Consequently, Serco life cost, together with the residual costs continued to pursue its claim against and risk adjustments; and an Annex titled the MOD. “Solution Confidence Evaluation” which provided a commentary on strengths and 109. In our view, while there may have been weaknesses of the proposal. Its receipt grounds for not releasing some material, the by Serco did not prevent, and may have information made available did not provide prompted, the issue of the pre-action an explanation of the award decision protocol letter by their solicitors on 22 June. which could be expected to satisfy a losing bidder. It is notable that Mr Justice Fraser 106. The pre-action protocol letter attested in February 2019, in response to a court that the losing bidder was not satisfied a application by Serco for disclosure, held clear explanation had been provided and that “the MOD should have voluntarily that the documentation in relation to some provided this documentation months ago”.23 evaluation criteria did not support the final scores. There was a particular focus in the 110. Our conclusion is that MOD faced the letter on the cost evaluation and a request continuation of the litigation partly because: for an explanation as to “how [MOD] has a. The CADN letter to Serco had not given calculated the residual cost sums and them an adequate explanation of why evaluated price during the procurement”. they had not won the bid; 107. In addition to the pre-action protocol b. The record-keeping and absence of letter, GLD’s Commercial Litigation Team a summary evaluation report made received a letter from Serco dated 27June it difficult for MOD to assemble the requesting the provision of a wide range information needed to articulate the of further information. They responded by reasons convincingly; supplying copies of the AWARD records of the individual and consensus scores c. Examination of the evaluation records for the bidders for the evaluation criteria themselves showed deficiencies in identified in the pre-action protocol recording how final evaluation decisions letter for each stage of the procurement were reached; and process; and a summary of the MOD’s d. MOD’s responses to Serco’s requests for clarification questions sent to the winning fuller disclosure were inadequate.
23 Serco Limited v Secretary for State for Defence [2019] EWHC 515 (TCC)
Independent Review and Lessons Learned 23 24 Defence Fire and Rescue Project Annex A Terms of Reference - Summary
Programme Review objectives • The overall evaluation process, how it was developed, prepared and assured, whether & scope it was fit for purpose and how the final The Permanent Secretary and accounting officer dialogue phase was planned and managed would like an independent review to: • The process for selecting, training and • Assess the contributing factors to the managing evaluators and the recording of procurement process being open to scores challenge and the lessons to be learnt • The design, development, management, from the programme delivery - covering assurance and use of the financial model specifically, legal, commercial and financial advice and process, governance and • The issuing of award and non-successful assurance. award letters to bidders and the information provided • Identify and recommend the most important actions that could be taken on other • Governance and assurance of the programmes across Defence and potentially procurement wider Government including identification of • Use of external legal and financial advisors those that need to be taken urgently.
The review will not look at the whole of the procurement process since 2014 or be an in- depth review of the programme. It will focus on the final phase of the competitive dialogue from Sept 2017 and the evaluation of bids and the management of the procurement from that point onwards. Though not restricted to looking at any areas of the programme during that period if others appear important to the review team, we would expect the review to look into the following areas as a minimum:
Independent Review and Lessons Learned 25 Out of Scope Reporting and timescales
Action to be taken in regard to settlement or The output from the review will be a report finalisation of contract award and mobilisation. consisting of no more than a dozen core pages covering: Review Panel • Background and pertinent facts
A small independent review panel will be led by a • The lessons learnt and key findings non-executive director with relevant experience and skills from another relevant Government • Recommendations and most important department - Tony Poulter. Tony will be supported actions including identification of those that by an independent team consisting of a legal, need to be taken urgently. procurement and IPA specialist with dedicated administrative support to conduct the review. The Report should be delivered directly to the Permanent Secretary, ideally no later than the The review panel will report directly to the end of July. This assumes a start date of 3 June. Permanent Secretary, with advice and day to day support from the Director General Finance.
The review panel will have full access to the people and evidence needed to quickly and effectively get to the facts which will include Government Legal Department, the Cabinet Office and Deloitte and Legal Counsel as external advisers.
26 Defence Fire and Rescue Project