NATIONAL AUDIT OFFICE
REPORTBY THE COMPTROLLERAND AUDITORGENERAL
Ministry of Defence: Developmentsin the Reliability and Maintainability of DefenceEquipment
ORDEREDBY THEHOUSEOFCOMMONS TOBEPRINTED 28OCTOBER1994
LONDON:HMSO E8.15 NET 690 MINISTRY OF DEFENCE: DE”ELOPMENTS IN THE RELIABILrrY AND MAINTAINABlLrrY OF DEFENCE EQ”rPMENT
This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act.
John Bourn National Audit Office Comptroller and Auditor General 14 October 1994
The Comptroller and Auditor General is the head of the National Audit Office employingsome 800 staff. He, and the NAO, are totally independent of Government. He certifies the accounts of all Government departments and a wide range of other public sector bodies: and he has statutory authority to report to Parliament on the economy, efficiency and effectiveness with which departments and other bodies have used their resources. Contents
Page
Introduction and background 1
Developments in the management of reliabiity and maintainability 1
Conclusions 22
Glossary of terms and abbreviations 25
Appendices
1. Main conclusions and recommendations of the PAc’s 31st Report 26 1988-89 - Reliability and Maintainability of Defence Equipment - and Departmental Response in Treasury Minute dated 18 October 1989 (Cm 831)
2. Case studies 30
3. Surveys by NAO consultants 38 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT Report
Introduction 1 The Ministry of Defence (the Department) have long been concerned about the and reliability and maintainability (R&M) of equipment supplied to the Services for background two reasons. First, unreliability increases equipment support costs - by 21 billion a year according to the Department’s estimate in 1989 and improvements in reliability would therefore yield significant savings. Second, unreliability has adverse operational consequences by reducing the availability of equipment for training in peacetime; use in wartime; and through the effect on morale of loss of confidence in equipment
2 In February 1989, following a report by the National Audit Office (HC 173 of 1988-89) the Committee of Public Accounts examined the Department on the effectiveness of their management of R&M. In their Thirty-first Report, Session 1988-89 (HC 2061, the Committee noted that, despite a policy of treating R&M as an equal priority to performance, cost and time, the Department had often sacrificed R&M in order to achieve an early in-service date or to reduce initial procurement cost. The Committee were convinced, that if real achievements were to be made in realising the benefits of improved R&M, the Department must show a preparedness to delay or halt projects where R&M had not been fully considered or where evidence pointed to levels of R&M below those specified. The Committee also considered that it was in the field of contracting, particularly in regard to tighter specifications and greater use of incentives and warranties, that most scope existed for achieving an improvement in R&M. The Committee expressed a wish to return to this subject in the future.
3 The Treasury Minute of 18 October 1989 (Cm 8311 included a constructive response to the Committee’s specific conclusions and recommendations and reported on the wide range of action that was underway (see Appendix 11. The National Audit Office have examined the Department’s progress towards implementing the Committee’s recommendations. The expression “R&M” is generally used throughout the report as a generic term and is not intended to imply that considerations of both reliability and maintainability necessarily apply in each case.
Developments Departmental Action in reliability 4 In 1990, the Department drew up an action plan for improving the R&M of and defence equipment. The plan was endorsed by the Department’s Procurement maintainability Executive Management Board and the Department’s Principal Administrative Officers’ Committee. The newly formed post of Director of Reliability was tasked with co-ordinating action and monitoring progress. The Director made his first progress report in February 1993 in which he indicated that substantial advances had been made. MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAlNABlLlTY OF DEFENCE EQUIPMENT
5 The following paragraphs consider the Department’s progress on the Committee’s recommendations under five headings:
a) recommendations relating to the role of the Directorate of Reliability, the Committee for Defence Equipment Reliability and Maintainability (CODERM) and specialists in promoting improvements in R&M;
b) recommendations relating to the contracting for, and project management of, R&M;
c) the recommendation relating to the Department’s improvement of their Life Cycle Costing systems;
d) the recommendation relating to the Department’s improvement of their defect reporting systems; and
e) the impact of the action taken in reducing the cost of unreliability.
Committee recommendations (iii), (vi), (vii) and (viii) relating to the role of the Directorate of Reliability, CODERM and specialists in promoting improvements in R&M
Committee 6 (iii) The Committee stressed the importance of positive and effective action recommendations by the Department on the 1988 Report by PA Consultants on Contracting for Reliability.
(vi) The Committee expressed surprise that there were only 30 R&M specialists and looked to early progress by the Department in fulfilling their commitment to increasing this number.
(vii) The Committee welcomed the decision to appoint a Director of Reliability to provide a focus for R&M matters within the Department.
(viii) The Committee were concerned that the effectiveness of CODERM as a forum for developing attitudes on R&M matters should be improved and steps taken to ensure that CODERM’sdecisions were implemented.
Treasury Minute 7 The Department stated that they had taken positive and effective action on the 1988 PA Consultants’ report in the appointment of a Director of Reliability: and that the Director would be responsible for implementing those decisions taken by CODERM which did not naturally fall to any particular Controllerate or Service. The Department stated they had issued preliminary advice to Contracts Officers on contracting for reliability and were drafting guidance to Project Managers. The Department stressed the role already being played by seminars and publicity organised by CODERM in emphasising the importance of R&M. The Department also accepted the need for more R&M specialists and stated that steps were being taken to identify and provide the ex&a numbers needed. MINISTRYOFDEFENCE:DEVELOPMENTSINTHERELIABILlTY ANDMAINTAINABILITYOFDEFENCE EQUIPMENT
Subsequent 8 The Director of Reliability was appointed in July 1989 and is accountable to the developments Chief of Defence Procurement. He leads the Department’s drive for improved R&M of defence equipment and is responsible for policy and the monitoring of its implementation through scrutiny of projects. The complement of the Directorate was increased by two to eight in 1992 and the number of R&M specialists within the Department has also been increased.
9 The drive for improved R&M is coordinated and controlled through the action plan drawn up by the Director of Reliability in 1990. The plan covered a range of activities including improving R&M procedures and guidance: improvements to project statements of requirement and their scrutiny by the Director of Reliability; the inclusion of provisions for R&M in contracts; improvements in data collection; and increasing the number of R&M specialist staff. The Director of Reliability monitors progress in implementing the action plan which was updated and reapproved by the Procurement Board in February 1993.
10 The National Audit Office assessed the Directorate in terms of how they had carried out their role of monitoring to ensure that projects had taken full account of R&M requirements and CODERM’sguidance: how they had improved the effectiveness of CODERM, particularly in issuing guidance and standards and increasing the awareness of R&M; and how they had provided R&M training. The National Audit Office also assessed the availability of specialist R&M staff and the role of the R&M cells.
(a) Scrutiny of Project R&M by the Directorate 11 A significant new task given to the Directorate on its formation in 1989 was the scrutiny of projects for R&M at the approval stages. The Department’s 1990 Action Plan required the Directorate to increase to 100 per cent the proportion of relevant projects scrutinised and noted that additional staff would be required to do so. Two additional staff joined the Directorate for this purpose during 1992. In July 1992, new project sponsorship procedures were introduced which required staff to send to the Directorate all statements of requirements and procurement strategies for projects whose estimated development costs exceed E2 million or whose estimated production costs exceed E5 million.
12 The Department’s progress report on the Action Plan noted in February 1993 that whilst all such projects were being scrutinised for R&M, the depth of scrutiny needed to be increased. The Department told the National Audit Office that since then, R&M specialists in the Directorate had acquired greater experience and as a result the depth of scrutiny had improved.
13 The purpose of the Directorate’s scrutiny is to ensure that the R&M requirements for equipment projects are realistic, measurable, clearly specified and supported by an adequate contract strategy. The majority of project submissions for approval relate to the early stages of projects and so the Directorate of Reliability’s scrutiny is focused on the early stages when their advice can most easily and cost-effectively contribute to satisfactory R&M. In MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABlLlTY AND MAINTAINABILITY OF DEFENCE EOUIPMENT
1993 the Directorate had some 114 projects referred to them. Three of the Directorate’s staff were involved in assessing their R&M content and on average looked at each project about three times. This contributed to a heavy workload.
lb1 CODERM 14 All relevant areas of the Department are represented on CODERM’smain committee and representatives of industry, academic bodies and the British Standards Institute participate on its sub-committees. The Director of Reliability is Chairman of CODERM and has taken action to improve its effectiveness in a number of areas.
l By drafting the Defence Standards promulgating requirements and best practice to achieve good R&M. These are issued through CODERM, and since 1989, six Defence Standards have been published or re-issued. They are extensive and cover a range of issues relevant to good R&M including standards on In-Service Reliability Demonstrations and Off-the-Shelf procurement.
l A number of these Defence standards apply NATO-wide since they promulgate the Allied R&M Publications issued under NATO Standardisation Agreement (STANAGl4174. Such publications are authorised by most of the NATO countries and are therefore common standards for the NATO signatories and valid in collaborative projects and overseas procurements.
. Relevant sections of the 1992 Procedures for Project Sponsorship of Staff Targets and Requirements have been developed by CODERM to provide a tri-service basis for establishing R&M requirements in place of previously existing single service instructions and to set up a R&M panel system (see paragraph 291.
l The CODERM news-letter has been expanded and is published three times a year. It is an informal means of promoting R&M awareness and has a current circulation of 3600 copies.
15 In addition, the Directorate have reviewed Procurement Executive Controllerate instructionson R&M for currencyand relevance and co-ordinated action to issue R&M guidance under the 1991 revision of the Compendium of Guidelines for Project Management. The Director of Reliability chairs the North Atlantic Treaty Organisation’s R&M Policy Committee which also issues R&M guidance. R&M awareness has been promoted through the distribution of posters, videos and a Directorate of Reliability stand at defence equipment exhibitions.
(cl Training 16 In 1989, the training subcommittee of CODERM ran two training courses: a one day seminar and a five day course. Since 1989 the subcommittee has improved and extended the range of courses available. The one day seminar is taken frequently throughout the year to various parts of the country where there is a large presence of Service or Departmental staff 1539 Service and Department and 401 defence industry staff attended in 19931. The five-day course is now
4 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE: RELIABILITY AND MAINTAINARILITY OF DEFENCE EQ”,PMENT
run about eight times a year, on contract, by Portsmouth University and in 1993, 158 people attended the course. In addition, four new courses have been developed: a half-day seminar for Operational Requirements staff held twice a year; a two day seminar for one-star personnel held twice a year (in 1993 50 staff attended); an annual two-day seminar for R&M specialists; and two weeks of refresher training for R&M staff, based on a course previously offered by Birmingham University are now being conducted in-house.
(d) Availability of R&M Specialist staff 17 For the purposes of counting the numbers of specialists, the Department define a specialist as any person employed in an R&M post. On this basis, in their 1993 progress report, the Directorate identified 85 R&M specialists as being employed by the Department, 55 more than in 1989. However, much of this increase had arisen from the identification of posts which already existed but had not been counted before and the Directorate assessed the real increase to be of the order of ten R&M specialists. This was achieved at a time when other staffing levels were being reduced. In addition the Department encountered problems in obtaining suitable R&M staff because of a general shortage of skills and in response greater emphasis has been put on internal training. Just over half of the specialist staff are deployed in the R&M cells (see Table 1 overleaf). The remainder are in support of the Service departments.
18 NATO guidance issued in 1989 and endorsed by CODERM is that R&M specialists should be trained to second degree level. The Department told the National Audit Office that they were aiming to ensure that each cell includes, or has immediate access to, someone with second degree training. However, difficulties were encountered because of a lack of suitable UK courses, For example, in 1991, the Royal Air Force had to send some students to the United States. The Department have since reached agreement with a number of UK universities to adapt existing MSc courses and in 1993 six staff were undertaking post graduate training.
(e) The Role of the R&M Cells 19 Setting the requirements for R&M is the responsibility of the Operational Requirements staff and procuring equipment to satisfy the requirement is the responsibility of project managers. They are supported by 12 R&M cells who provide routine technical R&M advice. Some of the cells are part of the Services’ Headquarters Support Directorates, and the others part of the Procurement Executive (Table 1 overleaf). They have day to day responsibility for ensuring that Operational Requirements staff and project managers are kept aware of the need for, and provided with the means of, achieving good levels of R&M. In view of their importance, the National Audit Office conducted a survey of ten of the R&M cells to assess their role. The survey focused on the resources and technical experience of the cells. Their impact and effectiveness was, in part, assessed by the National Audit Office’s examination of 14 projects (paragraphs 33-381. The main findings of the survey are summarised below. M,N,STRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MALNTAINABILITY OF DEFENCE EQUIPMENT
~,,, ,, ,i ,, ,. c. ” ,’ Currefd ‘:; C&r$hj Y /. .’ SeaSystems I Royal Navy Staff LandSystems/ Army Staff Air Systems/ : stilr Numbers Numbers RoyalAir Force Numbers
ProcurementExecutive DirectorFuture Projects 1 Engineering 9 - Directorateof 7 Controllerrites (Naval) Management ProjectTime and 1 suppoll CostAnalysis - DirectorGeneral Surface (Ordnance) Ships (Naval) Statisticsand 3.5 ReliabilityBranch :, DirectorGeneral Surface Weapons(Naval) ,si,,. ,.. 3.5 i_ :*,j DirectorGeneral Underwater Weapons(Naval) _ t .;;~, - DirectorGeneral Fleet 3 DirectorGeneral 5.5 LogisticsSuppori 4”: ,, Headquarters Support(Ships) Equipment Services(Operational support support21 Requirements) Directorates (RoyalAir Force)5 (Operational - DirectorGeneral Aircraft 1 - Directorof Land 4 [seenote 21 Requirements) (NW ServiceAmmunition
Source: Ministry of Defence Notes: 1. Staff numbershave been calculated on the basisof time spenton Reliabilityand Maintainabili~activities. 2 Theoverall Logistics Suppoil Servicesorganisation a/so includes the ten R&M specialistsat the CentralServicing Development Establishmentat SwantonMorley Table1 shows the numberof Reliabilityand Maintainabilitycells by Serviceand ProcurementExecutive controllerates.
l The cells’ direct line management responsibility is to the Services’ Headquarters Support Directorates or to the Controllerates. They have close working relationships with the Directorate.
. Collectively the cells have some 46 staff and eight cells have increased their staffing by a total of 11 over the last five years. Seven cells said that they were adequately staffed but three considered that they were unable folly to discharge their responsibilities with present staffing levels.
l In two cells, three staff had had extensive prior experience of R&M; in the remaining eight cells staff had had limited, or no prior experience of R&M. The cells considered that it took one to two years in post to develop the skills to do their work fully. On average most staff stay in post for about four years.
. In the Royal Air Force Operational Requirements cell, three staff have MSc qualifications in reliability engineering since this is a condition of appointment of the posts they occupy. Three staff in the other cells are taking similar degrees. While this is a considerable advance on the situation
6 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELlABlLITY AND MAINTAINABILITY OF DEFENCE EOUlPMENT
five years ago, no other cell staff have, or are taking, such a level of training. Most cell members had attended CODERM’sfive day project management course.
l All cells were equipped with computer software to assist them in their work, and some computer equipment and software is being enhanced as financial constraints allow.
. Cells considered that they advised on a very high proportion of the total number of projects, but said that they relied on projects being referred to them before they could provide advice on them. Two cells did not consider that all Category C projects (those under E40 million estimated development cost or E75 million estimated production cost) were referred to them. Most cells considered that they looked at projects in sufficient detail but one considered that, as they were usually the last to be consulted when project requirements were being determined, they were constrained as to the time they could spend on detailed aspects.
20 These findings indicate that since 1989 the R&M cells have considerably improved their effectiveness in providing advice and promoting consideration of R&M in equipment development. They have also contributed to the improved standard of R&M requirements and the increased use of contract provisions for R&M; and they have assisted in CODERM training and standards.
21 The findings also indicate that, at present, staff allocated to the cells have limited prior experience of R&M. reflecting the difficulties of obtaining highly trained staff. However, the level of specialised training provided by tbe Department has increased, with some staff trained to second degree level (paragraph 18). Nevertheless the Department consider that there is a need to continue to train people in R&M. The findings show, furthermore, that the length of tours of attachment to the cells is on average four years, This is quite long by Departmental standards, but it does not necessarily ensure accumulation of long term expertise. The Department told the National Audit Office that they acknowledged both the advantages gained from the flow of staff into and out of the R&M cells and the need to accumulate long term expertise. Their career development policies aimed to achieve this balance both by developing the full potential of individuals and by providing sufficient trained staff for the Department.
Committee recommendations (ii), (iii), (iv), (v), (xi), (xii), (xiii) and (xiv) relating to the contracting for, and project management, of R&M.
Committee 22 (ii) The Committee acknowledged the effort the Department were now putting recommendations into R&M.
7 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT
(iii) The Committee stressed the importance of positive and effective action by the Department on the 1988 report by PA Consultants on Contracting for Reliability.
(iv) The Committee were concerned that R&M had not been “designed in” and demonstrated at the early stages of projects. Now that it was designed in at the feasibility stage and demonstrated at subsequent stages, they expected to see improved reliability and fewer reliability modifications in subsequent procurements.
(v) The Committee suggested that to provide assurance that reliability was being designed in at early stages, the Department should identify contractors’ planned R&M expenditure for all projects.
(xi) The Committee sixessed the need for the Department to keep alert to the danger that on fixed price and competitive contracts some contractors might seek to minimise R&M programmes and not divulge the results achieved.
(xii) The Committee expected the Department to place greater emphasis on the analysis and weighting of R&M proposals at tender assessment stage and ensure that they obtained full visibility of contractors’ activities in future.
(xiii) The Committee considered that the Department should place greater emphasis on full test, analyse and fix programmes followed by demonstrations before equipments entered service.
(xiv) The Committee were disappointed that there were relatively few examples of contractual incentives for R&M. They expected the Department to take a stronger line in future on the use of incentives and warranties.
Treasury Minute 23 The Department confirmed their commitment to improving R&M in future projects and reaffIrmed their policy that R&M should be afforded full consideration along with other factors such as performance, cost and timescale. The Department stated that they would make clear to the defence industry the importance that the Departmentattached to R&M. They further undertookthat, in the case of future projects, they would:
a) place greater emphasis on R&M at the tender assessment stage;
b) require project managers to ensure that R&M was “designed in” at the feasibility and development stages;
c) monitor contractors’ planned and actual efforts to achieve specified R&M:
d) set clear reliability targets in development contracts with an element of payment attached to achievement of the targets; or, if this was not possible, require project managers to pay particular attention to a contractor’s effort and expenditure on reliability as work progressed;
8 MINISTRYOFDEFENCE:DE"ELOPMENTSINTHERELIABlLITY ANDMAlNTAlNABlLITYOFDEFENCEEQ"IPMENT
e) emphasise full test, analyse and Rx programmes during development;
fi emphasise the need for reliability demonstrations before equipment enters service; and
g) take a stronger line on the use of contractual incentives and warranties for R&M.
Subsequent 24 The National Audit Office assessed progress in implementing these Developments recommendations through a review of the central action taken by the Department; an examination of 14 equipment projects to determine whether greater emphasis was being given to improving R&M; surveying defence contractors to ascertain their views on the Department’s approach to achieving R&M: and a comparison of the Department’s approach with that of large commercial organisations. In undertaking their assessment the National Audit Office were assisted by TA Consultancy Services Ltd.
(a) Action taken centrally by the Department 25 The Chief of Defence Procurement wrote to the defence trade associations in January 1990 to make them aware of the Department’s commitment to improving R&M and of the Department’s intention, in future, to include contractual conditions for the achievement of R&M. He told them that if equipments did not meet their contractual requirements for R&M, then the onus would be on contractors to carry out the work needed to put things right.
26 Since 1989, the Department have developed their contracting policy for R&M in accordance with their wider policies of taut fixed price contracts which make prime contractors fully responsible for all aspects of performance and which aim to achieve value for money through competition. Under this policy the Department seek to contract for R&M as specific terms of contract, thus treating R&M equally with equipment performance. They require that contracts should not be completed until the specified R&M are achieved and, in certain cases, final payment may be withheld until this has been done.
27 The Department consider that interim achievements of R&M during development programmes are merely indicators of progress and should attract only interim payments repayable if the contract is not fulfilled. Their policy is that payment should be linked to the demonstration of R&M in-service and where this is not practicable alternative approaches including warranties should be adopted. However, most of what is offered under warranties is already the Department’s right under the Sale of Goods Act and they consider that warranties should therefore be used with caution in order not to undermine their normal legal rights.
28 To ensure the success of this policy, the Department further seek to state their requirements clearly and to award contracts to contractors who are competent, appropriately resourced and financially sound.
9 MlNlSTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABlLITY OF DEFENCE EQUIPMENT
29 The Land Systems Controllerate introduced an R&M Panel System to regulate the provision of R&M requirements. This has been developed and adopted throughout the Department. Panels generally sit on all projects where estimated development costs exceed E5 million or estimated production costs exceed El0 million. They consist of:
. Concept Panel/Panel A which advises the project sponsor on the R&M requirements for the Staff Target and Staff Requirement;
l Project PaneL’Panel B which advises the project manager on aii R&M matters, including the consistency of the Technical Specification with the Staff Requirement and on matters during Project Definition, Development and Production: and
l In-Service Panel/Panel C which advises the project manager and Service equipment support manager on all R&M matters concerning in-Service use of the equipment.
30 The R&M Panel system was promulgated in the Department’s “Procedures for Project Sponsorship of Staff Targets and Requirements” in July 1992 and adopted last by the Sea Systems Controllerate in January 1993. When the Panel system is properly applied, the Department have found it to be highly effective. However, the Directorate of Reliability reported to the Procurement Board in February 1993 that not all projects were implementing the system fully and in particular, that Concept Panels were not being formed early enough or, in some cases, at all. The Department told the National Audit Office that they were considering monitoring the formation of R&M panels.
31 The Department also have other means of controiling and monitoring progress on R&M of projects. Since 1992 project managers have been required to obtain the agreement of the operational requirement sponsor to the contract specification for R&M. The Department monitor the percentage of relevant contracts with provisions for R&M. For example, in 1991-92, 92 per cent by value of contracts were recorded as containing provisions for R&M against a target of 80 per cent and the Department believe that in 1993-94 all contracts will comply.
32 Project achievement against targets for delivery to time, cost and operational performance standards which include R&M is also reviewed. In addition the Deparnnent’s procedures now require the setting up of Project Review Boards, but there is no separate system of routine progress reporting on compliance with R&M procedures. This could be important for higher risk projects, particularly during their later stages of development and production. The Department told the National Audit Office that although their policy was to keep routine progress reporting to the minimum in the interests of economy, they were considering procedures to evaluate all aspects of projects at the end of major phases and the resource implications. The evaluations would include R&M and would provide a timely and comprehensive basis for disseminating lessons learned on projects.
10 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT
(b) Examination of projects 33 In order to assess the extent to which greater emphasis was being placed on improving R&M the National Audit Office examined 14 projects where there had been some scope to influence R&M requirements since the Committee of Public Accounts’ Report in 1988-89. The projects (Table 21 included representatives from the Department’s Land, Sea and Air Systems Controllerates and ranged in size from categories A and B (over f.40 million estimated development costs or over E75 million estimated production costs) to categories Cl and C2 (over
Procurement CategoryA/B Category C Estimated (f115million Description cost (fxi-115 Description cost in value or f million million) f million above) (93-94 prices) (93-94
I,r,*,_l_-,s m.L”;,jI. s.;ehist; * “. ? ,, iorpebo : bS,&:qs,ems” .C i) “1,005 Sea-Dart-LLF Fuzefor sea-to-air missile
LandSystems Phoenix Unmannedair 264 RCEATM Anti-tankmine 32 reconnaissance vehicle
AS90 155mmHowitzer 486 SWIG Improvement 44 packagefor Swingfire anti-tankweapon
Challenger2 Main BattleTank 1,000 Vehicle ArmouredLand 57 ProtectionKits Rover
SmallArms Smallarms firing 25 Trainer rangesimulator
Air-launched anti-radarmissile
HarderTIO Z-seattrainer aircraft
Low light visual 248 systemfar fast jets
Skynet4, Stage2 Communication 343 satellite
Thetc estimatedvalue of theseprojects is somef5,004 millions
11 MINISTRY OF DEFENCE: DE”El.OPMENTS IN THE RELIAEm.ITY AND MAINTAIN*e,LrrY OF DEFENCE EO”lPMENT
E5 million and under S40 million estimated development costs; or over El0 million and under E75 million estimated production costs). The results of this examination for five projects are summarized in Appendix 2.
34 Eight of the projects examined showed that very satisfactory attention had been given to R&M requirements and that the Committee’s recommendations where applicable had been adhered to. At the time of the National Audit Office’s examination some of these projects still had R&M activity, such as in-service reliability demonstrations, in prospect. Two particularly good cases were:
. The Challenger 2 Main Battle Tank’s R&M requirements are linked to substantial milestone payments based on a reliability growth test. Together these milestones account for 37 per cent of development expenditure. The contractors, Vickers plc, have now met these very tight and demanding development R&M requirements. There is also contractual provision for an in-service reliability demonstration to prove that the R&M requirements are met in production vehicles.
l Spearfish. This Sea Systems’ torpedo project started in the 1970s. thereby pre-dating the Committee’s recommendations. It was subject to many reliability problems including ambiguous requirements and poor contractual incentives. At acceptance trials, it achieved less than half its required reliability.
Since 1990, however, the Project Office, assisted closely by the Directorate of Reliability, have negotiated within the original contract price a reliability assurance programme with the contractors (GEC-Marconi) to rectify all these problems. New reliability demonstrations are drawing to a close and indicate so far that the required reliability will be met.
35 These findings suggest that on many projects a marked improvement has occurred in the Department’s management of R&M since 1989. For the remaining projects the National Audit Office found that generally R&M guidance was complied with but there were a number of areas where some problems had arisen. These are summarised below.
. Clear and timely specification of R&M requirements. This applied to two projects: Phoenix, a remotely-piloted air reconnaissance vehicle; and the Aimed Controlled Effect Anti-Tank Mine (ACEATM). The Phoenix contract was let in 1985 and pre-dated the Committee’s Report. The Phoenix In Service Date has now been delayed by about three and a half years until late 1995. The project’s contracted reliability requirement is inadequate. There is no problem with the maintainability requirements. In 1992, the Department investigated the option of changing the reliability requirements in the contract. However, this option was considered impractical because the contract was originally awarded under firm price competitive conditions and any change to the reliability requirements would have required technical, programme and price changes. Earlier action, probably in 1989, might have been more successful. Generally, design changes to incorporate new requirements are easier and less costly the earlier they are undertaken. MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELlABlLlTY AND MAINTAINABILITY OF DEFENCE EQUPMENT
l The staff requirement for ACEATM was written after the United Kingdom was contractually bound to a tri-national technical specification with France and Germany and also while tenders were being assessed. The National Audit Office accepted that it was extremely difficult for the Department to secure adherence to normal procedures when dealing with collaborative projects. Nevertheless, it was important, in such circumstances, to ensure that the Department’s essential requirements were taken account of, if collaborations were to be justified.
l Monitoring and visibility of R&M during development. In the case of the Swing Fire Improved Guidance (SWIG) project, an improvement package for the Swing Fire Anti-Tank Guided Weapon, there were requirements for reliability plans and procedures which were monitored by the Department’s reliability panels. But there were no contractually-linked R&M trials during development. There were three R&M related milestone payments for the production of maintenance manuals, an ease-of-maintenance demonstration, and the completion of a Service Reliability Assessment Programme. However, the last mllcstonc falls due two years after the equipment is in-service. There is also uncertainty because of recent defence reductions and unexpected changes in Army deployment that the equipment will receive enough in-service use during the Assessment Programme to test its R&M requirements fully. In the National Audit Office’s view, these arrangements for trials and milestones were insufficient to give the Department contractual means to ensure the R&M requirements were properly addressed during development. l With ACEATM, a collaborative project, the contract did not provide for a reliability growth programme. One was offered at additional cost but this was rejected by the other two countries as unaffordable. In addition, the contract did not include specific terms such as definition of failure. Instead it required the contractor to provide his own definition of failure, which he has subsequently done. l R&M testing prior to acceptance. Greater emphasis is now placed on In Service Reliability Demonstrations and other forms of testing prior to acceptance of equipment. However, the National Audit Office found in the case of four projects that there was potential to tighten the scope of such testing. l For example, for the Advanced Short Range Ah-to-Air Missile (ASRAAM), an in Service Reliability Demonstration is tied to retention of five per cent of the development price, some f12 million. However, irrespective of the results of the demonstration in order to allow for the possibility that the RAF might not carry out sufficient air carriage, under the contract, half the retention has to be paid after one year and the remainder after two years. The Department have long been aware of this problem and are currently seeking to amend the ASRAAM contract to tie retention payments more firmly to reliability achievement.
13 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND iWJNTA,NABlL,TY OF DEFENCE EQUIPMENT
l Incentives to achieve specified levels of R&M. Financial incentives to secure achievement of better R&M are now widely used, taking the form of milestone payments, warranties or incentive schemes. However, in their sample of projects the National Audit Office found one example where incentives might have been stronger.
. In the case of the SWIG improvement package for the Swing Fire Anti-Tank Guided Weapon, only f60,OOO of milestone payments out of a total project cost of f44 million were originally dependent, under the connact, on successful completion of the project’s two-year Service Reliability Assessment Programme; however, the contract has recently been amended to increase this figure to f200.000 and extend the Assessment Programme to 27 months. The contractor is also liable to pay for the costs of any modifications to the equipment made necessary by shortfalls from the R&M requirements discovered during the Assessment Programme; but this liability would not include consequential or operational costs. Furthermore, because of defence reductions and unexpected changes in Army deployment, it is not certain that Swing Fire will undergo enough use during the Assessment Programme to test the R&M requirements fully.
l The Small Arms Trainer is an Off-The-Shelf fire arms training range simulator involving a limited degree of customisation. Its principal justification is to achieve cost savings against the alternative of live firing ranges. Good R&M is crucial to achieving this and the contract for the equipment expressly sets and requires adequate levels of R&M. At the tender stage, the Department weighted R&M at less than half the weighting attached to achieving specified performance requirements; and the project’s R&M requirements were not comprehensively tested prior to acceptance. This route was followed because the project was an Off-The-Shelf procurement of a mature product with largely known R&M and because the Department intended to use a maintenance contract further to ensure good R&M In Service.
l The maintenance contract, as obtained, provided an incentive scheme and a contractually-binding clause obliging the contractor to remedy any reliability defects,including designdefects, that might arise during the first three years of the equipment’s ten-year life. Moreover, under the Sale of Goods Act and English Law, the Department are entitled to claim remedy of defects during the first six years of the equipment’s life.
l The maintenance contract is working but has areas of weakness. For example, the National Audit Office considered that the incentive scheme was too highly graduated to be fully effective and that the payments to which it related were inadequate. Whilst the principle of using incentives and obligations to remedy defects as a means of securing the R&M of Off-The-Shelf projects is sound, it requires careful thought in its application.
14 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABILITY OF DEFENCE EOUIPMENT
l The Department acknowledged these potential weaknesses. But they took the view that, given that the equipment was a mature product and that some 650 sets of equipment are in use worldwide, and given their extensive rights to the remedy of any design defects, the steps taken had been sufficient to address the risk of inadequate R&M.
36 Most of the projects which the National Audit Office examined were fixed or firm price. Monitoring of contractors’ R&M efforts in the case of these projects was addressed through “test, analyse and fix” and demonstration programmes. There was therefore no contractual provision to allow the Department to monitor contractors’ expenditure on R&M in these cases since this would clearly have been inappropriate.
37 Of the 14 projects examined by the National Audit Office, four included specific warranties designed to give the Department redress if the specified levels of R&M were not achieved; the other ten projects relied upon the terms and conditions of contract underpinned by the buyer’s rights in law. The Department consider that careful attention should be paid to the wording of warranties in order that their inclusion does not undermine the Department’s rights under the Sale of Goods Act. A judgement is made by the project manager in a case by case basis, advised by his contracts staff, as to whether the Department’s R&M requirements are more likely to be met by the inclusion of specific warranties (which have the merit of their certainty of operation) or by reliance upon taut contract conditions underpinned by the buyer’s rights at law.
38 Overall, the results of the National Audit Office’s examination of these 14 projects showed that the Department had usually adequately defined and specified what R&M was required for a project; but had sometimes encountered difficulties in translating the R&M required into contractually-binding provisions during development and production. In addition, of the six projects which encountered difficulties, most (four) were lower-value category Cs rather than higher value category As or Bs.
(c) Views of Defence Contractors 39 In order to assess the views of Defence contractors on the Department’s current approach to R&M, the National Audit Office’s consultants carried out a survey of twelve firms representative of the United Kingdom’s Defence industry. Nine contractors responded to the survey and the results are summarised in Appendix 3. The main findings are as follows:
. The great majority of contractors said that the Department were according R&M equal priority with equipment performance.
. All contractors considered that R&M contractual conditions had become stricter over the past five years and that R&M targets or requirements were always specified in the contract. Although in nearly all responses payments were associated with R&M milestones, some contractors considered that the actual incentives to achieve R&M were negligible.
15 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAlNTAlNABlLITY OF DEFENCE EQUIPMENT
w Only three of the nine respondents considered that the Department’s R&M requirements were realistic. The major concern expressed was that targets were sometimes set too high.
. Generally, contractors considered that the Department monitored R&M effectively and sufficient progress reviews were held. But some considered that the level of interest in monitoring R&M progress tended to decline as projects entered production and in-service phases.
. Contractors considered that the Department’s project officers did not have adequate R&M knowledge and background. However, the situation appeared to improve as projects progressed.
. The Department’s specifications for R&M demonstrations were generally considered to be inadequate for a number of reasons, including deficient failure definitions and insuiRcient duration times for demonstration trials.
40 On the last point, the Department consider that they have taken positive steps to improve demonstration of R&M contractual compliance. A Defence Standard on In Service Reliability Demonstrations was published in January 1993 and a standard on Maintainability Demonstrations is being prepared.
(d) Comparison with the private sector’s approach to R&M 41 The National Audit Office commissioned their consultants to carry out a survey of six commercial organisations, to assess how the Department’s approach to R&M compared to that in the private sector. Five companies replied and the results are summarised in Appendix 3. The main points are listed below.
l Like the Department, all respondents specified R&M requirements to suppliers and took R&M into account in selecting and approving them.
. All respondents specified suppliers’ procedures for carrying out R&M activities. However, in contrast to the Department’s policy and practice, only three of the five specified the means of demonstrating R&M.
o Noneof thecontractors considered that their R&M contractual conditions had become stricter over the past five years.
42 The Survey suggested that the Department were now adopting a more rigorous approach to R&M than their commercial equivalents.
Committee recommendation relating to the use of Life Cycle Costing systems
Committee 43 (ix) The Committee emphasised that the lack of Life Cycle Costing systems to recommendation enable the costs and benefits of improving R&M to be evaluated was a serious weakness which they expected the Department to take early and positive action to remedy.
16 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT
Treasury Minute 44 The Department stated that all proposals with significant financial implications would be subject to formal investment appraisals and that these appraisals would in future include the best available data on Life Cycle Costing.
Subsequent 45 The National Audit Office found that out of the 14 projects they examined four action were too far advanced at the time the Committee’s recommendations were published to make an investment appraisal practicable. Of the remaining ten projects, seven had been subject to an investment appraisal. But for only one project, Challenger 2, had the life cycle costs of alternative levels of R&M been calculated in order to identify which was the most cost effective. This reflects the inherent difficulty of examining the cost implications of small changes in R&M requirements.
46 However, in 1991, the Department established new arrangements to ensure the implementation of Life Cycle Costing. These include a policy committee on which the Director of Reliability is represented and four sub-committees dealing with data availability, standards and definitions, techniques and industrial liaison. Progress in developing models to determine the Life Cycle Costs of alternative levels of R&M is summarised below.
. Air Systems Controllerate use their existing Life Cycle Costing models to examine the Life Cycle Cost implications of the R&M targets being discussed for inclusion in projects. These models have been used in this manner for the last two years and the technique continues to be refined. They are being applied to all projects which can benefit from them; because of the timeframe no project to which they had been applied had reached sufficient maturity to be included among the projects examined by the National Audit Offtce (paragraphs 33-38).
l Land Systems Controllerate have used a series of Life Cycle Costing models to analyse projects, including alternative levels of R&M since 1989. These are used for all category A projects (those exceeding E75 million estimated development costs or El50 million estimated production costs) and most category B projects (those exceeding f40 million estimated development costs or E75 million estimated production costs).
. Lower value projects, and most Land Systems projects in the National Audit Office sample were of this type, are only subjected to this type of modelling in special circumstances. Individually, the models Land Systems are using have a number of serious limitations - for example, some are not able to produce year by year castings, making them unusable for investment appraisal. Used together, however, the models are less limited. Land Systems Controllerate are developing, in conjunction with the Army Department (the Quarter Master General), an Equipment Support Model which will have much improved capabilities particularly for Life Cycle Costing estimation. The project definition phase of this model is due to be completed by the end of October 1994.
17 MINISTBY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAlNTAINABlLlTY OF DEFENCE EQUIPMENT
l Sea Systems Controllerate have developed a suite of models and methodologies since 1978 to predict the availability, reliability and Life Cycle Costs, and conduct Investment Appraisals, of alternative platform and equipment designs. These models can be used to produce Life Cycle Costings for design options with alternative levels of R&M. Comprehensive Investment Appraisals are prepared for alI platform projects and there are Life Cycle Costing and R&M studies for all Category A and B projects (that is, projects with estimated development costs of over f40 million or estimated production costs of over f75 million). This expertise is also being used to advise in new platform concept design work.
47 Models for costing alternative levels of R&M are dependent on a number of assumptions for example, how many equipments will be bought, how they will be deployed and what maintenance policy wiIl be used. The results of such castings are indicative, but the models are powerful tools to help ensure that the optimum levels of R&M are set for projects.
Committee recommendation relating to the use of defect recording systems
Committee 48 (xl The Committee were concerned that present defect reporting systems were recommendation unable to provide adequate R&M data. They expected the Department to set clear objectives and stages for the achievement of improvements.
Treasury Minute 49 The Department undertook to set clear objectives and stages for the achievement of necessary improvements in their current defect reporting systems, which had not been designed to provide suflicient data for full R&M analysis.
Subsequent 50 Reliable data on equipment performance, including the incidence and cause of developments defects, are important to monitor the levels of R&M achieved; to influence R&M specifications for similar future equipment; and to provide evidence for possihle claims against contractors for R&M faults, The Department consider that one of the keys to improving their data recording systems lies in the increased use of automation and computer technology,
51 To meet this need the Department decided that in the short term the Services should take interim action to improve R&M data collection, Fully comprehensive action on these lines has not yet been practicable because it would have pre-empted or wastefully duplicated the new Service systems scheduled for implementation by about 1996-97 as part of the Department’s overall non-operational lnformation Technology Strategy. It is these new Service systems which are intended to provide comprehensive data.
52 Action taken by each of the Services to date is summarised in Table 3 opposite. This shows that the systems currently in use have a number of deficiencies; that the interim systems now or shortly to be implemented will remedy some of these; but it is not until 1997 when new Information Technology systems have been introduced that automated data collection will be available.
18 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELlABlLlTY AND MAINTAlNABlLlTY OF DEFENCE EQUIPMENT
Royal Navy Army RoyalAir Force
CurrentSystem CurrantSystem CurrentSystem CurrentNavy systems depend on a numberof CurrentArmy defectreporting systems are Althoughthecurrent MaintenanceDataSystem farms usedin serviceto reportdefects, largelymanual and, accordingto a recent capturessome data on everyaircraft in-service incidents,repair activity and sparesusage. CODERMreport, the standardof datacapture and someother in-serviceequipments it has a Theserange from high percentagecapture is not good,with manydefects going numberof sh0rtc0minQs:it requiresmanual systemsassociated with nuclearpropulsion unrecorded,though since September1993 a completionof forms which makesit error and submarinesafety, to simpler systemsfor new EquipmentFailure Report form has been proneand it doesnot systematicallyidentify surfaceships. All dependon manualinput of in use. usagewithout which dataon equipment datavia mainly paperforms and arethus prone defectscannot be easilytranslated into an to error.Similarly, data cannot easily be R&M measurement. transmitted,collated or analysed.
Interim System Interim System Interim System All majorships and all submarineshave An interim computerisedsystem, the Giventhe system’srelative sophistication, the On-BoardAutomatic Data Processing BattlewinningEquipment Reliability Return has Air Forcehave not consideredit cost-effective InformationSystems (OASIS) installed. An beenintroduced at a cost of f600,OOO.It is to embarkon substantialinterim EngineeringAdministration System is being specificallydesigned to collectavailability and improvements. incorporatedwhich will facilitatethe basicreliability data which will be input at unit productionand transmissionof defectreports level but will be capableof aggregationto and R&M datainto the Availability,Reliability Army-widelevel. All majorArmy equipment and MaintainabilityManagement Information with a significantbattlefield role will be covered. System.Initial fits of the Engineering ManagementSystem have been made to four ships and two submarines.Fitting to a further eight ships has beenapproved and plansare in placeto extendthis to additionalships and submarinesin 1995-6.
FutureDevelopments FutureDevelopments FutureDevelopments A new engineeringmanagement system is The Unit ComputingSystem (UNICOM), which Majorimprovements are plannedwith the beingdeveloped as part of the Upkeepand is being deployedover the period1994-97, will implementationof the RoyalAir Force’s StoresSupply System (UPKEEP), which is in captureall necessaryR&M datathrough its Logistic InformationTechnology Strategy in turn part of the NavalLogistics Information equipmentsupport modulewhich is currently tranchesfrom 1995 onwards.This Strategy System(NLIS). UPKEEP will include beingdeveloped and which will be was approvedin 1990and a development applicationswhich will provideR&M data,the incorporatedover the period1995-97. It will contractlet in 1992. It will coverthe whole field exactrequirements for which are currently feed informationto higher-levelsystems, now of Air Forcelogistics, including R&M, and will beingexamined. beingintroduced, which will then be ableto subsumeand extendthe existingcapabilities of providenot just basicdata but analysesand the MaintenanceData System. It includesa The FleetAir Arm havea separatestrategy, modelsof the life cycle implicationsof R&M. requirementto providedata for in-depth R&M FleetAir Arm InformationTechnology (FAAIT), analysisof an extensiverange of equipments which has madesignificant progress. It will and it will havea facility which will enable collectimproved R&M and cost dataand is equipmentavailability to be monitored. It will scheduledfor completionin 1995,appreciably also be ableto compareactual equipment in advanceof the three Servicestrategies. performancewith forecastsand provide informationon the cost of maintenance,as well as carry out complexmodelling. soone: Ministry of Defence This table recordsthe progresseach Service has madein implementingdefect reporting systems
19 MlNlSTRY OF DEFENCE: DEVELOPMENTS IN THE RIXIAHILITY AN” MAlNTAlNABlLITY OF DEFENCE EOUIPMENT
53 Sophisticated measures for the processing and analysing of information require that the data that they process are accurate. Manually collected data are sometimes highly subject to human error. They are also subject to staffing constraints. For example, the Naval Engineering Support System (NESS) 2, a trial R&M data system, proved unsuccessful because ship-board manpower resources to collect data were not available.
54 Scope still exists for the use of manually-collected data where these data can be accurately and cost-effectively gathered. For example, the Army’s Battle winning Equipment Reliability Return initiative (see Table 3) will provide Commanders with information on equipment availability and the Department with fleet reliability data. However, in many areas, the way forward is through the automation of data recording by means of the installation of electronic measurement devices. One of the most important types of such devices is Elapsed Time Indicators which record the time during which an equipment has been used. These are quite simple devices with little capacity for recording which significant events have occurred - as opposed to what time has passed during equipment use. They are not universally applicable. But, in many cases, they can provide measurements crucial to the accurate calculation of levels of R&M achieved.
55 The Department told the National Audit Office that progress had been made towards automation. Recently the Navy have conducted trials with hand-held computers as a first step in this direction. Moreover, as more computer software is included in defence equipment, so it becomes increasingly possible to automate data recording. However, the increasing use of software also makes the recovery of such data in a directly usable form more difficult without a well programmed data base.
56 The decision as to which equipment should be fitted with Elapsed Time Indicators and other automatic data gathering devices, and when, is taken by project managers as advised by the appropriate R&M cell. If the devices are to be installed this must be done during design and production because it is usually not cost-effective to retrofit them when the equipment is in service. The National Audit Office found that neither the Department nor the Services were monitoring centrally the extent of the installation of such devices.
57 A Strategy for R&M Data Collection. The Department have recognised that they need a strategy for R&M data collection to establish what information is required; what analysis is necessary to convert raw data into usable outputs: to avoid unnecessary duplication between the Services; to ensure that Operational Command will be able to access support information costs and other data without having to recover them in different forms from each Service; and to ensure that existing and future systems will be adequate to meet R&M data collection and analysis requirements. The Department have not yet fully developed their overall strategy, although the Department Plan aims to develop procedures for the analysis and electronic gathering of data on equipment availability and R&M by March 1995.
20 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAlNABlLlTY OF DEFENCE EQUIPMENT
58 The Directorate of Reliability had, in early 1993, proposed to appoint a committee to establish a strategy for R&M data collection. This task was, however, subsumed by the Steering Group for the Computer Integration of Requirements, Procurement, Logistics and Support. A primary aim of this Steering Group is the development of a Department-wide strategic approach to information which will include a harmonised R&M data collection regime.
59 The Services’ Information Technology Strategies (paragraph 511 are now quite far advanced and the Department’s Procurement Board have specifically required that delay should be avoided in the development of approved projects. Consequently, the scope for an overall strategy to influence the emerging R&M data collection systems is limited. In this situation, thcrc is a need, recognised by CODERM in 1992, to co-ordinate the systems. CODERM noted that there was a lack of common approach between the Services’ various data collection systems and considered that there was a need for harmonisation. The Department told the National Audit ORIcethat the over-arching strategy being developed by the Steering Group for Computer Integration of Requirements, Procurement, Logistics and Support would provide such harmonisation.
60 Some preliminary action has been taken. In 1992 CODERM produced a position paper on R&M data collection and they have also issued a Defence Standard on the subject. A working group called the Follow-up of Liability for Equipment Defects Working and Implementation Group (FLEDWIG) was set up to examine the Services’ defect-reporting systems, to ensure that they provide as suitable a basis as possible for making claims against contractors, which will include those arising from R&M faults. Twenty-two items of information were identified as necessary for this purpose and each of the Services have agreed to introduce systems to provide the required information. These systems will, in the interim, necessarily be manual. The Army, however, have already revised their defect reporting system and their Equipment Failure Report, introduced in September 1993, incorporates the 22 items of information referred to.
Committee recommendation on reducing the cost of unreliability
Committee 61 (i) The Committee noted that the Services had estimated that unreliability recommendation added over El billion a year to support costs. They accepted that not all such costs could be saved but noted that the Department considered 50 per cent to be a very fair goal for the level of achievable savings.
Treasury Minute 62 The Department stated that the cost of unreliability was too high and that saving half this cost in the years ahead was a reasonable goal. They stated that they were planning a sustained programme to bring about substantial improvements in the reliability of new and modified equipment as it was developed and brought into service. MiXISTRYOFDEFENCE:DEVELOPMENTS1NTHERELlABlLlTY ANDMAlNTAlNABILlTYOFDEFENCEEQUlPMENT
Subsequent 63 Savings, as a result of improved R&M, are realised only as new equipment, developments benefiting from the new R&M policies, is brought into service and this is now starting to happen, However, the Department are unable to quantify the R&M savings achieved since 1989. This is because any measure of savings requires a firm baseline but the Department told the National Audit Office that the figure quoted in 1989 was very much a broad estimate and a more accurate baseline was not available. Furthermore, other variations such as any change in the equipment, its role and use, would need to be discounted to arrive at those savings which could be accurately attributed solely to improvements in R&M. But these are difficult to isolate. Finally, although the Department are improving their data collection systems, they are not yet generally advanced enough to quantify R&M improvements and the resulting cost savings.
64 Nevertheless, there is evidence that the Department have been successful in implementing the Committee’s recommendations and from this it can be assumed that improvements in R&M have been achieved and savings will flow increasingly from them as new equipment comes into service. The Department are confident that the cost of unreliability, estimated at El billion in 1989, will be significantly reduced. Considerable progress has been made and the Department recognise that there is scope for further progress and are committed to a continuing drive to improve the R&M of defence equipment.
Conclusions 65 Overall, the Department have responded positively to the conclusions and recommendations of the Committee of Public Accounts, and have pursued them in a systematic manner by the creation of a central focus for R&M and through their action plan. The National Audit Office’s findings were as follows:
a) The creation of the Directorate of Reliability to lead the drive for improved R&M and the implementation of the action plan have been successful in raising the priority given to R&M. The Directorate’s role in scrutinising projects’ statements of requirements is very important in ensuring that R&M is given full consideration from the outset and this has been effective in achieving improvements (paragraphs 11-13).
b) The Directorate of Reliability have made good progress in improving the effectivenessof the Committee for DefenceEquipment and Reliability and Maintainability (CODERM):developing the range and quality of R&M training; and in generally promoting an awareness of the importance of R&M through extensive guidance and publicity (paragraphs 14-16).
c) Since 1989 the number of recorded R&M specialists has increased by 55, from 30 to 85 staff. Much of this has arisen from the better identification of existing posts and the real increase is of the order of ten staff. However, this was achieved at a time when other staffing levels were being reduced. The Department also encountered problems in obtaining suitably trained R&M staff because of a general shortage of skills and in response greater emphasis has been placed on internal training (paragraphs 17-18).
22 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELlABlLlTY AND MAINTAINABILITY OF DEFENCE EQUIPMENT
d) A large proportion of R&M specialists are deployed in the R&M cells and they have been effective in ensuring that greater attention is given to R&M. The level of training provided to staff in the cells has increased, with some staff trained to second degree level. Nevertheless, the Department consider that there is a need to continue to train staff in R&M (paragraphs 19-21).
e) The Department have taken positive action to make the Defence industry aware of the importance they attach to R&M. They have also developed their contracting policy to contract for R&M as specific terms of contract and to link payment to contractors to the demonstration of R&M in-service. Where demonstration is not practicable, alternative approaches including warranties are adopted (paragraphs 25-28).
fJ The Deparlment have a variety of means of monitoring and controlling progress on R&M both for individual projects and for equipment procurement as a whole. However there is no separate system for monitoring the detailed implementation and success of policies for improving R&M on projects. This could be important for higher risk projects, particularly, during their later stages of development and production. The Department are seeking to develop further means of monitoring and disseminating the lessons learned across the range of project activities which includes R&M (paragraphs 29-32).
g) Eight of the 14 equipment projects examined by the National Audit Office showed that very satisfactory attention had been given to R&M requirements. These findings suggest that a marked improvement has been achieved in the Department’s management of R&M. However, some problems had arisen on other projects. For example, in the case of two projects there needed to be a clearer and more timely specification of R&M requirements; with four projects there was potential to tighten the scope of R&M testing prior to acceptance; with one project incentives to achieve specified levels of R&M might have been stronger; and in the case of the Small Arms Trainer the contract provided for an incentive scheme which was subject to a number of weaknesses (paragraphs 33-38).
hl In general, Defence contractors consider that the Department have made contractual conditions stricter for R&M and have given it equal priority with equipment performance. In addition, a survey by the National Audit Office suggested that the Department were adopting a more rigorous approach to R&M than their commercial equivalents (paragraphs 39-42).
i) Progress has been made in developing models to determine the Life Cycle Costs of alternative levels of R&M and a committee structure was established in 1991 to promote the implementation of this important technique (paragraphs 45-471. jl In the short term, the Services have taken interim action to improve R&M data collection and defect reporting. These systems have a number of deficiencies but more comprehensive data is scheduled to become available MlNlSTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT
in 1997 when new systems currently in development are introduced as part of the Department’s overall non-operational Information Technology Strategy (paragraphs 50-521.
k) One of the keys to improving data collection is the use of electronic devices, such as Elapsed Time Indicators, to measure equipment usage and performance. The Department consider that progress has been made in implementing these although there is no monitoring of the extent to which they are being installed (paragraphs 53-561.
1) The Department have recognised the need for a strategy for R&M data collection to avoid unnecessary duplication between the Services; and to ensure that existing and future systems will be adequate to meet R&M data collection and analysis requirements. The Department have not yet fully developed their overall strategy, although the Department Plan aims to develop procedures for the analysis and electronic gathering of data on equipment availability and R&M by March 1995. In addition, a Steering Group for Computer Integration of Requirements, Procurement, Logistics and Support has been established to ensure a Department-wide strategic approach to information which will include a harmonised R&M data collection regime (paragraphs 57-601
m) Savings from improved R&M are realised only as new equipment, benefiting from the new R&M policies, is brought into service and this is now starting to happen. However there are difficulties in quantifying the savings arising from improved R&M and it is not possible to quantify the Department’s progress towards achieving their goal to halve the cost of unreliability. Nevertheless, as a result of the considerable improvements they have introduced, the Department are confident that the cost of unreliability, estimated at Sl billion in 1989, will be significantly reduced (paragraphs 63-641.
66 Since 1989 the Department have set in motion a sustained programme to bring about improvements in R&M. The role of the Directorate of Reliability as a central focus, the introduction of better R&M specifications and tauter contracting should ensure that savings will arise as new equipment comes into service. In addition, improved R&M should increase the availability of equipment for training in peacetime and use in wartime, and assist in minimising life cycle costs. This is a significant advance and the Department intend to ensure that there is sustained pressure to continue the improvements already begun.
24 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT Glossary of terms and abbreviations
ACEATM Aimed Controlled Effect Anti-Tank Mine
ASRAAM Advanced Short Range Anti-Aircraft Missile
CODERM The Committee for Defence Equipment Reliability and Maintainability
In Service Reliahiity A demonstration of the reliability achieved by an equipment against the Demonstration requirements specified in the contract using production standard equipment (ISRD) under agreed in-service conditions
Life Cycle Costing Forecasts of the whole life cost profile of equipment - including cost of design and development, production, in-service support and costs of, or receipts from, disposal
Maintainability Is the ease with which equipment can be inspected, overhauled, repaired and tested
Reliability Is the probability that equipment or components will work as and when required
Reliability Growth The enhancement of system/equipment reliability through the identification, Test analysis and correction of design failures and the verification of the effectiveness of the corrective action
Staff Requirement Defines in detail equipment function and performance in operational and environmental terms. It is prepared in the light of the feasibility study stage and forms a basis for project definition
SWIG Swing Fire Improved Guidance
Test, analyse and fvc The sequence of activities by which reliability shortfalls are identified; analysed and corrected through redesign or other modification to eliminate the cause of the shortfall; and finally validated warranty An agreement collateral to a main contract, the breach of which gives rise to a. claim for damages, but not a right to reject goods and treat the contract as repudiated
25 MINKTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AN” MAlNTAlNABlLlTY OF DEFENCE EOUIPMENT Appendix 1
Main conclusions and recommendations of the PAC’s 31st Report 1988-89 - Reliability and Maintainability of Defence Equipment - and Departmental Response in Treasury Minute dated 18 October 1989 (Cm831).
Conclusions and recommendations
PAC 2(i) The Services have estimated that unreliability adds over El billion a year to support costs. We accept that not all such costs can be saved but note that the Department consider fifty per cent to be a very fair goal for the level of achievable savings.
TM 166 MOD agrees that the current annual cost of unreliability estimated at over El billion is too high; believes that a saving of about half of the cost of unreliability is a reasonable goal for the years ahead, and plans a sustained programme, to bring about substantial improvements in the reliability of new and modified equipment, as it is developed and introduced into service.
PAC 2 (ii) We acknowledge the effort the Department are now putting into R&M but regret that, although R&M studies over the past ten years have produced common findings, the Department have been slow to react.
TM 167 MOD acknowledge that progress with the implementation of the findings of past studies has been disappointing. In general good reliability and maintainability (R&M) increases operational availability and reduces life-time costs. Nevertheless, in equipment procurement, project managers have to take into account other important criteria such as performance characteristics, financial allocations, and prospective in-Service dates. Whilst compromises have to be made, MOD accepts that, on occasions, R&M has not received sufficient priority. Thus MOD confirms that it is committed to improving R&M in future projects and reaffirms its policy that R&M is to be afforded full consideration along with other factors such as system performance cost and timescale.
PAC Z(i) We stress the importance of positive and effective action on the 1988 report by PA Consultants.
TM 168 MOD is taking positive and effective action on the 1988 report by PA Consultants, MOD has now appointed a Director of Reliability and issued preliminary advice to Contracts Officers on contracting for reliability. MOD is also drafting guidance to Project Managers.
26 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABlLlTY AND MAINTAINABILITY OF DEFENCE EQUIPMENT
PAC Z(b) We are concerned that action has not been taken in the past to “design in” and demonstrate R&M at the early stages of projects. As R&M is now designed in at the feasibility stage and fully demonstrated at each subsequent stage, we expect to see an improved level of reliability together with a reduction in reliability modifications in future procurements,
TM 169 MOD confirms that project managers are now required to show that R&M is “designed into” projects at the feasibility and development stages and subsequently demonstrated. Future procurement should show an improved level of reliability and a reduced need for reliability modifications.
PAC2 (v) In order to obtain some measure of assurance that R&M is being designed into these early stages we suggest that for all projects the Department should identify contractors’ planned level of R&M expenditure.
TM 170 MOD accepts the need to monitor contractors’ planned and actual efforts to achieve specified levels of R&M. MOD will make clear to the defence industry, through the Trade Associations, and discussions with individual contractors, the importance that it attaches to R&M. It plans to set clear reliability targets in development contracts as part of the overall incentive package, with an element of the payment tied to achievement of these targets. Where it is not possible to agree such arrangements with contractors, Project Managers will be required to pay particular attention to a contractor’s effort and expenditure on reliahillty as work progresses.
PAC2 (vi) We are surprised that at present there are only 30 R&M specialists in a Department that has a procurement budget of f9 billion. We note that the Department are committed to achieving an increase and look to them to make early progress.
TM 171 MOD accepts that more R&M specialists are needed but it also has to be recognised that the skills required are in short supply. To alleviate this situation steps are being taken to identify the additional numbers required, and to see how these could be provided through recruitment, internal training or the use of consultants. However the cost and time required to train staff in R&M techniques is appreciable, and the Procurement Executive generally already faces problems in recruiting and retaining staff.
PAC2 (vii) We welcome the decision to appoint a Director of Reliability to provide a focus for R&M matters within the Department. The tasks that he will perform are in our view of crucial importance and we will watch developments with considerable interest.
TM 172 MOD notes that the Committee welcomes the appointment of a Director of Reliability to provide a focus within the Department for R&M matters and that it intends to watch developments with considerable interest.
PAC 2 (viii) We welcome the decisions to strengthen the Committee for Defence Equipment Reliability and Maintainability (CODERM)arrangements and the future involvement of the new Director of Reliahillty in its activities, We are
27 M,N,STRY OF DEFENCE: DEVELOPMENTS IN THE KELlABlLITY AND MAlNTAINABlLlTY OF DEFENCE EQUIPMENT
particularly concerned that its effectiveness as a forum for developing attitudes on R&M matters should be improved and that there should in future be clearly defined accountability for ensuring that its decisions are implemented.
TM173 As the Committee acknowledges, MOD has taken action to make CODERM more effective as a forum for developing attitudes on R&M matters, and also to ensure that accountability for implementing its decisions is clearly defined. The Director of Reliability, who will be chairman of this Committee, will be responsible for implementing those decisions made by CODERM which do not naturally fall to any particular Controllerate or Service. R&M seminars organised by CODERM and news letters, videos etc have already attracted considerable interest both within the MOD and in Industry and are effectively emphasising the need for greater attention to be paid to R&M features.
PAC2(ix) We note with concern that, despite their crucial importance, Life Cycle Costing (LCC) systems do not always exist to enable the costs and benefits of improving R&M to be evaluated. We emphasise that this is a serious weakness and we expect the Department to take early and positive remedial action, When fully operational, we expect these systems to form a key part of the decision making process.
TM174 MOD continually seeks to improve its approach to LCC and its systems for financial appraisal. In particular the investment appraisal is now regarded as an integral part of the decision making process and all proposals with significant financial implications must be treated in this way. Investment appraisal will in future include the best available data on LCC. , PAC 2(x) We are concerned that present defect reporting systems are unable to provide adequate R&M data, and that the resources needed to remedy this are not available. We expect the Department to set clear objectives and stages for the I achievement of improvements. 1 1 TM175 MOD agrees that the current defect reporting systems do not always provide sufficient data for full analysis for R&M purposes. Indeed they were never i designed to do so. MOD will set clear objectives and stages for the achievement I of necessary improvements in these systems.
PAC2bi) We recognise the danger that on fixed price and competitive contracts some contractors may seek to minimise R&M programmes and resist divulging the results achieved. We stress the need for the Department to keep fully alert to 1 this possibility as procurement methods progressively involve greater competition and more fixed prices.
TM176 MOD does not accept that fixed price and competitive contracts necessarily lead to a reduction in the amount of effort that contractors will devote to R&M. Clearly drafted tender documents and contract specifications constitute an effective contractual requirement for R&M whatever the type of contract used.
28 MlNISTRYOFDEFENCE:DEVELOPMENTSINTHERELIABlLITY ANDMAINTAINABILITYOFDEFENCEEQUIPMENT
PAC 2(x$ We expect the Department to place greater emphasis on the analysis and weighting of R&M proposals at the tender assessment stage and to ensure that they obtain full visibility of contractors’ R&M activities in future.
TM 177 MOD is now placing greater emphasis on R&M aspects at the tender assessment stage and on obtaining full visibility of R&M activities and their costs.
PAC 2(xiii) We are concerned that the Department have permitted contractors to curtail or cancel planned R&M activities. We consider that they should place greater emphasis on full “test, analyse and Rx” programmes, followed by demonstrations before equipments enter service.
TM 178 MOD accepts the need for emphasis to be placed on full “test, analyse and Rx” programmes during development and, the need for reliability demonstrations before equipment enters Service. This is already being reflected, in appropriate cases, in contracts now being placed.
PAC Z(xiv) We are disappointed to learn that, despite their acknowledged importance in improving R&M, there are relatively few examples of contractual incentives for R&M. We expect the Department to take a stronger line in future on the use of incentives and warranties,
TM 179 MOD intends to take a stronger line on the use of contractual incentives and warranties for R&M and guidance has been issued ahout their future use.
PAC Z(xv) We are pleased to note that the Department have plans to achieve specific improvements in a number of key areas which we have listed at paragraph 33 of our Report: and that the Department’s concern for improved reliability and maintainability is shared by British contractors. We therefore look forward to seeing these developments take effect and wish to return to this subject in the future.
TM 180 MOD welcomes the Committee’s appreciation of the efforts being made by the Department and British contractors to improve R&M and notes the Committee’s intention to return to the subject in the future.
29 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAlNABILlTY OF DEFENCE EQUIPMENT Appendix 2
Case studies
The National Audit Offriceexamined 14 equipment projects at various stages of development to assess how R&M requirements were managed. The results of the examination of five of these projects are summarised in the following case studies.
1 Spearfish
2 Advanced Short Range An-to-Air Missile (ASRAAM)
3 Aimed Controlled Effect Anti-Tank Mine (ACEATM)
4 Small Arms Trainer
5 Phoenix
30 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELlABlLlTY AND MAINTAINABILITY OF DEFENCE EQUIPMENT Case Study 1
Spearfish
Cost: El,005 million (1993-94 prices)
Description: Heavyweight torpedo for Royal Navy submarines
Main Contractor: GE&Marconi
Background Spearfish started in the 1970s and its development phase began at the end of 1981. It therefore predates the recommendations of the Committee of Public Accounts. The Initial Production Order for the project is now scheduled for completion in 1995.
The National The reliability requirements stated in the contract were ambiguous and difficult Audit Office’s to measure. R&M payments were not linked to the achievement of specific Findings on R&M stages of the prqject, and in addition, Contract Acceptance Trials between 1987-89 showed very poor reliability.
However, after 1989 and the Committee’s recommendations, the Department took significant action to resolve these problems. The Royal Military College of Science, Shrivenham undertook a design review of the equipment’s R&M in 1990. Based on its findings, a Reliability Assurance Programme was drawn up with substantial help from the Directorate of Reliability. Under the Programme, a new and more satisfactory statement of the R&M requirements was agreed with the contractors, including a demonstration of reliability through in-water firings of 40 separate torpedoes, and linked to approximately f20 million of outstanding stage payments. For the next production order, in late 1994, the contract requires each torpedo to be reliability tested through an in-water firing and torpedoes will only be paid for in blocks of 20 after each block has demonstrated satisfactory levels of R&M. The contractors are also obliged to take corrective action at no extra cost to the Department if problems arise in-service.
Given the project’s history and contractual position prior to 1989, the Department have taken very successful recovery action to ensure good R&M performance.
31 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAlNTAlNABILlTY OF DEFENCE EQUIPMENT Case Study 2
Advanced Short Range Air-to-Air Missile (ASW)
Cost: E607 million (1993-94 prices)
Description: An air-to-air combat missile, using an infrared seeker as a guiding advice. The missile is highly manoeuvrable to ensure good short range effectiveness
Main Contractor: British Aerospace
Background ASRAAM was originally an international, collaborative project involving the United Kingdom, West Germany, Norway and Canada. Following the withdrawal of the other three nations, the United Kingdom decided to pursue the project as a national programme.
The requirement for the project was endorsed in 1984. However, a number of factors, including the international complications to which the project was subject, meant that the invitation to tender for the contract was not issued until 1991. A co&-act was signed with British Aerospace in 1992 and the current expected In-Service Date is 1998.
The National The levels of R&M established for ASRAAM by the requirements were the Audit Office’s highest for any United Kingdom missile system and will mean that the missile Findings on R&M will normally not require routine maintenance. Although the cost implications of these levels of R&M were not calculated, they were minimised by competitive tendering.
The contract provides for a good milestone plan. All contractual payments are linked to a milestonetarget, someof which relate to R&M. No paymentsare made until the pertinent milestone and all previous milestones are met.
There are contractual provisions for the contractor to achieve progressively increasing levels of demonstrated reliability during development and for testing of reliability during production and in-service.
A five per cent retention of development costs (about El2 million) is made at the end of the contract pending achievement of the pass criteria on the reliability demonstration or achievement of certain amounts of airtime in-service. Fifty per cent of the retention, however, will be released after one year, and 100 per cent after two years, regardless of the results of the reliahility
32 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELlABILlTY AND MAlNTAINABlLITY OF DEFENCE EQUIPMENT demonstration or airtime flown. The only incentive the contractor has to meet the reliability demonstration, therefore, is that he stands to receive the retention somewhat earlier than otherwise.
The contractor is, however, liable for any additional expenditure required to enable the missile to meet its specified R&M requirements - although such a provision does not give the Department as strong an assurance as the contractual right to reject the equipment if not satisfactory.
The Department have long heen aware of this problem and are currently seeking to amend the ASRAAM contract to tie retention payments more firmly to reliability achievement.
33 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT Case Study 3
Aimed Controlled Effect Anti-Tank Mine (ACEATM)
Cost: E32 million (1993-94 prices)
Description: A one-shot anti-tank mine, which uses acoustic sensors to recognise armoured vehicles, and i&a-red and laser detectors to pin-point their position
Main Contractor: ARGES (A French led consortium)
Background ACEATM is a collaborative project between France, the lead nation, the United Kingdom and Germany. The requirement was endorsed in 1990. The development phase is covered by a contract let under French law. A Development Memorandum of Understanding between the three collaborating nations empowers a French-staffed Executive Agency to carry out the day to day management. The development phase is scheduled for completion towards the end of 1994. A separate contract will be negotiated for the production phase of the project. The In-Service Date for the first batch of deliveries is 1999.
The National The R&M requirements in the development contract were hased on the project Audit Office’s technical specification which was agreed between the three collaborating Findings on R&M nations in August 1989. Under the occasional peculiarities of the collaborative system, the staff requirement was drafted and endorsed in 1990 after the technical specification had been agreed.
The invitation to tender required all tenders to include plans for the achievement of reliability and specified the R&M requirements that all tenders must meet. The contract specifies milestone payments some of which are related to reliability targets.
The contract does not provide for a “test, analyse and fix” programme to ensure that R&M is improved during development. The contractor is bound to carry out a certain number of trials demonstrating the reliability of critical components but there is no provision for a dedicated demonstration of the reliability of the system. The development contract provides that the contractor must demon&ate that he has met the R&M requirements, but it does not specify how. However, the contract does require a Fault Reporting and Corrective Action System to be applied and other reliability programme elements including modelling and prediction: analysis of the effect and criticality of failure modes; and parts selection.
34 MINlSTRYOFDEFENCE:DEVELOPMENTSINTHEREL,AB,LITY ANDMAINT4iNABlLlTYOFDEFENCEEQUIPMENT Case Study 4
Small Arms Trainer
Cost: E25 million (1993-94 prices)
Description: Computer-simulated fire arms training range on which new recruits receive initial instruction in small arms handling
Main Contractor: Ah-tech-SAV Ltd
Background The project was an Off-The-Shelf procurement of a mature product with some limited alterations to meet some of the Department’s particular requirements, principally relating to the use of the SA80 weapon. The Staff Requirement was approved in March 1991 and an invitation to tender was issued in October 1991 with tenders received in February 1992. The production contract was let in November 1992 and was compliance-tested three times in 1993, failing the first two times but passing the third. As the system’s basic design fixes its specifications, including R&M, the scope for changing the R&M of any Off-The-Shelf procurement is limited. In this case, the level of R&M offered and made subject to express terms was deemed to be acceptable.
The National No concept panel met: a concept panel is supposed to oversee the development Audit Office’s of a project’s R&M requirements during the feasibility and definition stages of a Findings on R&M project. However, the project’s R&M requirements were adequately defined.
Tender assessment procedures weighted the equipment’s performance at 60 per cent, but R&M at only 25 per cent: this is contrary to the Department’s policy that performance and R&M should be given equal weight. However, the Department had also sought to provide acceptable levels of R&M by including them as express terms in a maintenance contract together with an incentive scheme.
As this was an Off-The-Shelf procurement of a mature equipment, the R&M requirements were not comprehensively tested prior to acceptance of the equipment into service: only some qualitative testing of reliability and some limited testing of maintainability were carried out. However, the maintenance contract seeks to ensure that the required R&M of each system is met for a period of three years and to provide an accurate assessment of R&M based on a substantial quantity of reliable data after acceptance of the equipment.
There is an incentive scheme based on the availability of the system for training. However, the scheme, though working, has areas of weakness. For example, the National Audit Office considered that it is too highly graduated and
35 MIN,STRYOFDEFENCE:DE"ELOPMENTS,NTHERELlABlLlTY ANDMAlNTAINABlLlTYOFDEFENCEEQUIPMENT
the payments to which it relates are inadequate to its purpose. The current contract embodying the scheme covers each set of equipment for a period of three years.
A contractually-binding clause requires the contractor to remedy, at his own expense, any reliability defects, including design defects, that arise during the first three years of the equipment’s ten-year life. The Department’s rights to claim remedy of defects under the Sale of Goods Act and English Law would be good for the first six years of the equipment’s life.
A very sophisticated In-Service Reliability Plan and Reliability Programme Plan have been drawn up by the equipment supplier. Their purpose is to monitor the achievement and maintenance of contracted reliability levels.
36 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT Case Study 5
Phoenix
Total .i264 million (1993-94 prices)
A medium range remotely piloted air vehicle providing real-time thermal imagery to a ground station. Its principal role is target acquisition for the Multiple Launch Rocket System and Depth Fire artillery, with a secondary role of reconnaissance for intelligence purposes
Main Contractor: GEC-Marconi Avionics Ltd
Background The Staff Requirement dates from 1984 and therefore much of its management pre-dates the PAC Report. The contract was let in 1985 hut the project is now delayed by about 3.5 years until 1995 because of technical and contractual difficulties.
The National l The Phoenix R&M requirement is that a Troop of 18 Air Vehicles should have Audit Office’s a 90 per cent probability of completing a 16 hour Battlefield Day, during Findings on R&M which no more than one air vehicle would have to be in the air at any one time. In 1992, the Operational Requirements Staff suggested that higher reliability levels might be needed.
l The contract called for reliability to be assessed by theoretical predictions rather than in-service demonstration. These predictions were updated using real data from the project during the development programme. The Directorate of Reliability have stressed that reliability predictions should be treated with caution.
l There is no contractual provision for an in-service demonstration of the project’s reliability. Consequently, the Department have no contractual means of evaluating what the reliability of the equipment actually is and it will be difficult for the Department to impose liability for any failures to meet the reliability requirement on the contractor.
These problems arose some time before 1989 and, hence, before the Committee’s Report and the Department’s new policies on R&M. However, action since 1989 to put these problems right was only considered in 1992 when it was probably too late to renegotiate the contract or take other remedial measures. Remedial action should have been considered much earlier, perhaps by as much as three years. MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MALNTAlNABlLlTY OF DEFENCE EQUIPMENT Appendix 3
Surveys by NAO Consultants
(a) Survey of Defence Contractors The NAO’s consultants carried out a survey across a number of contractors in the Defence industry. The questionnaire sought to ascertain the contractors’ views on the Department’s approach to achieving R&M. Replies were received from nine contractors out of twelve approached.
The responses to each question were analysed by the consultants and, where appropriate, a general conclusion reached. These are shown ia the table below.
1. WereR&M targets/requirementsspecified in the contractand Targets/requirementsare alwaysspecified in the contractbut the weremy reaastfc? rrrajurilyof contractorafeel they are not realistic.Generally, requirementsare eithertoo high, impracticableor difficult to comprehend.
2. Did R&M targets/requirementsrelate to the total equipment With the exceptionof one contractor,the answerto this questionis including software? ‘yes’.However, three contractors expressed the concernthat, although includedin the requirement,the existenceof softwarehad no bearing on the level of reliabilityspecified. In otherwords softwarereliability is assumedto be 100 per cent.
3. Wasan operationalduty cycleand environmentdefined? The responsesto this questionare varied. Generally an operationalduty Werefailure definitionsdefined and agreed? cycle and environmentare defined (in one instance,late in the design cycle) but failure definitionsare often disputedand not agreeduntil the developmentphase or just beforetrials are carriedout. In two instances this had led to misunderstandingsbetween the Departmentand the contractor.
4. Whereappropriate, were the R&M requirementsand in-Service No.Any dataon the in-serviceachievements for existingsimilar achievementsfor existing similar equipmentavailable? Were the equipmentare very difficult to obtain. In the few caseswhere quantifiedachievements sufficiently detailed and credible? informationis providedit is either not credibleor not appropriate
5. WereR&M characteristicsfor the Departmentsupplied items Forcontractors where Departmentallysupplied items arean issue, definedand werethese compatible with overallsystems R&M characteristicsare, on the whole, not defined.Under these requirements? circumstances,contractors tend to excludesuch itemsfrom their R&M analysesand assessments.In the one instancewhere characteristicsare defined,they are definedagainst a differentset of conditionsand environments.
38 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABlLlTY AND MAINTAINABILITY OF DEFENCE EQUIPMENT
6. Whowasthe MODofficer responsiblefor R&M? A very variedresponse to this questiondepending on the natureof the (ProjectManager?) project.The importantconclusion is that all contractorshave a nominatedfocus for R&M.
7. Wereyou providedthe opportunityto discussand commenton Generally,‘no’. Furthermore,when questionsare raisedby the the R&M specificationsand understandMOD’s justification for contractoKMOD are not able to provideadequate justification. thesespecifications?
8. Did MODR&M projectofficers have adequate knowledge and No. However,a numberof contractorsfeltthe situation improvedas the backgroundin the appropriatedisciplines? projectprogressed due to specificor on-the-jobtraining.
9. Weresufficient R&M progressreviews held with the MODProject The majorityof contractorsare of the opinion that sufficient progress Office?Did the attendantMOD Project Officers agree to consider reviewsare held. However,three contractorsare concernedabout the suggestionsmade by the Contractoror to providefurther level of importanceattached to the R&M reviews.For example, R&M explanation/information? reviewsform a very minor pari of the generalproject review and thereforetend to be pushedto one side or not addressedat all.
10. Wasthe Department’smonitoring of R&M progresseffective? The consensusof opinion is ‘yes’ but with reservations.Either the methodof monitoring progressis not consideredappropriate or the level of interestassociated with monitoring R&M progresstends to declineas the projectenters the productionand in-servicephases. In one instance,interest in R&M progresswas sparkedonly when problemswere identified.
11. Did the DepartmentProject Officers comment on R&M work With the exceptionof one contractor,the Departmentdo offer submittedbythe Contractor?Were the commentsconstructive? commentson R&M submissions.However, these comments are invariablynot constructive.The majorityof contractorsfeel that the problemstems from the lackof experienceand expertiseof the MOD officersconcerned (see response to question8 above).
12. Werethe Departmenrsstatements that the R&M characteristics Generally,the answerto both questionsis ‘yes’. However,a significant of an equipmentshould be given equalimportance to performance numberof contractorshave experience of R&M importancebeing considerationsacted upon in reality,ie do the DepartmentProject downgradedby MODeiMer as a result of conflictsarising with other Managersplace sufficient importance on R&M? Is R&M performanceaspects or as a result of trade-offstudies themselves. consideredin trade-offstudies?
13. Overthe past 5 yearshave R&M contractualconditions become With the exceptionof one contractorthe answeris a very firm ‘yes’.The morestrict? remainingcontractor feels that althoughthe Department’sR&M requirementsare certainly more common, they are not necessarilymore strict.
14. Wereany financialpayments associated with R&M milestones? With the exceptionof one contractortheanswer to the first questionis Did theseprovide incentives for the achievementof R&M? ‘yes’. However,a numberof contractorsare experiencingmoves towardswhat they perceiveas penaltiesfor failing to achieve requirementsrather than incentivesfor satisfyingthem. In somecases the incentivesare considerednegligible and are thereforeignored.
39 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELlABlLITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT
15. Wereyou requiredto examinethe effectof R&M on Life CycleCost The responseto this questionis varied.In somecases the responseis and LogisticSupport Costs? ‘yes’ but not in much detail;in others‘no’ but a requestis madeto supply datato enablethe Departmentto examinethe effects;and in the remaindera simple‘no’.
16. Wasthe meansof R&M demonstrationspecified by MOD Generally,the responseto the first questionis ‘no’ for a numberof consideredadequate? What action was takenby the contractor reasons,for example: or MODProject Office ifthe R&M demonstrationshowed * failure definitionsare deficient/inappropriate; non-compliancewith the requirements? - timing of tests is inappropriate(eg systems have been in-service for 2 yearsbefore tests arecompleted); - specifiedtrials durationsare too short to be meaningful:and - MODfail to providepeople of requiredskill levels,expertise and knowledgeof equipment. A numberof contractorsare not yet in a position to commenton the actionsrequired should non-compliancebe demonstrated.However, two contractorsclaim that paymentswere withheld whilst the accept/rejectcriteria of the trial were re-examined.
17. Doesthe defectreporting and datacollection system for the Whenit was madeclear to the contractorsthat this questionrefers to equipmenthave the capabilityto accuratelyidenti?, R&M the Departmen%system, the responsewas a very definite‘no’. Datais problems? eithervery difficult to obtainor is of a very poor quality.One contractor hascompared the Department’sdata for particularincidents with their own datafor the sameincidents and found the Department’sdata to be totally lackingin respectto someincidents and deficientin detailfor others.
L18. Doyou receiveregular data on the in-serviceR&M characteristics Exceptwhere support contracts are in placeto supply R&M data,the of the equipment? answerto this questionis ‘no’.
19. HaveMOD succeeded in enforcingR&M requirements? Onecontractor gave a firm ‘yes’.Otner contractors speaK of the impact R&M requirementshave had on the status of R&M within their organisationwithout giving their impressionof how the requirements haveaffected their equipment.
40 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAINABILITY OF DEFENCE EQ”,PMENT
(b) Survey of procedures used by major equipment purchasers in industry The NAO’s consultants examined the procedures used by commercial organisations to ensure that equipment purchased is procured with adequate levels of R&M. Replies were received from five of the six organisations approached. The survey was limited to a small number of major equipment purchasers in industry who can be reasonably considered to he the commercial equivalents of the Department.
Notes All these organisations, except B, were included in a similar survey carried out as part of the NAO’s previous 1988-89 study; organisation B is a close equivalent to the original examined in 1989.
Although the sixth organisation did not respond to the survey it did confirm that its R&M approach had changed little over the past 5 years.
1. Doesyour organisationspecify Yes Yes Yes Yes Yes R&M requirementson suppliers?
2. What is the most important 1. Qualityof 1. Safety 1. Safety 1. Profiiability 1. Profitability reasonfor specifyingR&M Service requirements?(a) Safety,(b) 2. Profitability 2. Profitability 2. Safety 2. Qualityof Profitability,(c) Qualityof Service, Sewice (d) Other. 3. Safety 3. Qualityof Service
3. Doesyour organisationuse Yes Yes Yes Yes Yes in-serviceachievements for similar existing equipmentto set R&M requirements?If not, what other methodis usedto establishthe requirements(es legislation)?
4. Doesyour organisationprovide No No Yes- incentives No Yes incentivesfor the achievementof invariably the R&M requirements? provided.
5. Overthe pastfive years haveyour No No No No No organisation’scontractual conditions becomemore strict?
6. DoesR&M havean impacton the Yes,although not Yes Yes- R&M Yes- very Yes- significant. approvalor selectionof significant dueto providesa minor significant. prospectivesuppliers? limited numbers input to selection of suppliers. process.
41 MINISTRY OF DEFENCE: DEVELOPMENTS IN THE RELIABILITY AND MAINTAlNABlLITY OF DEFENCE EQUIPMENT
7. Doesyour organisationrequire an Yes Yes Yes No No R&M plan from suppliers?
8. Doesyour organisationmonitor Yes No Yes No Yes the progressof R&M during development?
9. What percentageof project Approximately5% Approximately Approximately1% Lessthan 1% Approximately1% expenditurewould your 10% organisationexpect to be devotedto R&M?
10. Doesyour organisationspecify Yes- mainly own Yes-own Yes- Yes-own Yes-own on suppliersprocedures for organisation. organisation’s organisation’s organisation’s, organisation’s, carrying out R&M activities? derived procedures.
11. Doesyour organisationrequire Yes- just starting No No No NO suppliersto examinethe effect to do so. of R&M on Life CycleCosts and LogisticSupport Costs?
12. Doesyour organisationspecify on Yes-varied. No Yes-varied, No Yes-varied suppliersa meansof R&M usually dependingon demonstration?If so, what is theoretical. equipment. the meansmost frequently specified?
42