Report by the Comptroller and Auditor General NATIONAL, AUDIT OFFICE

Ministry of Defence: Reliability and Maintainability of Defence Equipment

Ordered by the House of Commons to be printed 1 February 1989 Her Majesty’s Stationery Office, London E4.90 net 173 MINISTRY OF DEFENCE: RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUEMENT

This report has been prepared under Section 6 of the National Audit Act, 1983 for presentation to the House of Commons in accordance with Section 9 of the Act.

John Bourn Comptroller and Auditor General National Audit Office 1 February 1989

TheComptroller and Auditor Generalis the headof the NationalAudit Office employing some 900 staff. He, and the NAO, are totally independent of Government. He certifies the accounts of all Government departments and a wide range of other public sector bodies; and he has statutory authority to report to Parliament on the economy, efficiency and effectiveness with which departments and other bodies use their resources. MINISTRY OF DEFENCE: RELIABILm AND MAINTAINABILITY OF DEFENCE EQUPMENT

Contents

Pages Summary and conclusions 1

Part 1: Background and Scope of the NAO Study 8 Part 2: The Department’s R&M Policy and Initiatives 10 Part 3: The Department’s Arrangements for Implementing R&M Policy 15

Part 4: The Role and Performance of Contractors 19

Part 5: Contract Incentives and Warranties 23

Glossary of terms and abbreviations 26

Appendices

1. Major R&M studies and initiatives since 1978 28 2. Surveys by NAO consultants 30 MINISTRY OF DEFENCE: RELIABILm AND i&‘JiVTmABILITY OF DEFENCE EQUIPMENT

Summary and conclusions

1. The Ministry of Defence have for many years been concerned about the reliability and maintainability (R&M) of equipment purchased corn Defence contractors. They have estimated that unreliability is increasing support costs by over El billion a year. While all unreliability cannot be eliminated the Department accept that considerable improvements could be achieved which would yield significant savings. R&M also affects the availability of equipment; for example, during peacetime between a third and a half of the Royal Air Force’s fast jet fleet is not available for training because of scheduled maintenance, modification action, inspection, or unscheduled rectification. There is substantial room for improvement in all these areas. 2. The problem of defence equipment R&M is not unique to the United Kingdom; the United States in particular have recognised the need for improvements in the R&M of their defence equipment. In 1979 the Public Accounts Committee (PAC) commented that an improvement in R&M was potentially the most cost-effective measure directly open to the Department in the equipment field. 3: This Report records the results of an examination by the National Audit Office (NAO) of the effectiveness of the Department’s R&M management and the achievements of both the Department and industry since the PAC last examined this matter in 1978-79. The NAO’s fieldwork was completed before publication of the Defence Committee’s Fifth Report of 1987-88 (HC 431). However, this NAO Report takes account of the evidence on this subject given to the Defence Committee by the Department. I intend to provide PAC with further details to supplement this Report, on a confidential basis.

4. There is one general point to be made plain at the start. The Armed Services and British contractors take pride in the performance of their equipment; many equipments perform superbly well at home and overseas. The NAO found that there is now a marked interest and concern on the part of both the Department and British contractors to make further progress. In co-operation - and co-operation is the key - they are working together to improve the quantity and quality of the resources directed to reliability and maintainability and to provide the working procedures, tests and contractual arrangements that will be conducive to this end. This general statement conditions the following account of the issues examined and the NAO’s detailed findings which are summarised below. Findings and On whether the Department have demonstrated sufficient commitment to conclusions their policy on R&M and have taken adequate action to influence its achievement 5. The need for good R&M has long been recognised by the Department and they have undertaken numerous studies in an attempt to improve

1 MINISTRY OF DEFENCE: RELIABILITI AND MAINTAINABILITY OF DEFENCE EQUEMEiVT

their arrangements for achieving it (Appendix 1). In September 1987 the Procurement Executive Management Board endorsed the basis of a strategy for improving the reliability of equipment. This was followed, in March 1988, by a tri-service policy statement from the Department on the importance to be attached to the achievement of the proper balance between operational availability, performance, and whole life cost of defence equipment through the application of their R&M policy. This raises complex issues calling for difficult trade-off decisions against operational performance, scheduled in-Service dates and initial purchase cost. The NAO’s main findings and related conclusions on this issue were that: (a] The Department’s ultimate goal is the achievement of high operational effectiveness with low whole life costs. For this reason it has for many years been their policy that R&M be afforded equal priority to performance. However, both the NAO’s examination and the Department’s own studies have indicated that R&M has consistently been sacrificed to performance, initial purchase cost and time. In view of the major impact of R&M on operational availability and life cycle costs, the Department need to ensure that sufficient weight is attached to R&M considerations when designs and specifications are being prepared (paragraphs 2.2-2.4, 2.14(c), 2.18- 2.20 and 3.23). (b) It is of considerable importance that R&M is adequately considered when procurement decisions are made. Organisational steps are required to ensure that this has been done. The NAO therefore endorse recommendations made to the Procurement Executive Management Board to strengthen the role of the Directorate General of Defence Quality Assurance on R&M, in particular to provide reliability inputs to Equipment Policy Committee papers. The knowledge that these inputs are required should help to ensure that proper attention is given to reliability problems at the earlier stages of project development. The NAO consider, however, that if real improvements are to be achieved there must be a will to delay or halt projects where R&M has not been fully considered, or where evidence points to levels of R&M below those specified (paragraphs 2.5-2.6). (c) The Committee for Defence Equipment Reliability and Maintainability (CODERM), the Department’s focal point for R&M, has exerted little influence in securing the implementation of the Department’s policy because it has no executive status or direct access to funds; furthermore it has a rapid membership turnover. The two committees to whom CODERM are accountable have themselves no executive responsibility in the area and have offered CODERM little direction. The Department need to consider how the promotion of R&M policy can be made more effective (paragraphs 2.7-2.13). (d) The many MOD studies on R&M have for the most part found a lack of commitment in this area, ineffective management, inadequate resources, a need for better information and scope for tighter contracting, particularly in the specification of R&M requirements. However, action to eliminate these weaknesses has generally been limited. The Department need to take concerted action to remedy the shortcomings highlighted by their own studies (paragraphs 2.14-2.16). MIMSTRY OF DEFENCE: RELIABILITY AND MAINTAINABILITY OF DEFENCE EQWMENT

(e) The main underlying obstacles to progress have been the inability of the Department to quantify conclusively the costs and benefits of improving R&M; the fact that the financial consequences of poor R&M fall on the Services Support budget whilst the investment in R&M is met by the Procurement Executive equipment budget; the pressure from Operational Requirements and Financial staffs to meet cost and timescale constraints; the lack of sufficient top-level commitment by the Department and industry to the rigorous implementation of agreed R&M principles at project level; and practical difficulties in defining contract terms and enforcing them. Until these factors are addressed substantial improvements in R&M are unlikely (paragraph 2.17). (f) The NAO found that contractors on fixed price contracts can be reluctant to divulge the results of R&M programmes to the Department. There is also the danger that they may plan minimal R&M programmes in order to reduce costs. The Department should ensure that their requirements for information and demonstration of achievement are clearly spelt out in the specifications and that contractors’ R&M proposals are thoroughly analysed in the course of tender assessments (paragraphs 2.19-2.20).

(g) The United States’ Armed Services have each taken major practical initiatives to improve R&M. They have sought to address three key areas where progress in the United Kingdom has been most limited: accountability by contractors to project managers and by them to the Services; high-level involvement within the Department; and commitment of all parties, particularly industry. The Department should examine the scope for implementing positive measures of the kinds employed in the United States (paragraphs 2.21-2.24).

On whether the Department have established suitable organisational arrangements to ensure that their R&M policy is properly implemented 6. The Operational Requirements staffs specify equipment needs (including R&M) and project managers are required to meet them. Both groups are supported by advice from a number of Department branches. The NAO found that the arrangements for providing R&M advice were complex and involved a multiplicity of branches and personnel, and that the instructions and information upon which those responsible for R&M depend were not fully reliable. The NAO’s main findings and related conclusions on this issue were that: (a) Operational Requirements branches, who are responsible for representing Service interests during the procurement cycle, are not represented on CODERM and have not contributed directly to the formulation of R&M policy. The NAO view this as an important weakness in the present arrangements (paragraph 3.3). (b) R&M project committees are usually established for major projects. The NAO recognised their importance as the only forum which brings together the wide variety of Department and contractor R&M expertise but found it difficult to determine their overall effectiveness in advancing R&M (paragraph 3.5).

3 (c) Although there are a number of Department branches with specialist technical knowledge on R&M matters, project managers are not obliged to seek or accept their advice. The number of such specialist staff is very small in relation to the size of the procurement programme, their numbers are below the authorised complement and they lack a suitable R&M career structure. In view of their policy of giving equal priority to performance and R&M, the Department should ensure that project managers are provided with timely and authoritative advice on both aspects and that they act on it. The NAO therefore endorse the conclusion of the Deparment’s own consultants that a substantial strengthening of the R&M organisation and staffing is required (paragraphs 3.6-3.9). (d) The Research Establishments have in recent years largely withdrawn from R&M activities. The Department should re-examine the Establishments’ investment in R&M against other priorities w&in the Defence research programme (paragraph 3.10). (e) R&M management and engineering requirements have been defined in two Defence Standards prepared by CODERM but few major equipments which have been wholly subject to these requirements have yet entered service. However, these guidelines are not mandatory. The Department should ensure contractors’ R&M programmes include the full range of tasks necessary to achieve Service R&M requirements (paragraphs 3.11-3.14). (f) The Services’ defect reporting systems have been unable to provide adequate data on R&M and the substantial resources required to improve the situation are not currently available. Service users consulted by the NAO feel that they have too little influence on R&M matters. The NAO emphasise the importance of the Department having prompt and reliable data from all three Services on the availability of their equipment and the reasons for non-availability. Improved arrangements are also needed to take full account of users’ experience (paragraphs 3.15-3.17 and 3.22). (g) Life Cycle Costing models have so far only been used on relatively few equipments, partly because the required data are not always readily available. The Sea and Air Systems Controllerates have developed their own Life Cycle Costing systems while the Land Systems Controllerate is only now actively studying the subject. Since a lack of appreciation of life cycle costs may lead to ill-informed decisions, the Department should develop life-cycle data sources and ensure that submissions to the Equipment Policy Committee on all major projects include castings derived from Life Cycle Costing models (paragraphs 3.18-3.21). (h) In the light of achieved performance, the Department’s R&M requirements sometimes appear to have been over-optimistic or unsoundly based. None of the Defence contractors consulted by the NAO thought that they were currently provided with sufficient explanation of the R&M requirements set by the Department, or adequate feedback of R&M information on in-Service projects; some felt that requirements were not always realistic and lacked clarity and completeness. On the other hand the Department have always invited industry to question requirements if they consider them to be unrealistic or uneconomic (paragraphs 3.23 and 4.3).

4 On the extent to which R&M requirements are identified by de Department and specified in contracts, and the action taken by contractors to meet them

7. While the Department are responsible for the achievement of good R&M, the main direct influence is the performance of contractors. The NAO examined projects to assess the Department’s performance in setting requirements and negotiating enforceable contracts. The NAO also sought to establish how the Department ensured good performance by contractors at the various stages of procurement. The NAO found a number of weaknesses in procedures and, with one or two exceptions, a level of R&M achieved which was well below the Services’ original requirement. The NAO’s main findings and related conclusions on this issue were that: (a) Contractors are increasingly drawing up their own R&M specifications and the Department have only limited ability to monitor them. This increases the risk that specifications could be loosely drawn, thus enabling the contractor to avoid responsibility for achievement of the level of R&M required by the Services. Some contractors are over optimistic when assessing whether their designs will meet the Department’s R&M requirements. The Department need to ensure, for example, by making greater use of milestone payment schemes, that contractors’ R&M programmes meet the needs of the Services (paragraphs 4.44.6 and 4.11). (b) In a number of instances the Department gave R&M a relatively low priority during the early stages of a new equipment. Since it is at these stages that an equipment’s intrinsic R&M is principally determined, and as R&M can only be added later at significant extra cost, the Department should ensure that R&M is fully considered, with appropriate use of risk assessments, during design (paragraphs 4.8 and 4.10). (c) It was not usually possible to identify the contractor’s expenditure on R&M. The Department’s guidance should provide for R&M expenditure in major projects to be separately identified (paragraph 4.14). (d) In some cases reliability growth programmes were only partially implemented due to cost and time constraints. These “test, analyse and fix” programmes are essential if potential R&M design weaknesses are to be removed. The Department should therefore ensure that sufficient time and resources are made available (paragraphs 4.16- 4.17). (e) Ideally R&M levels should be confirmed at the end of development from dedicated R&M demonstrations but this is an expensive process and was not provided for in most of the projects examined. In industry full scale introduction of new technology normally occurs only when sufficient knowledge and experience has been gained from an equipment demonstrator programme. R&M demonstrations included in more general evaluation trials for the Department at the end of development have, on occasion, provided a useful alternative to dedicated R&M demonstrations and the Department should consider their wider use where dedicated R&M demonstrations are not practicable. While the Department should

5 MINISTRY OF DEFENCE: RBLIABILITI AND MAINTAINABILITY OF DEFENCE EQUIPMENT

avoid needlessly costly work, it is clearly important for the level of R&M achieved in development to be demonstrated before production commitments are made (paragraphs 4.184.21 and 5.12). (f) The Department required some form of R&M production testing in each of the projects examined by the NAO that had reached, or were about to enter, the production stage. However, this was not always fully reflected in the contractors’ planned activities. The Department should ensure that equipments meet the Department’s R&M requirements before contractual acceptance (paragraphs 4.224.23). (g) In those cases where in-Service R&M demonstrations have been carried out, they have proved useful. The Department might, therefore, consider making in-Service demonstrations a requirement in future contracts (paragraphs 4.244.26). (h) Contractors’ commitment to good R&M is mainly evident when the cost to the contractor of failing to meet the specified level of R&M warrants Board attention (paragraphs 4.28 and 5.6). On the use by the Department of contract incentives and warranties as a means of ensuring that R&M requirements are met 8. Contractual incentives and warranties can be an important measure in securing good R&M but establishing enforceable contracts can be difficult. The NAO’s main findings and related conclusions on this issue were that: (a) The Department have not always found it possible to pursue contractual remedies when contractors failed to achieve specified R&M requirements. This has been mainly due to inadequately expressed specifications but even express warranties have not always proved to be an effective remedy. However, the Department are now making greater use of legal advice to try to ensure that their contracts are enforceable and the NAO welcome this move (paragraph 8.3). (b) There have been few examples of contractual incentives and warranties for R&M on major projects. The Department’s consultants have identified two weaknesses preventing the use of such provisions: R&M specifications have not been detailed enough to be used effectively in contracts: and the measurement of achieved R&M was rarely rigorous enough to be used as a basis for contractual payments (paragraphs 5.4 and 5.8). (c) The Department’s consultants recently concluded that contractual provisions for R&M could be applied to most types of equipment and all stages of procurement, and that the benefits of an increase in effective R&M incentive and warranty provisions in contracts could, in terms of life cycle costs, be horn two to seven times the additional contract cost. These conclusions are confirmed by experience in the United States. In view of the considerable benefits attaching to the use of contractual incentives for R&M, the NAO urge the Department to extend their application as far as possible (paragraphs 5.9 and 5.14). General conclusions 9. Improvements in the reliability of Ministry of Defence equipment could result in very substantial savings in life cycle costs and improvements in value for money. The procedures to be adopted for achieving good R&M have long been widely recognised within the

6 MINISTRY OF DEFENCE: BELLtBILITY AND MAINTAINABILITY OF DEFBNCB EQUIPMENT

Department but their implementation has frequently been sacrificed to performance, time and initial purchase cost. The Department have recently made a high-level commitment to the achievement of good R&M but success will depend on: how the Department in practice face up to pressures horn other priorities and the Department’s willingness to forego short-term benefits; the amount and authority of R&M expertise that is made available; and how effective the Department are in negotiating with contractors tight and enforceable specifications. iWNISTRYOFDEFENCE:FJWABLITYANDMAINTAINABILITYOFDEFENCEEQ"PMENT

Part 1: Background and Scope of the NAO Study

Department estimated that the cost of the Jaguar aircraft failing to achieve its specified R&M requirement 1.1 The operational effectiveness of Defence amounted to Z496 million, which represented 13 per equipment depends not only on its technical cent of the aircraft’s whole life cost. In the United performance but also on reliability and States experience has also shown that a modest maintainability (R&M] where: investment in reliability of around 2 per cent of the cost of acquisition yields an investment return ratio better - reliability is the probability that equipment than 1:7. However, probably the greatest benefit of or components will work as and when required; improved R&M would be in the enhancement of the and Services’ operational capability through increased availability. During peacetime, between a third and a - maintainability is the ease with which half of the fxont-line fast jet fleet of the Royal Air Force equipment can be inspected, overhauled, is not available for imining because of scheduled repaired and tested. maintenance, modi6cation action, inspection or unscheduled rectification. These, along with the level of logistic support, are the two most important factors in determining 1.4 Difficulty in achieving the R&M requirements for availability. new equipment has also contributed to the cancellation of projects. Recant examples are the SP 70 self-propelled howitizer, where cancellation cost 1.2 The Defence Appropriation and Management Accounts do not separately identify the costs and the Department E88 million, and the Nimrod Mission benefits of R&M. However, special studies by the Air System Avionics where cancellation of the Nimrod Force Department and the Ministry of Defence AEW project cost E880 million. The Army’s armoured (Procurement Executive) suggest that unscheduled fighting vehicles have also had a long history of repair and maintenance stemming from unreliability unreliability, from the engine problems of Chieftain to currently costs the Royal Air Force some 6500 million the gearbox of Challenger. a year or 20 per cent of its annual operating budget. PAC Reports Based on this, and the known reliability of Army and Navy equipment, the Services estimate that 1.5 In their Sixth Report of 1978-79 the Committee unreliability adds over El billion a year to support of Public Accounts examined the contract incentive costs. These assessments cover the costs of Service arrangements for achieving higher standards of R&M maintenance staff, spares, repairs in industry and in the Hawk aircraft. The PAC noted that high resultingmodifications. The Department informed the reliability was potentially the most cost-effective NAO, however, that this figure should be treated with measure directly open to the Department in the caution in that it will never be possible to eliminate equipment field. The Department informed the unreliability completely. They acknowledged, Committee that they laid the main emphasis on trying however, that improvements in reliability could be to obtain, in the first place, the information from achieved at a price that would be justified given the which sensible R&M requirements could be derived. potential for savings in support costs. Thereafter, they tried to ensure that from the earliest stages of design the contractor directed sufficient 1.3 Ministry of Defence studies (Appendix 1) have effort and attention towards achieving those indicated that there would be a substantial requirements. The Committee recommended that the investment return on R&M expenditure. For example, Department should review their instructions to the Royal Air Force have estimated that in-Service project managers to ensure that they reflected best reliability modifications cost at least 70 per cent of known practice. The Treasury and the Department the total cost of acquisition and that, in certain cases, accepted the Committee’s recommendation and this cost could be reduced to as little as 7 per cent if confirmed that the applicability of incentive schemes provided for in the original design. Furthermore, in a to future projects would be assessed on a case-by-case recent review into the cost of aircraft reliability the basis.

8 Scope of the NAO examination maintenance activities were not covered. Also excluded were areas where special arrangements 1.6 Against this background the National Audit apply, ie ammunition and computer software. Office (NAO) examined four key issues:

(a) whether the Department have demonstrated 1.8 For the purpose of their examination the NAO sufficient commitment to their policy on R&M obtained views and information across a wide range and have taken adequate action to influence its of Department and Service areas including R&M achievement (Part 2); sponsors, procurers, users, maintenance and support (b) whether the Department have established authorities, research organisations and R&M specialist suitable organisational arrangements to ensure staff. The NAO staff were assisted in an advisory that their R&M policy is properly implemented capacity by R&M consultants, Rex Thompson and (Part 3); Partners, who also canvassed the views of the Defence industry and major equipment suppliers on (c) the extent to which R&M requirements are specific R&M issues (Appendix 2). identified by the Department and specified in contracts, and the action taken by contractors to meet them (Part 4); and 1.9 In consultation with the Department, the NAO selected a sample of major projects for detailed (d) the use by the Department of contract examination. The selection was spread across the incentives and warranties as a means of ensuring three Services and contained projects still under that R&M requirements are met (Part 5). development as well as completed projects. The sample was intended to provide examples of both 1.7 The NAO examination concentrated on the R&M good and bad R&M practice within the Department. procedures affecting the formulation and application Such projects have long timescales and those of R&M design characteristics prior to and during examined had not been affected by the more recent development. Other factors such as quality control Department initiatives such as those reported in during production and operating and paragraphs 2.5-2.6.

9 MINISTRY OF DEFENCE: RFU.%BILlTy AND MAINTAINABILlTy OF DEFENCE EQUPMENT

Part 2: The Department’s R&M Policy and Initiatives

2.1 In designing equipment for maximum cost- programme to keep the proposed R&M activities to a effectiveness the Department have to strike a balance minimum, or to exclude them altogether. between maximisiog technical performance, meeting the scheduled in-Service date, minimising life cycle 2.5 The Procurement Executive Management Board costs and maximising R&M. This part of the Report (PEMB) exercises general oversight of tbe examines whether the Department have demonstrated achievement of the Department’s policy objectives in sufficient commitment to their policy on R&M and this field. In September 1987 it considered a paper on have taken adequate action to influence its reliability and approved the basis of a strategy under achievement. It also considers the results of the many three headings: Department studies and reports on R&M over the last (a) Setting requirements - feasibility and ten years and reviews the American experience on project definition studies should in future R&M. consider the technically achievable levels of reliability and the risks associated with various R&M policy options; the volume of equipment and related support costs needed to meet the specified level 2.2 For many years it has been the policy within of availability; and the procurement options each of the Services that R&M should be afforded available. equal priority to performance. Moreover, in the case (b) Achieving reliability - the contractor of the Army the policy envisages that the should carry the risk and have an incentive to achievement of high standards of R&M should, if meet the requirement; to this end a range of necessary, be at the expense of operational contractual options should be available. performance and scheduled in-Service dates. (c) Monitoring reliability ~ each Service should establish a focus for in-Service reliability 2.3 In March 1988 tbe Department issued an monitoring; this should also aid the instruction, sponsored by the Committee for Defence development of Life Cycle Costing models and Equipment Reliability and Maintainability (CODERM), contractual demonstrations. which brought together the separate R&M policy statements of the three Services. It re-affirmed that 2.6 Other more specific recommendations within overall cost the achievement of availability considered by the PEMB were for the Directorate must be accorded equal consideration with General of Defence Quality Assurance to establish a performance. It recognised that a careful balancing of centre of expertise on reliability, perform reliability conflicting interests was required. It considered that and quality reviews, and make reliability inputs to the ultimate goal was the achievement of high EquipmentPolicy Committee papers. operational effectiveness with low whole life cost. The role and achievements of CODERM Specific directives were given to Operational Requirements staff and project managers for 2.7 The central committees responsible for quantifying and setting R&M requirements, examining furthering the Department’s R&M policy are shown in trade-off options and monitoring progress during Figure 1. Of these committees only CODERM is development. It is, however, too soon for their impact concerned solely with R&M matters. It was formed in to be assessed. 1976 to provide a focal point for the exchange of information and expertise between the various 2.4 The NAO noted that the policy of equal priority sections of the Department concerned with comes under significant pressure when a tight in- equipment availability and to assist in the Service date results in accelerated programmes or establishment of rationalised common policies, overlapping development and production. For procedures and methods. example, in the case of Blue Vixen, although recently increased provision has been made for R&M activity, 2.8 Throughout its early years CODERS exerted an early in-Service date and budgetary restraints have little influence on R&M. Several attempts were made, operated through much of the procurement in 1979 and again in 1981, to strengthen its role but

10 MINISTRY OF DEFENCE: RELIABILITY AND ivL%INTAINAB~~~ OF DEFENCE EQLIIPMENT

Figure 1

Service Departments MOD Procurement Executive MINISTRY OF DEFENCE: RHUiBILlTY AND MAINTAINABILITY OF DEFENCE EQUIP,‘v,.ENT these met with little success. Indeed, in canvassing 2.12 As regards the oversight of CODERM exercised support for change, the Air Force Department by these bodies the NAO noted that: concluded in 1981 that ignorance of CODERM’s existence was universal, and took this to be an (a) In 1983 the Defence Maintenance indication of its lack of effectiveness. Committee initiated a major study, by CODERM, into the reliability content of contracts which, in 2.9 In 1982 CODERM concluded that its structure turn, led to the PA Consultants study of was inadequate to support the wide ranging tasks contracting for reliability (paragraphs 2.15 and required of it and in the following year introduced a 5.6-5.9). However, the Principal Administrative revised organisation, incorporating three sub- Officers’ Committee expressed disappointment committees. It also formulated revised terms of in 1984 at CODERM’s lack of achievement reference for itself which were broadly to co-ordinate regarding R&M. In March 1986 the Defence all non-project R&M activities and to formulate R&M Maintenance Committee agreed to consider R&M policy, procedures, techniques and management more closely. However, the Defence practice for project managers to draw upon. Maintenance Committee subsequently approved revised terms of reference for itself in June 1986 that made no specific mention of R&M. 2.10 CODERM’s three sub-committees, concerned with management, training and techniques, undertake (b) The Defence Quality Assurance Board, for the detailed work. These sub-committees, as with its part, has had even less direct involvement CODERM itself, have had to delay important activities with R&M matters. In 1985 it reviewed a paper 1 due to shortage of manpower. In particular, the by the Director General of Defence Quality Techniques sub-committee’s review of the guidance Assurance on Defence-wide R&M activities. It to the Department and industry on R&M techniques is also endorsed the proposal by the Defence still incomplete after five years (paragraph 3.13). Maintenance Committee for a more detailed study of contracting for R&M (paragraph 5.7). 2.11 The NAO’s examination of the operation of CODERM and its sub-committees revealed several fundamental obstacles to it achieving its objectives: 2.13 The most important CODERM initiatives have been the establishment of t&Service R&M courses (a) CODERM has no executive status and and seminars to promote R&M at all levels of the therefore relies on its members to promote and Department and industry; and the production of two undertake work exCommittee. For this reason it R&M Defence Standards. The Management and has been unable to co-ordinate R&M activities, Training sub-committees have also played a major as evidenced by the large number of single part in publicising the cause of good R&M. Apart Service studies, differing working practices from this, however, CODERM has made generally within the Department, and the lack of guidance slow progress. on R&M data collection and analysis; (b) there has been a rapid turnover of R&M studies Connnittee members; the two key posts of Chairman and Secretary have been worst 2.14 Over the past ten years considerable attention affected in that both have had four incumbents has been paid to the need for an improvement in the since 1983; levels of achieved R&M and numerous studies have been carried out (Appendix 1). Generally, these (c) as is generally the case with Committees, studies have produced the same underlying findings CODERM does not have direct access to funds and recommendations, namely: and has to rely on money being allocated for CODERM-initiated tasks by the various branches (a] a lack of high-level awareness and represented on the Committee; and commitment to R&M, the remedy for which was sometimes seen to be a high-level focus with (d) since 1983 CODERM has been accountable executive powers over R&M on the model of the to both the Defence Maintenance Committee and US Services (paragraph 2.21); the Defence Quality Assurance Board. Neither body, however, has executive responsibility for (b) R&M requirements needed to be framed R&M and each has tended to react to rather than better; initiate events. Consequently their influence over the activities of CODERM in terms of securing (c) R&M was being sacrificed to considerations major change has been limited. of performance, initial purchase costs and time;

12 (d) the achievement of R&M needed to receive whereas the financial consequences of poor R&M greater priority and extra resources; fall on the Services Support budget: in addition Services personnel have to live with the (e) the Department’s management of R&M operational consequences of poor R&M; needed to be more effective; (f) better information was required on the (c) the pressure from Operational and Financial unreliability of equipment in-Service and the staffs to meet cost and timescale constraints; cost/benefit of reducing it; and (d) a general lack of sufficient top-level (g) more contractual pressure should be commitment within the Department and applied to achieve the required levels of R&M. industry to ensure that, compared with other objectives, R&M is given the priority which Life 2.15 The most recent report, by the Department’s Cycle Costing and investment appraisal can consultants in March 1988, stated that successful show is warranted; and R&M depends on a major development and reinforcement of the Department’s R&M policies and (e) practical difficulties in defining contract procedures. The consultants saw a need for a more terms and enforcing them. thorough project appraisal and risk assessment; more substantial early design and development stages; The competition initiative and fixed price detailed and comprehensive R&M specifications; and contracting careful monitoring of contractor R&M and in-Service performance. The report also commented on the need 2.18 In recent years the Department have made for changes in organisation in order to aid major efforts to increase the proportion of contracts implementation of R&M policies. They stated that awarded through competition and at fixed prices. CODERM was not an appropriate executive While these initiatives have undoubtedly brought mechanism; that a small central R&M policy unit was benefits, they have highlighted further the importance required; scarce R&M resources should be of fully incorporating R&M requirements in consolidated to make them more effective and better specifications and translating these into enforceable integrated with project teams; and more resources contracts (paragraph 5.3). were required. The consultants also reported that a senior post should be created to direct and co- ordinate the change process. These proposals are still 2.19 The NAO found evidence that contractors on under consideration. In particular consideration is fixed price contracts can be more reluctant than being given to the creation of a Director of Reliability contractors on cost plus contracts to divulge the post within the Directorate General of Defence results of R&M programmes to the Department. For Quality Assurance which would be a central focus for example, the project staff working on Spearfish R&M activity and brief the Chief of Defence and Vertical Launch Sea Wolf have experienced great Procurement on the reliability aspects of projects difficulty in obtaining information from contractors (paragraph 2.6). concerning their compliance with agreed programmes of work and the results that they are producing. 2.16 The NAO noted that practical action on the Requirements for information and demonstration of recommendations arising from these studies has achievement have not always been clearly spelt out in generally been limited. The effort required has been specifications. beyond the available resources, or the topic has been subsumed within a further study. However, the 2.20 There is also a danger that contractors bidding Systems Controllerates were aware of the need to for competitive contracts will plan minimal R&M improve R&M and have been taking action programmes in order to reduce price. The Department independently, in particular they have each formed a can counter this danger only by thorough analysis high ranking focus for R&M. and adequate weighting of R&M proposals in the course of tender assessments. The NAO note that the 2.17 In the view of the NAO there are five main Sea Systems Controllerate’s Surface Weapons Branch underlying obstacles to progress: has increased the weighting percentage given to R&M from 3 per cent to lo-15 per cent. However, the (a) the Department’s inability to quantify Department recognise that there is room for further conclusively the costs and benefits of improving developments, not least in increasing the amount of R&M; technical expertise available to the project offices for (b) investment in R&M is met by the the proper evaluation of contractors’ proposals and in Procurement Executive equipment budget, drawing up fully enforceable contracts.

13 MINISTRY OF DEFENCE: RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT

The United States approach Clear messages emerged from this study for various parties: 2.21 Regular visits have been made in recent years to the United States by senior Service staff and - Industry must take the lead more in Defence R&M experts to observe the measures taken establishing built-in R&M processes, or be to improve R&M. Generally the United States and the subject to closer scrutiny by the Department of United Kingdom have suffered from the same R&M Defense; and problems and the Department is monitoring the - approaches being adopted there. The difference lies Department of Defense advocacy of R&M and life cycle costs must be accompanied by in the approach adopted for their solution. The emphasis in the United Kingdom has been on pragmatic management action, including the use of incentives and warranties, otherwise their submitting broad papers and inviting agreement to “a way ahead”, whereas in the United States detailed commitment will be questioned. progmmmes, or action plans, have been developed with the emphasis on compliance with laid down 2.23 The United States Air Force introduced “R&M procedures and the achievement of practical results. 2000” early in 1985, which was also given support at In particular, the Americans have sought to address a very high level. An action plan was drawn up the three key areas where progress in this country has which identified six main objectives for the been limited: accountability, high-level involvement, improvement of R&M: clearer direction; better and the commitment of all parties, particularly organisation and training; more comprehensive industry. planning; fuller accountability and feedback; enhanced communication and motivation: and higher 2.22 In 1973 the United States Navy appointed an industrial commitment. Executive Director for Reliability, Maintainability and Quality Assurance in charge of a small high-level unit 2.24 The US Army have recently set up a Reliability to establish policy, monitor the performance of Special Study Group to examine how the issue of industry, and initiate training programmes. The reliability is handled in the procurement process, and philosophy that emerged was one of “designing for have introduced a “Reliability, Availability and reliability”. The RMQ‘I’ Project, published in 1986, Maintainability (RAM] Excellence Programme”. The studied the extent to which external factors forced aim of this programme is to deliver to industry the managers to focus on the short-term and thereby clear message that R&M and operating and support acted as an impediment to the achievement of R&M. costs are inseparable.

14 MINISTRY OF DEFENCE: RELIABILITY AND MADXTAINABILITY OF DEFENCE EQUIPMENT

Part 3: The Department’s Arrangements for Implementing R&M Policy

3.1 There is no single executive branch within the CODERM (paragraph 2.6) and as “customers” have Department dedicated to R&M. The Operational not contributed directly to the formulation of R&M Requirements staffs specify equipment needs policy. (including R&M], while project managers are responsible for meeting these needs. Both are supported in these activities by a number of branches (b) Project management within the Services and Procurement Executive. This part of the Report examines whether the Department 3.4 The project manager is responsible for the have established suitable organisational arrangements overall management of his project, including all to ensure that their R&M policy is properly technical, time and financial aspects. He is assisted implemented. It also examines the usefulness of the by a multi-disciplinary team, individual members of main instructions and information upon which the which are accountable to him for their work; but Department rely, namely Defence Standards, the some are not under his direct line management defect reporting system and Life Cycle Costing control. In this category are quality assurance project information. Finally it provides the views of the user support officers responsible for advising on all and the Defence industry on the Department’s aspects of the specification, management and approach to R&M. achievement for both quality and R&M. However, a proposal was made in September 1987 that the line The Deparbnent’s R&M organisation management responsibility for these staff should pass to the Systems Controllers, thereby placing them 3.2 The responsibility for ensuring that good R&M is under the direct control of the project manager. The achieved for a particular piece of equipment rests Director General of Defence Quality Assurance would primarily with the Operational Requirements retain a functional responsibility. The Department branches and project management. Specialist R&M decided in principle in December 1988 to implement support agencies are consulted as and when required the proposal on 1 April 1989. The detailed in an advisory capacity. organisation of the project support personnel is being worked out. Project managers sometimes appoint a (a] Operational Requirements branches technical officer on their staff to act as the project R&M focus. The NAO noted that the R&M training 3.3 Operational Requirements branches are and experience of these officers varied but most had primarily responsible for the preparation of Staff attended or were scheduled to attend the CODERM- Targets and Staff Requirements which set out the sponsored five day R&M course. required operational characteristics, in-Service dates and cost criteria. Departmental guidance emphasises that the formulation of the requirements and the 3.5 Project managers normally establish an R&M planning of in-Service dates should take account of project committee or working group to monitor the R&M. As sponsors of equipment for the Services, the progress of the R&M programme. Ideally such bodies Operational Requirements branches represent the should represent all sources of R&M input including Services’ interests with project management during the contractor. The NAO found it difficult to the procurement cycle. Project managers are required determine the overall effectiveness of these R&M to keep sponsors informed of a project’s R&M status project committees since practices within and and obtain their agreement before any decisions are between the Controllerates varied widely and fixed taken which may affect R&M requirements. price/incentive contracts reduced the visibility of Operational Requirements staff are mainly Service progress offered by contractors. To some extent the personnel and rely on R&M cells within the Services committees’ effectiveness can only be measured on a for specialist R&M knowledge (paragraph 3.7). The case-by-case basis. Nevertheless, these project NAO noted that Operational Requirements branches committees are the only forum which brings together have not taken up the invitation for representation on the wide variety of R&M expertise available to project

15 MINISTRY OF DEFENCE: RELIABILITY AND MAlNTAlNABILlTY OF DEFENCE EQIJE’MENT managers and are therefore an important part of the CODERM activities and their co-ordinating R&M system for generating good R&M. Panel was dissolved some years ago.

(c) Specialist R&M support agencies Standards and management information systems

3.6 Operational Requirements staff and project (a) Defence Standards managers, if they wish, may call upon the advice of technical staff who are specialists in R&M matters. 3.11 During design, development and production The main body of specialists are within the R&M cells there are a range of important R&M management and of the Services engineering support directorates and engineering requirements that help to determine the the Procurement Executive Systems Controllerates. levels of R&M attained. These requirements are Other specialists are with the Central Servicing defined in various Departmental guidelines but Development Establishment [Reliability Squadron) of notably in Defence Standards 00-40 and O&41 the Royal Air Force; the Research and Development prepared by CODERM. There are numerous other Establishments; and the CODERM Secretariat. Departmental instructions covering the detail of R&M. The Sea Systems Controllerate, in particular, has 3.7 The NAO noted that the total number of staff, adopted this approach as a means of assisting project about 30, wholly engaged on R&M activities within management with R&M procedural, engineering and these agencies is small in relation to the total modelling aids. procurement budget. Numbers are below the authorised complement; this situation is aggravated 3.12 Defence Standard 0040 was first issued in by the difficulty of recruiting and retaining qualified 1981 and m-issued in 1987. It sets out the R&M and experienced staff, and by the lack of a suitable management responsibilities of Operational R&M career sticture. The Department’s consultants Requirements staff, project managers and contractors. have concluded that a substantial strengthening of the It is not mandatory for the project manager to specify R&M organisation and resources would be required to the Defence Standard in contracts; project managers implement the more rigorous R&M policies and are able to select specific R&M tasks and tailor them procedures needed as an essential pre-condition for to meet the needs of a particular project. These tasks contracting for R&M (paragraph 2.151. are then incorporated in the Statement of Work for 3.8 More general R&M engineering advice, inclusion in the invitation to tender or contract and, particularly on maintainability, is provided by the as such, become mandatory tasks for the conimctor. Central Servicing Development Establishment and similar, but less-centralised, branches of the Army 3.13 Defence Standard O&41 contains guidance on and Navy. They also provide advice on the R&M the technical practices and procedures for reliability aspects of equipment design to contractors. For and was first issued in 1983. The section on Rapier a joint Army/Royal Air Force R&M Assessment maintainability has still to be completed by the Team was formed some years ago. CODERM Techniques sub-committee (paragraph 2.10). 3.9 The NAO noted that the specialist agencies act in an advisory capacity only and are sometimes 3.14 Defence Standards, or their equivalent, had distanced from Operational Requirements staff and been applied to all the development projects in the project managers. Moreover, there is some evidence NAO sample. However, because of tbe length of that the latter do not make full use of the specialist project timescales, few major equipments which have support available. For example, the Air Systems been wholly subject to these requirements have yet Controller& cell was not consulted about the anti- entered Service. submarine warfare variant of the EH 101 helicopter where the R&M content of the development (b) Defect reporting system programme gave cause for concern. 3.15 The availability of sound defect data (including 3.10 In the case of the Research Establishments the total usage and full details of the operational NAO observed that there has been a considerable environment) is important for the achievement of reduction in R&M specialists in recent years. Only the good R&M. It permits requirements to be more Royal Aircraft Establishment (Farnborough) and the analytically and realistically formulated, subsequent Royal Armaments Research and Development designs to be improved, and contractual achievement Establishment (Chertsey) still retain small R&M established. Each of the Services supports a data sections to carry out research and only the latter now centre for the collection, collation and analysis of provides any direct assistance to projects. The maintenance data which includes defect reports of Research Establishments have largely withdrawn from non-scheduled maintenance. The results are fed, on

16 MINISTRYOFDEFENCE:RELIABlLKYANDMAINTAlNABILITYOPDEFENCEEQUIPMENT

request, to equipment and project managers, @] enables decisions on R&M to be better contractors and the Directorate General of Defence informed, in particular the determination of Quality Assurance for any investigation and remedial initial R&M requirements; action required. (c) allows the cost impact of different levels of R&M to be measured in order to facilitate trade- 3.16 The inability of the defect reporting systems to off decisions with other priorities; and provide adequate R&M data, and the adverse effect (d) enables R&M programmes of work in this has on the achievement of good R&M, has been a competing tender proposals to be evaluated. recurrent theme of Department reports (paragraph 2.14). The NAO noted that the usefulness of the The Chief of Defence Procurement has recently systems remains limited by their inability to provide emphasised the importance of Life Cycle Costing to complete, consistent and reliable data. In particular Defence purchasing and its role in securing better the NAO noted that: value for money. The best way to follow up this initiative is now under consideration. (a) defect reporting is not always mandatory: 3.19 Life Cycle Costing is not new to the (b) not all equipments are covered by the Department. Within the Sea Systems Controllerate reporting systems; project managers have used a range of computer models to help with procurement and management (c) there is a lack of information on the length decisions. One of the models under development is to of time equipment is in use; be used to help evaluate whole ship design options. (d) reporting of defects is generally on a The Air Systems Controllerate has also developed manual basis and therefore expensive and prone Life Cycle Costing techniques and models and their to error and omission; Life Cycle Costing and R&M specialists form a single unit within the Controllerate. However, the NAO (e) misuse of equipment by Service personnel noted that progress in the Land Systems Controllerate is a cause of failure but cannot easily be has been slower. Only recently has the Land Systems distinguished without special investigation; and Controllerate provided a Life Cycle Costing Focus: funding for the necessary information technology (fJ standards of reporting vary within and systems is being sought. between the Services. 3.20 The NAO also noted that:

3.17 Efforts were being made to improve this (a) there has been no central direction on Life situation with the introduction of new management Cycle Costing, the Sea and Air Systems information systems from which R&M data can be Controllarates developed their own systems extracted. Improvements were also being made in the separately; collection process through the use of modern (b) the Life Cycle Costing Working Party of technology, eg hand-held computers to record data CODERM’s Techniques sub-committee has only and automatic devices for recording equipment usage. recently been re-convened and become active in However, the substantial rascmrcas required for promoting new initiatives; effective data collection systems apa not currently (c) data on in-Service costs of equipment is not available. centralised in a readily available and collated form; and Life Cycle Costing (d) the range of equipments covered by Life Cycle Costing models is limited as is the 3.18 The United Kingdom and the United States capability of some of the models; this experience suggests that support costs are two to undermines the requirement that Life Cycle three times as much as initial purchase costs. Costing data should be incorporated in Consequently, the Department need to be aware of submissions to the Equipment Policy Committee. the whole life cost of equipment before making major In the case of Rapier FSC the lack of such a procurement decisions. This is done through the use modal meant that only broad estimates were of Life Cycle Costing models whereby the cost profile provided to the Committee. of equipment can be built up and refined as cost data becomes available. The use of Life Cycle Costing 3.21 In 1986 the Department’s Internal Audit drew techniques: attention to the inadequacies of the existing arrangements. They concluded that there was (a] provides the justification for “spend to considerable scope for greater use of Life Cycle save” decisions involving expenditure on R&M; Costing by the Department.

17 MINISTRY OF DEFENCE: REL!.MXLITY AND MAINTAINABlLlTY OF DEFENCE EQUIPMENT

User views on Defence equipment R&M that they were currently provided with sufficient explanation of the R&M requirements set by the 3.22 The NAO visited British Forces Germany to Department; some felt that requirements were not ascertain the views of equipment users. Similar always realistic and lacked clarity and completeness. arrangements were made with the Navy in the United The Department informed the NAO, however, that Kingdom. Many of the Service units reported contractors are always invited to question problems with reliability, affecting both low and high requirements where they consider them to be technology equipments. The NAO were informed that unrealistic or uneconomic. A number of contractors the Army and Royal Air Force users have had many also felt that the implications of R&M specifications difficulties with mechanical equipments and components, and that the reliability of support were not fully understood by Department project equipments has been disappointing. Some built-in officers. Although improvements were noted in the test equipments have repeatedly indicated failures importance the Department attach to R&M, in the which, when investigated, have been classified as “no view of contractors there was still little evidence of fault found”. The Royal Air Force users stressed the R&M having equal status with performance, initial need for a whole systems approach to reliability - purchase cost and time. Contractors commented while individual parts of electronic systems might be favourably on the knowledge and background of fairly reliable, the cumulative effect of minor Department R&M specialists but noted that they were problems can result in low system reliability. In the not involved in all projects; tended to have heavy Navy and the Army the users, particularly work loads; and were not always timely in their engineering staffs, felt they had too little influence on response to contractors’ submissions. They R&M matters. All the Service users consulted considered the monitoring of R&M to be reasonably considered that R&M data collection arrangements effective but pointed out that not all projects were were inadequate, and that there was a need for subject to regular R&M progress reviews. Contractors enhanced ADP support, costing information and were emphatic that strict R&M contractual conditions built-in equipment data recorders. would lead to greater emphasis being placed on R&M Defence industry views on the Department’s in development but scana suggested the need for approach to R&M contractual incentives, not just penalties for non- achievement. None of the contractors felt they 3.23 None of the Defence contractors contacted received adequate feedback of R&M information on by the NAO consultants (Appendix 2) thought in-Service projects.

, MINISTRY OF DEmNCE: RELIABILITY AND MAINT~ABIL”‘Y OF DEFENCE EQUIPMENT

Part 4: The Role and Performance of Contractors

4.1 While the Department are ultimately responsible upon him. The NAO’s project examination revealed for the achievement of good R&M, the main direct that in the case of Blue Vixen, the contractor’s influence on this achievement is the performance of original R&M proposals did not contain provision for the contractor. The Department therefore need to reliability demonstration (paragraph 4.15) or specify clearly a progmmme of R&M work and production reliability acceptance testing (paragraph enswe that the contractor implements it. This part of 4.22) although they were included in the Staff the Report examines the extent to which R&M Requirement. The contractor’s R&M plans were requirements are identified by the Department and condemned by the Department’s R&M specialists as specified in contracts and the action taken by being totally inadequate and indicative of a lack of contractors to meet them. Particular regard was paid commitment to demonstrating reliability in to the projects examined in detail by the NAO, development and production. The plans were and the Department’s monitoring arrangements. subsequently improved but remained inadequate. Contract incentives and warranties are considered as Additional funding has now been provided, some a separate topic in Part 5. years after development work commenced, to permit an increased level of R&M activity. Formulating R&M requirements 4.5 The NAO examination revealed that it is 4.2 To act as a basis for the attainment of optimum increasingly the practice for the responsibility for R&M, the R&M requirements must be realistically drafting specifications for development work, attainable and contibute to the most cost-effective including R&M, to be passed to the contractor. This balance of whole life costs, time into Service and introduces risks since it is not always in the best performance. The Department therefore need reliable interests of the contractor to prepare a specification information on the range of feasible levels of R&M that defines his obligations clearly and which renders and the implications of different R&M requirements his achievement easily measurable. The risk is for various project timescales, costs and performance. increased as a result of the limited specialist R&M This knowledge comes mainly from an understanding resources which the Department have with which to of the achievements, problems and scope for review contractors’ specifications. improvement of existing equipment currently performing a similar role. Trade-off studies can then be done and the preferred option selected. The NAO 4.6 In many projects the completion of planned acknowledge that this is not an easy task where there R&M work now forms part of a milestone payment are considerable differences between successive types scheme. The development programme for Rapier FSC of defence equipment. provides a good example of milestones being used in this way. The taut contractual terms of this project, 4.3 The NAO’s project examination revealed several which were examined by the Treasury Solicitor, examples of what appeared to be over-optimistic or include payments during development against unsoundly based R&M requirements by the achievement of 12 major milestones. Payment against Department or unwise acceptance by the contractor of the third milestone was withheld until the contractor requirements which could not be reached. For had achieved the milestone requirement. If the final example, over-optimistic requirements were set for milestone, which includes demonstration of the early Rapier systems, and these were originally reliability, is delayed by more than three months, the maintained for the subsequent Rapier Field Standard Department will be entitled to recover from the C version. contractor 15 per cent of all milestone payments made in respect of the ground equipment pending Translation of re uirements into completion of the final milestone. Contract milestones contractually bin %*mg specifications which refer to achievements [as in the case of Rapier) rather than merely to the completion of programmes 4.4 To be effective R&M requirements need to be of work or the passage of time offer the prospect of clearly communicated to the contractor and binding ensuring that specified R&M requirements are met.

19 Contractors’ R&M programmes that the contractor still has a great deal of work to do if he is to meet his design predictions and that 4.7 Defence Standard 00-40 (paragraphs 3.11-3.14) achieved reliability may be well below the contracted requires contractors to draw up programmes of R&M level. work covering each stage of the procurement cycle. The R&M work should be an integral part of the contractor’s project programme and the contractor’s 4.12 R&M predictions and assessments by performance of R&M work should be monitored by contractors are useful only in so far as they are the project management team. Best results are usually derived from a basis acceptable to the Department. In obtained when R&M programmes have contractual the case of Vertical Launch Sea Wolf the status. Department’s project office was unaware of the basis of the prediction.

4.8 An important mechanism for monitoring contractors’ progress against R&M programmes is the 4.13 Built-in test equipment is being increasingly Design Review. It is for contractors to schedule such used to locate defects and thereby increase equipment reviews and for the Department to be aware of the availability. Sonar 2016 benefits from this feature and results and actions arising. It seemed to the NAO that it has made a significant contribution towards the value of these reviews would be enhanced by the meeting the maintainability requirement. extended use of quantitative risk assessments. Such assessments should address the risk that the Services’ (b) Contractors’ R&M activities during development required levels of R&M might not be achieved. 4.14 The aim of R&M tasks during the development 4.9 In the following paragraphs the elements of a stage is to identify potential weaknesses in the design contractor’s R&M programme are considered under and ensure that necessary corrective action is taken. the main stages of the procurement cycle, namely Progress is monitored against the contractor’s design (paragraphs 4.10-1.13), development Development Cost Plan but it is not normally possible (paragraphs 4.14-4.21) and production (paragraphs to identify the contractor’s expenditure on R&M. 4.22-4.23). However, the NAO noted that in the case of Rapier FSC some R&M expenditure had been separately (a) Contractors’ R&M activities during design identified in the Development Cost Plan, although this only amounted to 0.2 per cent of development 4.10 The key stages for R&M are the Feasibility cost. Study and Project Definition because if R&M is not “designed in” at these stages, it can be added later 4.15 The main activities of the contractor during only with diiculty and at higher cost. In the cases development should cover reliability growth they examined the NAO found that the Department [paragraphs 4.16-4.17) and demonstrations of R&M generally gave relatively low priority to R&M during (paragraphs 4.18-4.21). the early design stages. R&M was not addressed in the Feasibility Studies for Trigat, Blue Vixen and Spearfish,and received only limited attentionin the 4.16 Reliability growth programmes should provide case of Rapier FSC. The Project Definition stage was for: extensive practical testing in simulated omitted completely for Spearfish and Vertical Launch environmental conditions; the analysis of any failures Sea Wolf. produced; and, where weaknesses are discovered, the introduction of design changes (“test, analyse and fix”). This is an expensive and time-consuming 4.11 Some contractors tend to be cwer optimistic business as test facilities have to be sat up, prototype when assessing whether their designs will meet the equipments dedicated for the purpose and many Department’s R&M requirements. The NAO hours of testing undertaken. Accordingly, examination found that, whilst the contractor’s performance of the “test, analyse and fix” sequence in assessments of reliability for Sonar 2016 appeared to fall often conflicts with the need to meet in-Service be reasonable, there were several assessments for dates and minimise initial purchase cost. other equipments which had since proved to be unrealistic. For example, the level of in-Service reliability for Sky Flash was only one-sixth of the 4.17 Most of the R&M programmes examined by the figure which two years previously the Contractor NAO did make smne provision for reliability growth assessed he would be able to demonstrate. In the work but the Department have often permitted cases of Rapier and Spearfish present indications are contractors to exclude, curtail or cancel planned R&M

20 activities at the development stage. The NAO also can be induced and the subsequent repair times found that more effort was expended on the lass time recorded to arrive at a mean active repair time. consuming and less costly, but largely theoretical, However, of the projects examined by the NAO, only tasks such as prediction and assessment rather than Hawk, Sea Eagle, Warrior and Sky Flash incorporated practical work. Deficiencies in the extent and timing full maintainability demonstrations at the of reliability growth work proposed and carried out development stage. For Sky Flash the achievement have affected the Blue Vixen, Sky Flash, and Vertical demonstrated could not be measured against a Launch Sea Wolf projects. quantified requirement as none had been specified.

4.18 Before entering the production stage the (c) Contractors’ R&M activities during production Department’s project management should have assurance that the final design emerging at the end of 4.22 At the production stage the R&M objective is to the development is capable of satisfying the Services’ ensure that the levels of R&M demonstrated at the R&M requirements. This assurance can only be fully end of development are maintained during obtained from dedicated practical R&M production. The main techniques used here are demonstrations. These should indicate the R&M environmental stress screening (or “burn in”) and achieved by equipment of the latest build standard in production reliability acceptance testing. conditions that approximate to Service use. Such demonstrations are more likely to make a significant 4.23 All the projects examined by the NAO which contribution to the achievement of required R&M had reached, or were about to enter, the production when there is a contractual requirement for a stage included a requirement for scane form of testing. successful demonstration. However, this was not always fully reflected in the contractors’ planned activities: 4.19 Reliability is related directly to the probability (a) A continuous programme of testing was of failure ocurring. To be statistically sound a adopted on Sonar 2016 and Sea Eagle. reliability demonstration needs to cover an extensive period of testing on a representative number of (b) In the case of Blue Vixen “burn-in” and equipments. This is an expensive and time- production reliability acceptance testing were consuming process. Other obstacles include the excluded. Following considerable ccmcern about difficulty of reaching an agreed definition of failure. Blue Vixen’s reliability, the Department now For these reasons dedicated reliability demonstrations propose that burn-in and design reliability are rarely included in contractors’ R&M plans, and demonstration be conducted by the contractor at where they are, they have tended to be foregone or the additional cost of smne E3 million. They told curtailed by the Department. However, an extensive the NAO that the requirements were not programme of trialling during the design and included in the original version of the development of Warrior has resulted in an equipment contractor’s R&M plan because of budgetary which, despite increased complexity, is expected to constraints and in order not to jeopardise the be more reliable and relatively easier to maintain in-Service date and contractual negotiations. compared with its predecessor. Deliveries of Blue Vixen were expected to cmnmence in June 1989, three months before the results of the reliability growth programme 4.20 Although formal R&M demonstrations may be would have been known. However, both the rare, the level of R&M likely to be achieved can be development and production programmes have assessed as part of more general design evaluation since slipped. Even so, the results of the growth trials conducted at the and of development. The NAO programme, expected in October 1990, will found this to be the approach more commonly only become known after development is adopted, often to good effect. However, reliability is complete and four months before deliveries are not always properly assessed during evaluation trials. due to commence. The results of the programme For example, in the case of Spearfish the contractor will therefore come too late to influence made no attempt to quantify achieved reliability and development. Nevertheless aircraft integration the Department had to employ consultants for this continues until 1992 and the possibility exists of purpose at extra cost. The first full assessment of Sky retrospective modifications at extra cost. Flash reliability did not take place until after production had commenced. R&M achievement in-Service

4.21 Compared with reliability, maintainability is 4.24 Once introduced into Service the levels of R&M less difficult to demonstrate. A selection of defects displayed by equipment need to be monitored to

21 MINISTRY OF DEFENCE: RELLtBILlTY AND MAINTAINABKlTY OF DEFENCE EQUIPMENT assess whether the Services’ requirements are being funded separately by the project office. The results of met: to inform the various acceptance processes; and the Sonar 2016 trial were considered to be useful to determine whether any modifications might be because they pointed to several design weaknesses required. Deficiencies in the routine monitoring of which were subsequently modified; they also R&M through the collection and analysis of defect provided the assurances necessary for contractual and data have been considered at paragraphs 3.15 to 3.17. fleet weapon acceptance. The Warrior trials were in progress in late 1988.

4.25 Where in-Service demonstrations have taken Contractors’ R&M capabilities place they have generally served a useful purpose. Separate in-Service R&M demonstrations were an 4.27 Contractors’ R&M capabilities vary with the important feature of the Hawk project. In-Service size of the contractor. Small companies may be experience confirms that the high availability of the unable to employ specialist R&M engineers, whilst Hawk is due to the fact that it is easy to maintain and large companies may have R&M departments. has reasonable reliability. As a general rule, however, Contractors’ R&M resources are sometimes stretched full in-Service R&M demonstrations have been few very thin; where this happens R&M specialists can be and far between. As a result Service Equipment largely occupied in responding urgently to major Branches and the Procurement Executive only have a problems rather than making more measured general “feel” for the levels of R&M being achieved. contributions at earlier stages. However, the Department told the NAO that in- Service demonstrations will form an integral part of 4.28 It is important that contractors should not only future major air systems projects, ie Tucano, EH 101, have sufficient qualified specialists (ideally with EFA and Tornado MLU. design experience) but also that contractors’ top management should be committed to the achievement 4.26 The only other projects examined by the NAO of good R&M. However, such commitment was often which incorporated an in-Service demonstration were found only when the cost to the contractor of failing Sonar 2016 and Warrior. These were not to meet the specified levels of R&M was sufficient to requirements of the contracts but were initiated and merit Board attention (paragraph 5.6).

..,

.-22 MINISTRY OF DEFENCE: RELIABILITY AND MAINTAINABWTY OF DEFENCE EQLUI’MENT

Part 5: Contract Incentives and Warranties

5.1 In 1979 the Department told the PAC that they 5.4 There have been few examples of contractual saw contract incentive arrangements as one of the incentives for R&M. The most notable example is still measures that, in certain cases, could be taken to that of the Hawk fast jet trainer project which was the secure improved R&M. This Part of the Report subject of a Public Accounts Committee Report in considers the use made by the Department of contract 1976-79. Recently, however, the Department have incentives and warranties for R&M as a means of been showing a greater interest in incentives as a ensuring that R&M requirements are met. means of improving R&M performance by contractors. For example, the Tucano training aircraft contract provides for 5 per cent of the contract price to be Use of incentives and warranties by the withheld pending satisfactory demonstration of the Department R&M requirement.

5.2 The Department have a contractual right to 5.5 The Department’s use of warranties has been equipment of reasonable quality and with its long-standing and there are many types of warranties technical specifications met. Additional motivation which can be used in different circumstances and on can be provided for specific R&M reliability by particular types of equipment. Generally, they relate contract incentives or financial penalties. The to off the shelf purchases where defective items are Department’s policy is that incentives should only be returned to contractors under contract repair payable for achievement of higher than the minimum arrangements. However, R&M warranties have requirement and should only apply where there is a recently been applied to a number of land system need for high intrinsic R&M or substantial benefits in equipment contracts including the Rapier FSC life cycle costs. Penalties for non-compliance with the missile. requirement would involve the contractor in extra expense. With warranties a contractor is committed for a fixed term and price to applying remedies to Studies of contracting for R&M by the delivered equipments which fail to meet specified Department R&M requirements (paragraph 5.5). In some circumstances, and at extra cost, the contractor can (a) Review by CODERM (1984) be required to improve in-Service reliability over the longer term by means of a reliability improvement 5.6 CODERM was tasked in 1983 to undertake a warranty. t&Service study of reliability clauses in contracts with particular regard to incentive arrangements. CODERM reported in 1984 that the majority of 5.3 High levels of R&M are more likely to be contracts valued at El million or over contained a achieved if there are taut contractual terms and reliability specification. A sample of these conditions but the establishment of an enforceable specifications showed that, while some contained contract is by no means au easy task. Indeed, the binding requirements, others merely had targets and Department have not always found it possible to some were subject to negotiation after contract letting. pursue contractual remedies when contractors have A number of contracts also contained warranty failed to achieve specified R&M requirements because provisions. However, the study could only identify of inadequately expressed specifications. Even where seven Department contracts with incentive clauses, of express warranties have applied these have not which six were for air systems, three of which (Hawk, always proved an effective remedy. Recently, as a Sea Eagle and Skyflash] are in the NAO sample. On result of failing to enforce the maximum price ceiling the evidence available the CODERM report was in the LAW 80 anti-tank weapon contract, the unable to reach any firm conclusions - some Department have been endeavouring to strengthen the contracts were successful in their aims, with or enforceability of contractual agreements. To this end without incentives, whilst others had been a failure they are making greater use of their legal adviser and even with incentives, eg Skyflash. However, a the Treasury Solicitor to vet proposed contractual common factor of the successful R&M programmes terms (paragraph 4.6). This will not necessarily was a contractual requirement which, because of its overcome, however, the problem of securing close possible effect on company profits, generated specification of R&M which is largely technological. “boardroom” interest in reliability. This usually

23 MINISTRY OF DEFENCE: RELIABIL”?’ AND MAINTAINABILITY OF DEFENCE EQUPMENT involved some form of extensive reliability NAO examination of specific projects demonstration, failure of which would involve the contractor in extra expense. As there had been 5.10 Several of the contracts examined by the NAO difficulties in conducting an investigation to the included R&M incentives. For example, the Spearfish depth required, CODERM recommended a more development and production co&act incorporated an detailed study into the methods of contracting for R&M incentive scheme which provided for up to EO.5 reliability. million in reliability bonuses but repayment to a maximum of Zl million for excessive failures. (b) Review by consultants (1988) Maintainability attracts Z%million in bonuses but no penalties. At the time of the NAO examination the 5.7 In 1985 the Defence Maintenance Committee outcome was not known although unreliability had decided to sponsor a further study into contracting already been a major cause of development trial for R&M. Due to insufficient resources within the failures. Other examples of contract incentives were Department, they decided to employ outside found in Hawk, Rapier, Sea Eagle and Sky Flash. consultants managed in their task by CODERM. Funding difficulties, and the selection process for a Industry’s contractual approach to R&M suitable consultant, prevented a start on the project until September 1986. The final report was produced 5.11 To compare the approach of industry in the in March 1988 and is at present being considered United Kingdom in achieving adequate levels of R&M within the Department. with that of the Department, the NAO consultants contacted a small number of major equipment 5.8 The first phase of the study involved an purchasers (Appendix 2). This small sample examination of a representative sample of 23 of the confirmed the views of consultants employed by the Department’s projects. However, like CODERM in Department that warranties and incentives are more their 1984 report, they were unable to reach any firm widely used in indusm than by the Department. conclusions from such an examination. They However, the NAO consultants emphasised that considered that there was insufficient evidence and industry usually has a fairly wide choice of “off the experience within the Department on which to base shelf” equipment and technical risks are generally conclusions on the best approaches to contracting for lower and requirements simpler than those of the R&M. In particular, R&M specifications were not Department. detailed enough to be used effectively in contracts; measurement of achieved R&M was rarely rigorous 5.12 All the major equipment purchasers contacted enough to be used as a basis for contractual by the NAO consultants specify quantitative R&M payments; and only a very limited range of R&M requirements for purchased equipments. In so far as contractual approaches had been applied, with few this is successful, R&M achievement can be measured examples of R&M incentives or warranties. Although and contracts more easily enforced. As a general rule the consultants had observed indications of change in industry prefers to reduce the risk of poor R&M by a number of the Department’s recent projects, they progressive improvements in existing equipment. considered these to be the exception rather than the However, in the case of new technology full scale IUle. introduction normally occurs only when sufficient knowledge and confidence has been gained from an 5.9 In their final report the consultants took the equipment demonstrator programme. view that contracting for R&M could be applied to most types of equipment and all stages of United States experience of incentives and procurement. In their view, however, this would warranties involve a significant increase in the range of contractual approaches used and, in particular, the 5.13 The United States Department of Defense have greater use of R&M incentive payments and successfully used R&M incentives and warranties for warranties. In addition, the consultants considered many years and all three American Services have that the successful application of these approaches models which help project managers to estimate life would be dependent upon a number of pre- cycle cost benefits. The American procurement conditions, such as detailed and comprehensive R&M programmes require demonstration of R&M specifications and rigorous measurement of R&M in- achievement and some have been stopped 01‘delayed Service performance. They considered that the where this has not been forthcoming. Examples of the benefits to the Department in terms of life cycle costs value for money achievements arising from vigorous should be from two to seven times the additional implementation of R&M programmes are illustrated contract cost. by:

24 MINISTRY OF DEFENCE: RELIABIUTY AND MAINTAINABILITY OF DEFENCE EQUIPMENT

(a) At a programme cost of $44 million a mean by the Department’s consultants the average time between failure of 25 hours was achieved reliability achieved, in terms of Mean Time Between against a target of nine hours on the Lamps MK Failure, was shown to be 70 per cent higher than the III helicopter avionics project; estimated life minimum specified. Warranty costs were variable, cycle cost benefits were 51.7 billion over 20 depending on the period of warranty and its coverage Yl5XS. but the United States Department of Defense guidelines indicate warranty costs from 2 to 7 per (h) Incentives were used for the Fl8 aircraft cent of the total cost of warranted items; the average contract where the contractor was awarded $12 since 1984 has been 1.9 per cent. Warranty costs in million for achieving a mean time between the United Kingdom are likely to be higher as the failure of 8.3 hours against the contracted 3.7 scale of production is less. A recent General hours. Accounting Office report on warranty legislation in the Department of Defense substantiated the cost- 5.14 Since 1984, warranties have been a legal effectiveness of warranties. The report did say, requirement for all major United States defence however, that most warranties did not clearly identify contracts. In a sample of American projects examined performance requirements and measurement criteria. MINISTRY OF DEFENCFz RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT

Glossary of Terms and Abbreviations

ADP Automatic Data Processing

“Burn&” Contractor’s procedure in which equipment is subjected in the course of production to a programme of controlled stresses (vibration, temperature etc), simulating the environmental stress which equipment will meet in Service. The procedure enables defects to be corrected before the equipment is delivered.

CODERM The Committee for Defence Equipment Reliability and Maintainability

Equipment Policy Committee A high-level Departmental committee, chaired by the Chief Scientific Adviser, which examines major equipment proposals and recommends to Ministers whether they should go forward. It is attended by representatives from the Services, the Procurement Executive, the Treasury, Foreign & Commonwealth Office and the Department of Trade and Industry.

Life Cycle Costing Models for estimating the whole life cost profile of equipment - including cost of design and development, production and in-Service support.

Maintainability Is the ease with which equipment can be inspected, overhauled, repaired and tested.

NAO National Audit Office

PAC Public Accounts Committee

Procurement cycle The design (which covers feasibility and project definition), development and production stages of procurement.

Procurement Executive Concerned with all aspects of procurement policy and Management Board practice; it is chaired by the Chief of Defence Procurement.

Production reliability Testing of equipment before delivery. Gives assurance that acceptance testing the level of reliability achieved during development has been maintained throughout production.

R&M Reliability and maintainability

Reliability Is the probability that equipment or components will work as and when required.

26 MINISTRY OF DEFENCE: RELIABILlTY AND h4ALNTAlNABILITy OF DEFENCE EQIJPMENT

Staff requirement Defines in detail equipment function and performance in operational and environmental terms. It is prepared in the light of the Feasibility Study stage and forms a basis for Project Definition. staff target Expresses in broad terms the function and desired performance of new equipment: it is the basis on which the Equipment Policy Committee approve the Feasibility Study.

27 MINISTRY OF DEFENCE: RIXIABIIJTY AND MAIiVTADUBILlTY OF DEFENCE EQUIPMENT

Appendix 1

Major R&h4 studies and initiatives since 1978

Air Systems

1979: Reliability and Cost Study Group Report The Study Group investigated the relationship between investment in R&M and resulting benefits. They failed to identify any general relationship but instead listed nine “seeds of unreliability” and recommended the setting up of a central Directorate for R&M.

1984: Triumvirate Report This study looked at ways of securing improved reliability. It concluded that, although adequate procedures existed, the demands of performance, timescale and cost too often resulted in lack of attention to reliability. It made numerous recommendations, including improved training and the use of incentives.

1987: Air Force Board Standing Committee Paper The aim of this Paper, “The Reliability of Royal Air Force Equipment”, was to raise the matter at the highest level. It highlighted the importance of initial capital investment and the use of contractual incentives, stressed the need for greater consideration of R&M at the tendering stage and called for better management of project R&M activities.

Land Systems

1980: Reliability of Equipment Purchased by the Master General of Ordnance This study analysed the operational and cost penalties of low reliability and concluded that: reliability should receive greater consideration at feasibility stages; procedures for managing and reporting reliability should be strengthened; and a data bank of in-Service reliability performance should be established. The report led to a mandatory Defence Standard on the management of R&M.

1983: Military Technology Board Report on Equipment Reliability The report concluded that the main factor hindering the achievement of high reliability was insufficient resources for reliability in design and development; this was regarded as evidence of poor management. The main reason for the lack of resources was that the consequences of poor reliability were rarely quantified and therefore those managers applying the constraints were shielded from the results of their actions.

1986: The Reliability of Land Systems Equipment This Paper commented on a lack of top-level commitment, and the absence of a suitable mechanism to specify, demonsinte and monitor equipment reliability. The paper called for additional funding for reliability.

Sea Systems 1980: Material Support of the Fleet Study

The aim was to examine ways of improving reliability and the main outcome was an Admiralty Board directive placing equal emphasis on availability and performance. The report also recommended improved training and the setting up of centres of expertise. The Naval Availability Committee was formed to coordinate policy and to oversee implementation of other recommendations through the new R&M cells.

28 MINISTRY OF DEF’BNCE RELIABILITY AND MAINTAINABILITY OF DEFENCE EQUIPMENT

1988: Fleet Availability Study The aim was to consider how the operational effectiveness of the Fleet could be improved by establishing a mechanism for a more accurate assessment of availability. This would enable resources to be directed at weak areas. An interim report, in March 1988, observed that current assessment systems were limited.

Other studies and initiatives referred to in the report

1984: CODERM Study on the Reliability Content of Contracts (paragraph 5.6)

1907: Procurement Executive Management Board Paper on the Procurement of Reliable Equipmen (paragraphs 2.5-2.6)

1988: Ministry of Defence Consultants’ Study on Contracting for Reliability (paragraphs 5.7-5.9)

29 MINISTRY OF DEFENCE: RELIABILlTY AND M.MNTAINABKJTY OF DEFENCE EQUIPMENT

Appendix 2

Surveys by NAO consultants

(A) Survey of Defence Contractors

The NAO’s consultants carried out a survey across a number of contractors in the Defence industry. The questionnaire sought to ascertain the contractor’s view on the Department’s approach to achieving R&M. Replies were received from seven contractors out of the twelve approached. The responses to each question were analysed by the NAO end, where appropriate, a general conclusion reached. These are shown in the Table below.

The Department’s R&M management during development Are R&M requirements relevant to each project; are Requirements are not always relevant and there is they even ambiguous? occasional ambiguity. especially in the detail. Are project officers aware of the content and While staff may be aware of the content of R&M implication of the R&M specification? specifications, they do not generally appreciate the implications. Are R&M requirements realistic, end when project There were mixed reactions as to the realism of officers are questioned about them, do they provide requirements but few contractors were aware of them convincing justification? having been justified. Does each project have known personnel within the Not all projects have officers with a direct Department responsible for R&M; do these project responsibility for R&M, and where such staff exist, R&M officers have adequate knowledge and they tend to be overworked. background in the appropriate disciplines? Are regular R&M progress reviews held; do project Regular R&M progress reviews do not take place on officers accept actions and respond to data requests. all projects, but where they are held, the majority of Is monitoring effective? contractors considered them to be effective. Do project officers comment on R&M submissions and Comments are generally received either from project arethese made in a timely manner. Are the officersor fromthe ProcurementExecutive R&M comments relevant? specialists. Though relevant, comments are not always timely. Are statements of the importance of R&M acted upon The level of attention devoted to R&h4 has improved in practice, in terms of adequate funds for R&M but there is still little evidence of R&M being treated testing; or the correct weighting put on trade-offs on a par with performance, cost and time in tmde- between performance and R&M? OffS. Would strict R&M contractual conditions lead to Yes; especially if the achievement of good R&M is greater emphasis being placed on R&M during project rewarded, rather than just non-achievement being development? penalised. Do project managers place importance on R&M and An increasing level of importance is attached to R&M how is this realised in practical terms? but this is still seen by some contractors as insufficient. In some cases project officers are just concerned with whether the predicted levels of R&M will satisfy the requirement.

30 MINISTRY OF DEFENCE: RELIABILITY AND MADJTAPJABWI’Y OF DEFENCE EQUIPMENT

The Department’s R&M management in-Service Are regular R&M progress reviews held? Generally yes: but they tend to be limited to major projects.

Is there regular feedback of information on R&M for None of the contractors felt that they receive adequate in-Service projects? feedback: where information is received it is usually imperfect in some respect.

Have the Department’s requirements, in the past, The majority of contractors felt not: it is only the been contractually enforceable? exceptional project manager that will enforce his contractual requirements and achieve them. (B) Survey of procedures used by major equipment purchasers in industry NAO consultants examined the procedures used by commercial industry to ensure that equipment is procured with adequate levels of R&M. The survey was limited to a small number of major equipment purchasers in industry who were considered the commercial equivalents of the Department.

1. Are R&M requirements specified? Yes Yes W21 Yes YE%

2. Importance of reasons for setting requirements: a,b ha a,b c ah b,c [a) safety; [b) loss of revenue; and/or [c) improved quality of service.

3. R&M procedures specified: a a a a b a [a) own procedures; @I) based on British Standard BS5760; or [c) based on Defence Standards. [3]

4. [i) Do suppliers generally achieve R&M requirements? No [41 - Yes Yes - varied (ii) Is: a,b a,b a b b b (a) achievement rewarded: and/or (b) under-achievement penalised using warranty-like conditions?

5. Are: a,b a,b a - a,b a,b (a) predictive; or (b) testing methods in R&M used to make purchase decisions?

6. Are most equipments: b a 50%b a b b [a) off the shelf; or [b) specially developed?

7. (i) Is R&M monitored on development equipment? NO Yes Yes Yes Yes Yes (ii) Likely R&M percentage of design and development expenditure? [6] 10% l-2% - - - -

8. (i) Are suppliers approved? Yes Yes Yes Yes Yes Yes (ii] Is R&M part of approval decision? NO Yes Yes Yes Yes Yes

Notes: (1) New policies for R&M introduced nearly two years ago. (2) Requirements limited to specified defect rate for electronic components which are 100% inspected for quality on delivery. (3) Industry procedures are generally shorter and cover fewer activities than Defence Standard 00-41 but some liited reference to it is often made. (4) Some equipments in the past have not achieved R&M requirements with serious commercial consequences - greater emphasis is now being placed on R&M and the company intend to apply Life Cycle Cost methods in the future. (6) Expenditure on R&M was not known in most cases as such costs were often hidden amongst quality and other general costs.

32