Fault-Based Analysis: How History Can Help Improve Performance and Dependability Requirements for High Assurance Systems
Total Page:16
File Type:pdf, Size:1020Kb
Fault-Based Analysis: How History Can Help Improve Performance and Dependability Requirements for High Assurance Systems Jane Huffman Hayes David M. Pruett Elizabeth Ashlee Holbrook Geocontrol Systems Inies Raphael C.M. Incorporated University of Kentucky Dept. of Computer Science [email protected] [email protected], {ashlee|irchem2}@uky.edu Abstract Related work is discussed in Section 3. Our position is outlined in Section 4 as well as some Performance and dependability requirements are preliminary results. Finally, conclusions and future key to the development of high assurance systems. work round out the paper in Section 5. Fault-based analysis has proven to be a useful tool for detecting and preventing requirement faults 2. Fault-based analysis early in the software life cycle. By tailoring a generic fault taxonomy, one is able to better “The IEEE standard definition of an error is a prevent past mistakes and develop requirements mistake made by a developer. An error may lead to specifications with fewer overall faults. Fewer one or more faults [13]. To understand Fault- faults within the software specification, with Based Analysis (FBA), a look at a related respect to performance and dependability technique, Fault-Based Testing (FBT), is in order. requirements, will result in high assurance systems Fault-based testing generates test data to of improved quality. demonstrate the absence of a set of pre-specified faults. There are numerous FBT techniques. These 1. Introduction use a list of potential faults to generate test cases, generally for unit- and integration-level testing One needs only look to the increasing [20,3]. Research has been performed in the area of importance of computer systems in our society and software safety fault identification [20, 6], the increasingly trusted roles that they play to including research into numerous fault analysis understand how important it is to ensure that high techniques such as Petri-net safety analysis [16,17], assurance systems are correctly built. For example, Failure Mode, Effects, Criticality Analysis power generation has largely become digital (FMECA) [19], and criticality analysis [29].” [10] (instrumentation and control systems for nuclear Similar to FBT, FBA identifies static techniques, power plants are becoming increasingly digital) as such as traceability analysis, and specific activities well as aviation (autopilot and other important within those techniques that should be performed to systems of large, commercial aircraft are now ensure that a set of pre-specified faults do not exist. digital). Clearly we need these software systems to Fault-based analysis is risk-driven, and attempts to be dependable. Performance also plays an select V&V techniques to apply in order to best important role in many software systems (many achieve a project’s goals. In performing FBA, one functions are time critical). targets the strongest fault class or classes [12]. A In order to ensure that high assurance software more extensive survey of related work, such as systems possess the necessary performance and orthogonal defect classification [4] can be found in dependability qualities, it is essential that their [8]. requirements are specified correctly. Though Fault-based analysis, as applied to requirement performance and dependability are non-functional faults, can help prevent and/or detect faults early in requirements, they are still requirements and hence the software lifecycle, resulting in significant cost can potentially be improved using methods and savings [1]. In earlier work by Hayes [8], a generic techniques that have been proven to have general fault taxonomy was selected as the basis for applicability to the requirements domain. In this requirements FBA, requirements faults were paper, we argue that a process called fault-based examined, and a method for extending a taxonomy analysis can be applied to performance and was developed and implemented. Historical data dependability requirements for high assurance can be used to determine the fault types that are systems. most likely to be introduced and risk analysis can The paper is organized as follows. Section 2 presents an overview of fault-based analysis. NASA project Generic fault requirements taxonomy faults/problem reports Class-project to extend a fault taxonomy for a software project class Class A fault Class B fault Class C fault Class D fault taxonomy taxonomy (e.g., taxonomy (e.g., taxonomy (e.g., (e.g., Other projects Manned Exploration Aerospace, Earth Biological/Physical which do not fall into Manned Missions) Science, Space projects) Class A, B, or C) Science projects) Project-process to extend a taxonomy for a project CI and problem Fault taxonomy report for a specific information project CI-process to identify fault distribution across subsystems -- Data Fault distribution -- Process Figure 1: High level process for extending fault taxonomies. be performed to determine the fault types that release), one looks at the historical trends of would be most devastating if overlooked. requirements defect reports from the previous We can identify techniques that help reduce the releases. If beginning development of a new risk of, prevent, and detect prevalent or targeted product, one may look at the historical trends from fault types. These techniques are then applied as that organization on earlier products or from related part of the V&V (Verification and Validation) organizations (with personnel overlap, such as and/or IV&V (Independent Verification and management). Validation)effort [8]. To provide a requirements- based fault analysis approach, an overall 3. Tailoring of fault-based analysis methodology was defined [5]: (i) build a requirement fault taxonomy and a process for Our position is that FBA can be applied to tailoring it; (ii) build a taxonomy of V&V improve dependability and performance techniques and build a matrix of their validated requirements. Improved performance and fault detection capabilities; and (iii) develop dependability requirements will result in high guidance to V&V agents and software projects for assurance systems of improved quality. The US use of the fault-based analysis methodology and spends approximately $59.5 billion each year on assist in its adoption. Just as project quality software errors, according to a 2002 Study by the improves when one prioritizes requirements [7], National Institute of Standards (NIST). Lack of quality increases when a mechanism is in place to software requirements as well as incorrect prioritize fault countermeasures and choose the requirements contributes heavily to this problem most effective fault detection and prevention [15]. methods. We base our position on prior work with FBA, Note that we require defect reports related to as applied on various computer software requirements. This is the starting point for the capabilities of the International Space Station [14], FBA process. If starting a new release for an a manned flight system and thus high assurance existing system (writing requirements for the new [11]. In developing FBA, we developed a process that to collections of related projects, or domains. can be tailored or applied to: various levels of the Given defect information for a collection of software architecture, to historical “slices” of time, projects, the FBA process is applied (as discussed etc. Let us examine a few examples of such in Section 2) and the result is a tailored taxonomy “tailoring.” FBA can be applied at a generic level Table 1: Configuration item (CI) process for tailoring a taxonomy. Entry Criteria Activities Exit Criteria 1. All inputs are available 1. Select project-specific requirement fault 1. A CI-specific requirement 2. NASA has authorized taxonomy fault taxonomy has been use of project data 2. Select a CI from the list of project CIs developed 3. NASA has authorized 3. Categorize the fault for the CI according to the the taxonomy extension project-specific fault taxonomy project 4. Determine the frequency of faults for the CI 5. Identify the crucial fault categories for the CI 6. Repeat Steps 2 through 5 for all other CIs Inputs Process Controls/Metrics Outputs 1. Project-specific fault Process Controls: 1. Frequency counts of faults taxonomy 1. Maintenance of configuration control of 2. Crucial fault categories for 2. Requirement taxonomy the CI faults/problem reports 2. Maintenance and management of NASA CI 3. Prioritized fault list for the for the CIs data by project CI 3. CI-specific information Metrics: (goals priorities for all 1. Person hours for effort CIs 2. Number of CIs 3. Number of faults 4. Historic probability of occurrence 5. Fault exposure values (fault frequency occurrence) that helps to categorization percentage data for the International characterize the domain as a whole. For example, Space Station as a whole. By examining available we examined the fault profiles for a number of high data and performing trend analysis, we are able to assurance NASA systems, applied the FBA tailor taxonomy to prioritize the particular fault process, and created a tailored fault taxonomy for areas that have been historically significant. Table NASA Class A systems (NASA’s term for high 3 shows our tailored taxonomy for the particular assurance systems) [11]. Next, we wanted to apply configuration items we have examined [8]. FBA to a specific project. We took the Class A Once we have obtained the historical fault taxonomy, project specific information, and applied profile, we meet with engineers