DOCSLIB.ORG

The Role of Software in Recent Aerospace Accidents*

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
The Role of Software in Recent Aerospace Accidents* The Role of Software in Recent Aerospace Accidents* Nancy G. Leveson, Ph.D.; Aeronautics and Astronautics Department, Massachusetts Institute of Technology; Cambridge MA [email protected] and http://sunnyday.mit.edu Keywords: software safety Abstract factors are considered here, that is, the causal factors that allowed the specific events to occur This paper describes causal factors related to and that affect general classes of accidents. In software that are common to several recent the Challenger accident, for example, the spacecraft accidents and what might be done to specific even leading to the loss was the O-ring mitigate them. failure, but the systemic factors included such Introduction things as flawed decision making, poor problem reporting, lack of trend analysis, a "silent" or In the process of a research project to evaluate ineffective safety program, communication accident models, I looked in detail at a variety of problems, etc. spacecraft and aircraft accidents that in some way involved software [8]. The accidents Overconfidence and Overreliance on Digital studied all demonstrated the usual cultural, Automation: All the accidents involved systems organizational, and communications problems built within an engineering culture that had such as complacency, diffusion of or lack of unrealistic expectations about software and the responsibility and authority for safety, low-level use of computers. For example, the official status and inappropriate organizational Ariane 5 accident report notes that software was placement of the safety program, and limited assumed to be correct until it was shown to be communication channels and poor information faulty. The opposite assumption is more flow. These typical problems are well known realistic. and the solutions clear although sometimes difficult to implement. Software contributions to Engineers often underestimate the complexity of accidents are less well understood, however. software and overestimate the effectiveness of testing. It is common to see risk assessments The accidents investigated were the explosion of that assume testing will remove all risk the Ariane 5 launcher on its maiden flight in associated with digital components. This form 1996; the loss of the Mars Climate Orbiter in of complacency plays a part in the common 1999; the destruction of the Mars Polar Lander proliferation of software functionality and in sometime during the entry, deployment, and unnecessary design complexity. landing phase in the following year; the placing of a Milstar satellite in an incorrect and unusable In the aircraft accidents examined, orbit by the Titan IV B-32/Centaur launch in overconfidence in automation both (1) 1999; the flight of an American Airlines B-757 encouraged engineers to trust software over into a mountain near Cali, Columbia; the humans and give final authority to the computer collision of a Lufthansa A320 with an earthbank rather than the pilot, and (2) encouraged pilots to at the end of the runway at Warsaw, and the trust their computer-based decision aids beyond crash of a China Airlines A320 short of the the point where they should have. runway at Nagoya, Japan. Some of the technical inadequacies in high-tech On the surface, the events and conditions aircraft system design stem from lack of involved in the accidents appear to be very confidence in the human and overconfidence in different. A more careful, detailed analysis of the automation. In several of the Airbus the systemic factors, however, reveals striking accidents, the pilots found themselves fighting similarities. Only the root causes or systemic the automation for control of the aircraft---which * This research was partially supported by a grant from the NASA Ames Design for Safety program and by the NASA IV&V Center Software Initiative program. had been designed to give ultimate authority to modes for software are very different than those the automation. for physical devices and the contribution of software to accidents is also different: Even if automation is considered to be more Engineering activities must be changed to reflect reliable than humans, it may be a mistake not to these differences. Almost all software-related allow flexibility in the system for emergencies accidents can be traced back to flaws in the and allowance for pilots to override physical requirements specification and not to coding interlocks, such as the inability of the pilots to errors. In these cases, the software performed operate the ground spoilers and engine thrust exactly as specified (the implementation was reversers in the Warsaw A-320 accident because "correct") but the specification was incorrect the computers did not think the airplane was on because (1) the requirements were incomplete or the ground. Reliable operation of the automation contained incorrect assumptions about the is not the problem here; the automation was very required operation of the system components reliable in all these cases. Instead the issue is being controlled by the software or about the whether software can be constructed that will required operation of the computer or (2) there exhibit correct appropriate behavior under every were unhandled controlled-system states and foreseeable and unforseeable situation and environmental conditions. This in turn implies whether we should be trusting software over that the majority of the software system safety pilots. effort should be devoted to requirements analysis, including completeness (we have At the same time, some of the aircraft accident specified an extensive set of completeness reports cited the lack of automated protection criteria), correctness, potential contribution to against or nonalerting of the pilots to unsafe system hazards, robustness, and possible states, such as out-of-trim situations. A operator mode confusion and other operator sophisticated hazard analysis and close errors created or worsened by software design. cooperation among system safety engineers, human factors engineers, aerospace engineers, Confusing Reliability and Safety: Accidents are and software engineers is needed to make these changing their nature. We are starting to see an difficult decisions about task allocation and increase in system accidents that result from feedback requirements. dysfunctional interactions among components, not from individual component failure. Each of Engineers are not alone in placing undeserved the components may actually have operated reliance on software. Research has shown that according to its specification (as is true for most operators of highly reliable automated systems software involved in accidents), but the (such as flight management systems) will combined behavior led to a hazardous system increase their use of and reliance on automation state. When humans are involved, often their as their trust in the system increases. At the behavior can only be labeled as erroneous in same time, merely informing flightcrews of the hindsight–at the time and given the context, their hazards of overreliance on automation and behavior was reasonable (although this does not advising them to turn it off when it becomes seem to deter accident investigators from placing confusing is insufficient and may not affect pilot all or most of the blame on the operators). procedures when it is most needed. System accidents are caused by interactive The European Joint Aviation Authorities' Future complexity and tight coupling. Software allows Aviation Safety Team has identified "crew us to build systems with a level of complexity reliance on automation'' as the top potential and coupling that is beyond our ability to safety risk in future aircraft [5]. This reliance on control; in fact, we are building systems where and overconfidence in software is a legitimate the interactions among the components cannot be and important concern for system safety planned, understood, anticipated, or guarded engineering. against. This change is not solely the result of using digital components, but it is made possible Not Understanding the Risks Associated with because of the flexibility of software. Software: The accident reports all exhibited the common belief that the same techniques used for Standards for commercial aircraft certification, electromechanical components will work in even relatively new ones, focus on component software-intensive systems. However, the failure reliability and redundancy and thus are not effective against system accidents. In the aircraft be false in practice and by scientific experiments accidents studied, the software satisfied its (see, for example, [4]). Common-cause (but specifications and did not "fail" yet the usually different) logic errors tend to lead to automation obviously contributed to the flight incorrect results when the various versions crews' actions and inactions. Spacecraft attempt to handle the same unusual or difficult- engineering in most cases also focuses primary to-handle inputs. In addition, such designs effort on preventing accidents by eliminating usually involve adding to system complexity, component failures or preparing for failures by which can result in failures itself. A NASA study using redundancy. These approaches are fine for of an experimental aircraft with two versions of electromechanical systems and components, but the control system found that all of the software will not be effective for software-related problems occurring during flight testing resulted accidents. from errors in the redundancy management system and not in
Load more

Recommended publications
  • Standardizing Functional Safety Assessments for Off-The-Shelf Instrumentation and Controls
    University of Tennessee, Knoxville TRACE: Tennessee Research and Creative Exchange Masters Theses Graduate School 5-2016 STANDARDIZING FUNCTIONAL SAFETY ASSESSMENTS FOR OFF-THE-SHELF INSTRUMENTATION AND CONTROLS Andrew Michael Nack University of Tennessee - Knoxville, [email protected] Follow this and additional works at: https://trace.tennessee.edu/utk_gradthes Part of the Other Computer Engineering Commons, and the Systems Engineering Commons Recommended Citation Nack, Andrew Michael, "STANDARDIZING FUNCTIONAL SAFETY ASSESSMENTS FOR OFF-THE-SHELF INSTRUMENTATION AND CONTROLS. " Master's Thesis, University of Tennessee, 2016. https://trace.tennessee.edu/utk_gradthes/3793 This Thesis is brought to you for free and open access by the Graduate School at TRACE: Tennessee Research and Creative Exchange. It has been accepted for inclusion in Masters Theses by an authorized administrator of TRACE: Tennessee Research and Creative Exchange. For more information, please contact [email protected]. To the Graduate Council: I am submitting herewith a thesis written by Andrew Michael Nack entitled "STANDARDIZING FUNCTIONAL SAFETY ASSESSMENTS FOR OFF-THE-SHELF INSTRUMENTATION AND CONTROLS." I have examined the final electronic copy of this thesis for form and content and recommend that it be accepted in partial fulfillment of the equirr ements for the degree of Master of Science, with a major in Computer Engineering. Gregory D. Peterson, Major Professor We have read this thesis and recommend its acceptance: Qing C. Cao, Mingzhou Jin Accepted for the Council: Carolyn R. Hodges Vice Provost and Dean of the Graduate School (Original signatures are on file with official studentecor r ds.) STANDARDIZING FUNCTIONAL SAFETY ASSESSMENTS FOR OFF-THE-SHELF INSTRUMENTATION AND CONTROLS A Thesis Presented for the Master of Science Degree The University of Tennessee, Knoxville Andrew Michael Nack May 2016 Copyright © 2016 by Andrew Michael Nack All rights reserved.
    [Show full text]
  • A Software Safety Process for Safety-Critical Advanced Automotive Systems
    PROCEEDINGS of the 21st INTERNATIONAL SYSTEM SAFETY CONFERENCE - 2003 A Software Safety Process for Safety-Critical Advanced Automotive Systems Barbara J. Czerny, Ph.D.; Joseph G. D’Ambrosio, Ph.D., PE; Paravila O. Jacob, Ph.D.; Brian T. Murray, Ph.D.; Padma Sundaram; Delphi, Corp.; Brighton, Michigan Keywords: SW Safety, Safety-Critical Advanced Automotive Systems, By-Wire Systems Abstract A new generation of advanced automotive systems are being implemented to enhance vehicle safety, performance, and comfort. As these new, often complex systems are added, system safety programs are employed to help eliminate potential hazards. A key component of these advanced automotive systems is software. Software itself cannot fail or wear out, but its complexity coupled with its interactions with the system and the environment can directly and indirectly lead to potential system hazards. As such, software safety cannot be considered apart from system safety, but the unique aspects of software warrant unique development and analysis methods. In this paper we describe the main elements of a software safety process for safety-critical advanced automotive systems. We describe how this proposed process may be integrated with an established system safety process for by-wire automotive systems, and how it may be integrated with an established software development process. Introduction Expanding demand for further improvements in vehicle safety, performance, fuel economy and low emissions has led to a rapid and accelerating increase in the amount and sophistication of vehicle electronics. Emerging vehicle electronics systems are programmable, with a substantial software component, and are highly complex distributed systems. Increasingly, they are receiving driver inputs and directly controlling essential vehicle functions like braking and steering.
    [Show full text]
  • Appendix J Software Safety
    FAA System Safety Handbook, Appendix J: Software Safety December 30, 2000 Appendix J Software Safety SOFTWARE SAFETY...............................................................................................................................1 J.0 SOFTWARE SAFETY DURING LIFE CYCLE PHASES................................................................2 J-1 FAA System Safety Handbook, Appendix J: Software Safety December 30, 2000 J.0 Software Safety During Life Cycle Phases The safety process should support a structured program life cycle model that incorporates both the system design and engineering process and the software acquisition process. Prominent software life cycle models include the waterfall and spiral methodologies. Although different models may carry different lifecycle emphasis, the adopted model should not affect the safety process itself. For discussion purposes only, this enclosure adopts a waterfall model (subject to IEEE/IEA Standard for Information Technology-software life cycle processes No. 12207.) For brevity, only the development phase of the Standard is addressed in terms of the relationship to software safety activities. J.1 Safety Critical Software Development A structured development environment and an organization using state-of-the-art methods are prerequisites to developing dependable safety critical software. The following requirements and guidelines are intended to carry out the cardinal safety rule and its corollary that no single event or action shall be allowed to initiate a potentially hazardous event. The system, upon detection of an unsafe condition or command, shall inhibit the potentially hazardous event sequence and originate procedures/functions to bring the system to a predetermined “safe” state. The purpose of this section is to describe the software safety activities that should be incorporated into the software development phases of project development. The software safety information that should be included in the documents produced during these phases is also discussed.
    [Show full text]
  • A Study on Hazard Analysis in High Integrity Software Standards and Guideiines
    A Study on Hazard Analysis in High Integrity Software Standards and Guideiines Laura M. Ippolito Dolores R. Wallace U.S. DEPARTMENT OF COMMERCE Technology Administration National Institute of Standards and Technology Computer Systems Laboratory Gaithersburg, MD 20899 QC 100 isiisr .056 NO. 5589 1995 A Study on Hazard Analysis in High integrity Software Standards and Guideiines Laura M. Ippolito Dolores R. Wallace U.S. DEPARTMENT OF COMMERCE Technology Administration National Institute of Standards and Technology Computer Systems Laboratory Gaithersburg, MD 20899 January 1995 U.S. DEPARTMENT OF COMMERCE Ronald H. Brown, Secretary TECHNOLOGY ADMINISTRATION Mary L. Good, Under Secretary for Technology NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY Arati Prabhakar, Director ABSTRACT This report presents the results of a study on hazard analysis, especially software hazard analysis, in high integrity software standards and guidelines. It describes types of system hazard analysis (that influence software), types of software hazard analysis, techniques for conducting hazard analysis (along with some of their advantages and disadvantages), and other practices and processes that should be employed in order to ensure the safety of software. KEYWORDS High integrity software, software assurance, software development, software engineering, software hazard analysis, software quality, software reliability, software safety, system hazard analysis. Ill TABLE OF CONTENTS Page EXECUTIVE SUMMARY vii ABBREVIATIONS ix 1. INTRODUCTION 1 1.1. Review Process 2 2. CONTEXT FOR SOFTWARE HAZARD ANALYSIS 5 2.1. Criticality Assessment 5 2.2. Types of Software Related System Hazard Analyses 7 2.3. Types of Software Hazard Analysis 7 3. HAZARD ANALYSIS TECHNIQUES 11 3.1. Code Walk-Throughs 13 3.2.
    [Show full text]
  • Guide to Reusable Launch and Reentry Vehicle Software and Computing System Safety
    FAA Commercial Space Transportation Guide to Reusable Launch and Reentry Vehicle Software and Computing System Safety Version 1.0 July 2006 Guide to Reusable Launch and Reentry Vehicle Software and Computing System Safety Version 1.0 July 2006 Federal Aviation Administration Office of Commercial Space Transportation 800 Independence Avenue, Room 331 Washington, DC 20591 NOTICE Use of trade names or names of manufacturers in this document does not constitute an official endorsement of such products or manufacturers, either expressed or implied, by the Federal Aviation Administration. TABLE OF CONTENTS 1.1 INTRODUCTION ........................................................................................................ 1 1.2 Purpose .................................................................................................................. 1 1.3 Background ........................................................................................................... 1 1.4 Scope ..................................................................................................................... 1 1.5 Authority ............................................................................................................... 1 2.1 DEFINITIONS AND ACRONYMS ............................................................................ 2 2.2 Definitions ............................................................................................................. 2 2.3 Acronyms .............................................................................................................
    [Show full text]
  • Software Quality Assurance Applied Towards the Development of VHDL‐Based Safety Critical Hardware
    Software Quality Assurance Applied towards the Development of VHDL‐Based Safety Critical Hardware David A. Geremia Principal Electrical Design Engineer [email protected] 61st NDIA Annual Fuze Conference San Diego CA May 16, 2018 NOTICE ‐ This document does not contain export controlled technical data. Introduction The software used in today’s safety critical systems requires a significant amount of analysis and testing as well as traceability to the requirements “Software‐like” languages are treated similarly by today’s munition‐related safety technical review panels 2 Introduction Very High Speed Integrated Circuit (VHSIC) Hardware Description Language (VHDL) is one of these “software‐like” languages Requires the generation of the appropriate Level of Rigor (LOR) and the resultant analyses As part of the academic pursuit on which this presentation is based, software was created in order to automate the generation of the appropriate LOR tasks, establish traceability, & provide transparency 3 Background The implementation of safety features in safety critical systems has evolved in the last few decades Initially, safety features were implemented using a mechanical means such as springs, setback weights, rotors and shear pins Recently, electronics have been used in order to implement safety features i.e. analog and/or simple digital circuits 4 Background Most recently, software and “software‐like” devices are being used to implement safety features Field Programmable Gate Arrays (FPGAs) are hardware devices that are being used more often in today’s munition‐related safety‐critical applications in order to implement safety features A high‐level language (such as VHDL) is used to design the safety features which are implemented using an FPGA.
    [Show full text]
  • Joint Software Systems Safety Engineering Handbook ------Developed by the Joint Software Systems Safety Engineering Workgroup
    DEPARTMENT OF DEFENSE JOINT SOFTWARE SYSTEMS SAFETY ENGINEERING HANDBOOK ----------------------------------------- DEVELOPED BY THE JOINT SOFTWARE SYSTEMS SAFETY ENGINEERING WORKGROUP Original published December 1999 Version 1.0 Published August 27, 2010 Naval Ordnance Safety and Security Activity 3817 Strauss Avenue, Building D-323 Indian Head, MD 20640-5555 Prepared for: Director of Defense Research and Engineering Distribution Statement A Approved for public release; distribution is unlimited. Software Systems Safety Engineering Handbook Table of Contents Table of Contents 1 Overview ........................................................................................................... 1 2 Introduction to the Handbook ........................................................................... 3 2.1 Introduction ....................................................................................................... 3 2.2 Purpose .............................................................................................................. 4 2.3 Scope ................................................................................................................. 4 2.4 Authority and Standards ................................................................................... 5 2.5 Handbook Overview ......................................................................................... 6 2.5.1 Historical Background ...................................................................................... 6 2.5.2 Management Responsibilities
    [Show full text]
  • A Validation Metrics Framework for Safety-Critical Software-Intensive Systems
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Calhoun, Institutional Archive of the Naval Postgraduate School Calhoun: The NPS Institutional Archive Theses and Dissertations Thesis Collection 2009-03 A validation metrics framework for safety-critical software-intensive systems Cruickshank, Kristian John. Monterey, California. Naval Postgraduate School http://hdl.handle.net/10945/4891 NAVAL POSTGRADUATE SCHOOL MONTEREY, CALIFORNIA THESIS A VALIDATION METRICS FRAMEWORK FOR SAFETY- CRITICAL SOFTWARE-INTENSIVE SYSTEMS by Kristian John Cruickshank March 2009 Thesis Advisor: James B. Michael Thesis Co-Advisor: Man-Tak Shing Approved for public release; distribution is unlimited THIS PAGE INTENTIONALLY LEFT BLANK REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503. 1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED March 2009 Master’s Thesis 4. TITLE AND SUBTITLE A Validation Metrics Framework for Safety-Critical 5. FUNDING NUMBERS Software-Intensive Systems 6. AUTHOR(S) Kristian John Cruickshank 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8.
    [Show full text]
  • NASA Software Safety Guidebook
    NOT MEASUREMENT SENSITIVE National Aeronautics and NASA-GB-8719.13 Space Administration March 31, 2004 NASA Software Safety Guidebook NASA TECHNICAL STANDARD PREFACE Effective Date: 31 March 2004 This document has been issued to make available to software safety practitioners a guidebook for assessing software systems for software’s contribution to safety and techniques for analyzing and applying appropriate safety techniques and methods to software. Software developers and software safety engineers are the primary focus; however, software assurance (SA) engineers, project managers, system engineers, and system safety engineers will also find this guidebook useful. The document: Provides an overview of general software safety and good software engineering practices which contribute to software system safety. Provides the means to scope and tailor the software safety and software engineering activities to obtain the most cost effective, best quality, and safest products. Provides analyses, methods and guidance which can be applied during each phase of the software life cycle. Multiple checklists and examples are provided as well as specific instructions for applying FMEA/CIL and FTA to software. Includes development approaches, safety analyses, and testing methodologies that lead to improved safety in the software product. Procuring NASA Enterprise Programs or Centers shall review this document for applicability to NASA contracts as well as for applicability to its internal activities. Questions concerning the application of this publication to specific procurements or requests should be referred to the NASA Enterprise Program or Center. This guidebook cancels NASA-GB-1740.13-96, NASA Guidebook for Safety Critical Software Analysis and Development. /s/ Bryan O’Connor Associate Administrator for Safety and Mission Assurance NASA-GB-8719.13 1 Foreword This guidebook was created to provide specific information and guidance on the process of creating and assuring safe software.
    [Show full text]
  • Software Engineering for Safety: a Roadmap Robyn Lutz
    Software Engineering for Safety: A Roadmap Robyn Lutz Key Research Pointers Provide readier access to formal methods for developers of safety-critical systems by further integration of informal and formal methods. Develop better methods for safety analysis of product families and safe reuse of Commercial- Off-The-Shelf software. Improve the testing and evaluation of safety-critical systems through the use of requirements- based testing, evaluation from multiple sources, model consistency, and virtual environments. Advance the use of runtime monitoring to detect faults and recover to a safe state, as well as to profile system usage to enhance safety analyses. Promote collaboration with related fields in order to exploit advances in areas such as security and survivability, software architecture, theoretical computer science, human factors engineering, and software engineering education. The Author Robyn R. Lu~z is a senior engineer at Jet Propulsion Laboratory, California Institute of Technology. She is also an Affiliate Assistant Professor in the Department of Computer Science at Iowa State University, Ames, Iowa, where she teaches software engineering. Dr. Lutz has worked on spacecraft projects in fault protection, real-time commanding, and software requirements and design verification. Her research interests include software safety, software certification, safe reuse of product families, formal methods for requirements analysis, and fault monitoring and recovery strategies for spacecraft: http://www.cs.iastate.edu/-rlutz/; email: [email protected].
    [Show full text]
  • AC 450.141-1 Computing Systems and Software
    U.S. Department Advisory of Transportation Federal Aviation Administration Circular Subject: Computing System Safety Date: 10/15/2020 AC No: 450.141-1 Initiated By: AST-1 This Advisory Circular (AC) provides guidance for an applicant to identify computing system safety items, develop safety requirements for each computing system safety item, and mitigate the risks presented by computing system safety items in accordance with title 14 of the Code of Federal Regulations (14 CFR) § 450.141. An applicant must identify all computing system safety items and implement safety requirements for each computing system safety item based on level of criticality, in accordance with § 450.141(a) and (b). An applicant must then implement a development process appropriate for each computing system safety item’s level of criticality, in accordance with § 450.141(c). The FAA considers this AC an accepted means of compliance with the regulatory requirements of § 450.141. It presents one, but not the only, acceptable means of compliance with the associated regulatory requirements. The FAA will consider other means of compliance that an applicant may elect to present. The contents of this document do not have the force and effect of law and are not meant to bind the public in any way. The document is intended only to provide clarity to the public regarding existing requirements under the law or agency policies. If you have suggestions for improving this AC, you may use the Advisory Circular Feedback form at the end of this AC. Wayne R. Monteith Associate Administrator
    [Show full text]
  • Software System Safety
    Software System Safety Nancy G. Leveson MIT Aero/Astro Dept. ([email protected]) http://sunnyday.mit.edu MIT, Room 33−334 77 Massachusetts Ave., Cambridge MA 02139 Tel: 617−258−0505 c Copyright by the author,, February 2004.All rights reserved. Copying without fee is permitted provided that the copies are not made or distributed for direct commercial advantage and provided that credit to the source is given. Abstractingwith credit is permitted. Overview of the Class c ¢¡¢£¤¡¢¥¤¦¢§©¨ Week 1: Understanding the Problem Week 2: The Overall Process and Tasks Week 3: Hazard Causal Analysis (Root Cause Analysis) Week 4: A New Approach to Hazard, Root Cause, and Accident Analysis Week 5: Requirements Analysis Week 6: Design for Safety Week 7: Human−Machine Interaction and Safety Week 8: Testing and Assurance, Operations, Maintenance Week 9: Management and Organizational Issues (including safety culture) Week 10: Summary and Conclusions c ¢¡¢£¤¡¢¥¤¦¢§©¨ The Problem The first step in solving any problem is to understand it. We often propose solutions to problems that we do not understand and then are surprised when the solutions fail to have the anticipated effect. Accident with No Component Failures c ¢¡¢£¤¡¢¥¤¦¢§©¨ ¢¡© ¦¢¢ ¡¢ VENT LA GEARBOX LC CONDENSER CATALYST VAPOR COOLING WATER REFLUX REACTOR COMPUTER c ¢¡¢£¤¡¢¥¤¦¢§©¨ ¢¡© ¦¢¢ ¡¢ Types of Accidents Component Failure Accidents Single or multiple component failures Usually assume random failure System Accidents Arise in interactions among components No components may have "failed" Caused by interactive complexity and tight coupling Exacerbated by the introduction of computers. c ¢¡¢£¤¡¢¥¤¦¢§©¨ $ ¦¢¢ ¡¢ Interactive Complexity ¢¡© Complexity is a moving target The underlying factor is intellectual manageability 1. A "simple" system has a small number of unknowns in its interactions within the system and with its environment.
    [Show full text]