DOCSLIB.ORG

A System-Theoretic Safety Engineering Approach for Software

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A System-Theoretic Safety Engineering Approach for Software A System-Theoretic Safety Engineering Approach for Software-Intensive Systems Von der Fakultät Informatik, Elektrotechnik und Informationstechnik der Universität Stuttgart zur Erlangung der Würde eines Doktors der Naturwissenschaften (Dr. rer. nat.) genehmigte Abhandlung Vorgelegt von Asim Abdulkhaleq geboren in Taiz, Jemen Hauptberichter: Prof. Dr. Stefan Wagner Mitberichter: Prof. Dr. Nancy Leveson (MIT) Tag der mündlichen Prüfung: 06.02.2017 Institut für Softwaretechnologie 2017 D This PhD research Thesis is dedicated to my best friend Redwan Alakori who passed away on Saturday, 08.10.2016 because of an elevator accident in Cairo, Egypt. 3 A Software safety is a crucial aspect during the development of modern safety- critical systems. However, safety is a system level property, and therefore, software safety must be considered at the system-level to ensure the whole sys- tem’s safety [Lev11]. In fact, making software safe is more than just identifying software hazards. The software components in the systems need also to be verified extensively against their safety requirements to ensure a high level of system safety. The complexity of software systems makes it difficult to define appropriate software safety requirements with traditional safety analysis tech- niques such as FTA (Fault Tree Analysis) [Ves+81], FMEA (Failure Mode and Effects Analysis) [Mil49]. To cope with complex systems, a new technique called STPA (System-Theoretic Process Analysis)[Lev11] based on system and control theory has been developed by Leveson. In the software development process, formal verification and functional test- ing are complementary approaches which are used to verify the functional correctness of software; however, even perfectly reliable software could lead to an accident [Won+10; Lev11]. The correctness of software cannot ensure the safe operation of safety-critical software systems. Therefore, developing safety- critical software requires a more systematic software and safety engineering process that enables the software and safety engineers to recognize the potential software risks. For this purpose, this dissertation introduces a comprehensive safety engineering approach based on STPA for Software-Intensive Systems, called STPA SwISs [AWL15], which provides seamless STPA safety analysis and software safety verification activities to allow the software and safety engineers to work together during the software development for safety-critical systems and help them to recognize the associated software risks at the system level. To explore and evaluate the application of the STPA SwISs approach, we conducted a pilot case study and two industrial case studies based on automotive software systems. The pilot case study was conducted during the development of a software simulator of the Adaptive Cruise Control System (ACC) with a stop-and-go function using a Lego-Mindstorms EV3 robot. The first industrial case study was conducted on the Active Cruise Control System (ACC) at BMW Group. The second case study was conducted on the fully Automated Driving 5 system (AD) of autonomous vehicles at Continental. The results demonstrate the effectiveness of the STPA SwISs approach and show that it is scalable and applicable to real-world software systems. We also developed a safety engineering platform called XSTAMPP to support the application of STAMP methodologies and the STPA SwISs approach to help software and safety engineers to identify the software safety requirements, automatically verify their design and implementation against the STPA-generated software safety requirements with model checking tools and automatically generate safety-based test cases from STPA results. As a conclusion, we believe that the combination of the STPA-based safety analysis and the safety-based software verification activities is a practical and effective way to recognize software risks and assure the quality of software. The degree of automation added value to the proposed approach by allowing the safety and software engineers to perform seamless safety analysis, software safety verification and safety-based testing activities in one comprehensive ap- proach. Furthermore, this dissertation opens doors for interesting opportunities in software safety engineering. 6 Z Software-Sicherheit ist ein entscheidender Aspekt bei der Entwicklung moderner sicherheitskritischer Systeme. Er betrifft die Sicherheit des gesamten Systems und muss daher auch auf Systemebene behandelt werden. Das Erkennen von Software-Risiken allein reicht indes nicht aus, um die Sicherheit einer Software zu gewährleisten. Vielmehr müssen die Software-Komponenten des Systems auch umfassend gegen die Sicherheitsanforderungen des Gesamtsystems geprüft werden, um ein hohes Maß an Systemsicherheit zu erreichen. Aufgrund der Komplexität von Software-Systemen ist es schwierig, mit traditionellen Ana- lyseverfahren wie FTA (Fault Tree Analysis) oder FMEA (Failure Mode and Effects and Analysis) geeignete Software-Sicherheitsanforderungen zu definie- ren. Ein neues Verfahren zur Analyse komplexer Systeme ist die 2004 von Nancy Leveson entwickelte, auf einem systemtheoretischen Unfallmodell basierende Systemtheoretische Prozessanalyse (STPA). Bei der Software-Entwicklung sind formale Verifikation und Tests einander er- gänzende Ansätze, um die funktionale Korrektheit von Software zu prüfen. Doch könnte es bei selbst vollkommen zuverlässiger Software zu Unfällen kommen. Selbst bei absolut korrekter Software ist kein sicherer Betrieb si- cherheitskritischer Software-Systeme gewährleistet. Daher erfordert die Ent- wicklung sicherheitskritischer Software einen systematischeren Prozess bei der sicherheitstechnischen und der Software-Entwicklung, der die Entwickler be- fähigt, potenzielle Software-Risiken zu erkennen. Die vorliegende Dissertati- on stellt ein umfassendes, auf STPA für Software-intensive Systeme basieren- des Software-Sicherheitsverfahren namens STPA SwISs vor. Dieses Verfahren stellt verschiedene Maßnahmen für STPA-Sicherheitsanalysen und die Software- Sicherheitsverifikation bereit, die es Software- und Sicherheitstechnikern ermög- lichen, bei der Software-Entwicklung für sicherheitskritische Systeme zusam- menzuarbeiten und die damit verbundenen Software-Risiken auf Systemebene zu erkennen. Um den Einsatz des STPA-SwISs-Verfahrens zu erforschen und zu bewer- ten, wurden ein Pilotprojekt sowie zwei Fallstudien in der Industrie durchge- führt, beide an Software-Systemen für die Automobilindustrie. Die Pilotstudie wurde während der Entwicklung des Prototyps eines Software-Simulators des 7 Abstandsregeltempomaten (ACC) mit Stop-and-Go-Funktion mit einem Lego- Mindstorms-EV3-Roboter durchgeführt, die erste Fallstudie in der Industrie am Abstandsregeltempomaten (ACC) bei der BMW Group. Die zweite Fallstudie untersuchte das vollautomatische Fahrsystem (AD) selbstfahrender Fahrzeuge bei Continental. Die Ergebnisse sprechen für die Leistungsfähigkeit von STPA SwISs und zeigen, dass das Verfahren skalierbar und auf Systeme in der Praxis anwendbar ist. Ferner wurde eine Plattform namens XSTAMPP verwendet, die die Anwen- dung von STAMP-Methoden und von STPA SwISs unterstützt und Software- und Sicherheitstechnikern bei der Identifizierung von Sicherheitsanforderun- gen hilft, indem sie deren Entwürfe und Implementierungen anhand der von STPA generierten Sicherheitsanforderungen unter Einsatz von Model-Checking- Werkzeugen überprüft und auf Basis der STPA-Ergebnisse automatisch Testfälle generiert. Unsere Schlussfolgerung lautet, dass die Kombination der Analyse auf STPA- Basis mit den sicherheitstechnischen Software-Verifikationsverfahren eine zweck- mäßige und effektive Möglichkeit bietet, Software-Risiken zu erkennen und Software-Qualität zu gewährleisten. Der Automatisierungsgrad verbesserte den vorgeschlagenen Ansatz, denn dadurch war es Software-und Sicherheitstech- nikern möglich, die Sicherheitsanalyse, die Software-Sicherheitsverifikation sowie Softwaretests mit demselben umfassenden Verfahren durchzuführen. Darüber hinaus eröffnet diese Arbeit interessante Möglichkeiten in der Software- Sicherheitstechnik. 8 A First of all, I would like to express my sincerest gratitude to my main advisor Prof. Dr. Stefan Wagner for giving me this opportunity to work with his group and under his guidance, motivation, collaboration, precious support, suggestions and ideas for making this dissertation possible. I appreciate his continuous motivation and support. I would also like to express my deepest gratitude to my co-advisor Prof. Dr. Nancy Leveson for developing the STAMP model, making a safer world, believing in me, giving me the opportunity to work under her supervision, and accepting my works to be presented at the annual STAMP MIT meetings. Without her feedback, comments and suggestions throughout my work, this dissertation would not have been possible. My warm thanks go to the researcher at MIT, Dr. John Thomas for his feedback, suggestions and support. I am grateful to the current researchers and employees in the software engi- neering group that I collaborate with them in daily work, teaching courses and academic publications. Ivan Bogicevic, Jan-Peter Ostberg, Jasmin Ramadani, Daniel Kulesz, Kai Mindermann, Erica Weilemann, Wolfgang Fechner, Kornelia Kuhle, Horst Vöhringer, Verena Käfer, Yang Wang, Rainer Niedermayr and Daniel Graziotin. Sebastian Vöst was especially helpful in providing me the opportunity to evaluate my dissertation at BMW Group, Munich. Special thanks to the researchers and employees at Continental, Regensburg
Load more

Recommended publications
  • Standardizing Functional Safety Assessments for Off-The-Shelf Instrumentation and Controls
    University of Tennessee, Knoxville TRACE: Tennessee Research and Creative Exchange Masters Theses Graduate School 5-2016 STANDARDIZING FUNCTIONAL SAFETY ASSESSMENTS FOR OFF-THE-SHELF INSTRUMENTATION AND CONTROLS Andrew Michael Nack University of Tennessee - Knoxville, [email protected] Follow this and additional works at: https://trace.tennessee.edu/utk_gradthes Part of the Other Computer Engineering Commons, and the Systems Engineering Commons Recommended Citation Nack, Andrew Michael, "STANDARDIZING FUNCTIONAL SAFETY ASSESSMENTS FOR OFF-THE-SHELF INSTRUMENTATION AND CONTROLS. " Master's Thesis, University of Tennessee, 2016. https://trace.tennessee.edu/utk_gradthes/3793 This Thesis is brought to you for free and open access by the Graduate School at TRACE: Tennessee Research and Creative Exchange. It has been accepted for inclusion in Masters Theses by an authorized administrator of TRACE: Tennessee Research and Creative Exchange. For more information, please contact [email protected]. To the Graduate Council: I am submitting herewith a thesis written by Andrew Michael Nack entitled "STANDARDIZING FUNCTIONAL SAFETY ASSESSMENTS FOR OFF-THE-SHELF INSTRUMENTATION AND CONTROLS." I have examined the final electronic copy of this thesis for form and content and recommend that it be accepted in partial fulfillment of the equirr ements for the degree of Master of Science, with a major in Computer Engineering. Gregory D. Peterson, Major Professor We have read this thesis and recommend its acceptance: Qing C. Cao, Mingzhou Jin Accepted for the Council: Carolyn R. Hodges Vice Provost and Dean of the Graduate School (Original signatures are on file with official studentecor r ds.) STANDARDIZING FUNCTIONAL SAFETY ASSESSMENTS FOR OFF-THE-SHELF INSTRUMENTATION AND CONTROLS A Thesis Presented for the Master of Science Degree The University of Tennessee, Knoxville Andrew Michael Nack May 2016 Copyright © 2016 by Andrew Michael Nack All rights reserved.
    [Show full text]
  • System Safety Engineering: Back to the Future
    System Safety Engineering: Back To The Future Nancy G. Leveson Aeronautics and Astronautics Massachusetts Institute of Technology c Copyright by the author June 2002. All rights reserved. Copying without fee is permitted provided that the copies are not made or distributed for direct commercial advantage and provided that credit to the source is given. Abstracting with credit is permitted. i We pretend that technology, our technology, is something of a life force, a will, and a thrust of its own, on which we can blame all, with which we can explain all, and in the end by means of which we can excuse ourselves. — T. Cuyler Young ManinNature DEDICATION: To all the great engineers who taught me system safety engineering, particularly Grady Lee who believed in me, and to C.O. Miller who started us all down this path. Also to Jens Rasmussen, whose pioneering work in Europe on applying systems thinking to engineering for safety, in parallel with the system safety movement in the United States, started a revolution. ACKNOWLEDGEMENT: The research that resulted in this book was partially supported by research grants from the NSF ITR program, the NASA Ames Design For Safety (Engineering for Complex Systems) program, the NASA Human-Centered Computing, and the NASA Langley System Archi- tecture Program (Dave Eckhart). program. Preface I began my adventure in system safety after completing graduate studies in computer science and joining the faculty of a computer science department. In the first week at my new job, I received a call from Marion Moon, a system safety engineer at what was then Ground Systems Division of Hughes Aircraft Company.
    [Show full text]
  • Manuscript Instructions/Template
    INCOSE Working Group Addresses System and Software Interfaces Sarah Sheard, Ph.D. Rita Creel CMU Software Engineering Institute CMU Software Engineering Institute (412) 268-7612 (703) 247-1378 [email protected] [email protected] John Cadigan Joseph Marvin Prime Solutions Group, Inc. Prime Solutions Group, Inc. (623) 853-0829 (623) 853-0829 [email protected] [email protected] Leung Chim Michael E. Pafford Defence Science & Technology Group Johns Hopkins University +61 (0) 8 7389 7908 (301) 935-5280 [email protected] [email protected] Copyright © 2018 by the authors. Published and used by INCOSE with permission. Abstract. In the 21st century, when any sophisticated system has significant software content, it is increasingly critical to articulate and improve the interface between systems engineering and software engineering, i.e., the relationships between systems and software engineering technical and management processes, products, tools, and outcomes. Although systems engineers and software engineers perform similar activities and use similar processes, their primary responsibilities and concerns differ. Systems engineers focus on the global aspects of a system. Their responsibilities span the lifecycle and involve ensuring the various elements of a system—e.g., hardware, software, firmware, engineering environments, and operational environments—work together to deliver capability. Software engineers also have responsibilities that span the lifecycle, but their focus is on activities to ensure the software satisfies software-relevant system requirements and constraints. Software engineers must maintain sufficient knowledge of the non-software elements of the systems that will execute their software, as well as the systems their software must interface with.
    [Show full text]
  • Infrastructure (Resilience-Oriented) Modelling Language: I®ML
    Infrastructure (Resilience-oriented) Modelling Language: I®ML A proposal for modelling infrastructures and their interconnections Andrés Silva, Roberto Filippini EUR 24727 EN - 2011 The mission of the JRC-IPSC is to provide research results and to support EU policy-makers in their effort towards global security and towards protection of European citizens from accidents, deliberate attacks, fraud and illegal actions against EU policies. European Commission Joint Research Centre Institute for the Protection and Security of the Citizen Contact information Address: TP 210, EC JRC Ispra, Ispra (Va) Italy E-mail: [email protected] Tel.: +39 0332 789936 Fax: http://ipsc.jrc.ec.europa.eu/ http://www.jrc.ec.europa.eu/ Legal Notice Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use which might be made of this publication. Europe Direct is a service to help you find answers to your questions about the European Union Freephone number (*): 00 800 6 7 8 9 10 11 (*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls may be billed. A great deal of additional information on the European Union is available on the Internet. It can be accessed through the Europa server http://europa.eu/ JRC 63302 EUR 24727 EN ISBN 978-92-79-19324-8 ISSN 1018-5593 doi:10.2788/54708 Luxembourg: Publications Office of the European Union © European Union, 2011 Reproduction is authorised provided the source is acknowledged Printed in Italy Infrastructure (Resilience-oriented) Modelling Language: I®ML A proposal for modelling infrastructures and their connections Andrés Silva1 Roberto Filippini Universidad Politécnica de Madrid JRC of the European Commission Abstract The modelling of critical infrastructures (CIs) is an important issue that needs to be properly addressed, for several reasons.
    [Show full text]
  • IS2018 Book of Abstract
    th Annual INCOSE 28 international symposium Washington, DC, USA July 7 - 12, 2018 Delivering Systems in the Age of Globalization Status as of May 15 th , 2018 Book of Abstract Table of contents keynotes ............................................................................................................................................................................... p. 7 keynotes#Keynote#2: The Big Shift: Innovation and Systems Engineering ................................................................ p. 7 Papers .................................................................................................................................................................................. p. 8 Papers#107: A Framework for Concept and its Testing on Patents ............................................................................ p. 8 Papers#75: A Framework for Testability Analysis from System Architecture Perspective .......................................... p. 9 Papers#128: A Framework for Understanding Systems Principles and Methods .......................................................p. 10 Papers#31: A fresh look at Systems Engineering - what is it, how should it work? .....................................................p. 11 Papers#35: A Hybrid Liver-Candidate Transportation System to Improve Accessibility and Extend Organ Life in L ..p. 12 Papers#55: A Novel “Resilience Viewpoint” to aid in Engineering Resilience in Systems of Systems (SoS) .............p. 13 Papers#97: A successful use of systems approaches in
    [Show full text]
  • A Software Safety Process for Safety-Critical Advanced Automotive Systems
    PROCEEDINGS of the 21st INTERNATIONAL SYSTEM SAFETY CONFERENCE - 2003 A Software Safety Process for Safety-Critical Advanced Automotive Systems Barbara J. Czerny, Ph.D.; Joseph G. D’Ambrosio, Ph.D., PE; Paravila O. Jacob, Ph.D.; Brian T. Murray, Ph.D.; Padma Sundaram; Delphi, Corp.; Brighton, Michigan Keywords: SW Safety, Safety-Critical Advanced Automotive Systems, By-Wire Systems Abstract A new generation of advanced automotive systems are being implemented to enhance vehicle safety, performance, and comfort. As these new, often complex systems are added, system safety programs are employed to help eliminate potential hazards. A key component of these advanced automotive systems is software. Software itself cannot fail or wear out, but its complexity coupled with its interactions with the system and the environment can directly and indirectly lead to potential system hazards. As such, software safety cannot be considered apart from system safety, but the unique aspects of software warrant unique development and analysis methods. In this paper we describe the main elements of a software safety process for safety-critical advanced automotive systems. We describe how this proposed process may be integrated with an established system safety process for by-wire automotive systems, and how it may be integrated with an established software development process. Introduction Expanding demand for further improvements in vehicle safety, performance, fuel economy and low emissions has led to a rapid and accelerating increase in the amount and sophistication of vehicle electronics. Emerging vehicle electronics systems are programmable, with a substantial software component, and are highly complex distributed systems. Increasingly, they are receiving driver inputs and directly controlling essential vehicle functions like braking and steering.
    [Show full text]
  • Model for Safety Case Modeling and Documentation
    SAFE – an ITEA2 project D3.1.3 Contract number: ITEA2 – 10039 Safe Automotive soFtware architEcture (SAFE) ITEA Roadmap application domains: Major: Services, Systems & Software Creation Minor: Society ITEA Roadmap technology categories: Major: Systems Engineering & Software Engineering Minor 1: Engineering Process Support WP3 Deliverable D3.1.3: Proposal for extension of Meta- model for safety case modeling and documentation Due date of deliverable: 31/03/2013 Actual submission date: 28/03/2013 Start date of the project: 01/07/2011 Duration: 36 months Project coordinator name: Stefan Voget Organization name of lead contractor for this deliverable: fortiss Editor: Maged Khalil (fortiss GmbH) Contributors: Maged Khalil (fortiss GmbH), Eduard Metzker (Vector Informatik GmbH) Reviewers: Eduard Metzker (Vector Informatik GmbH) 2013 The SAFE Consortium 1 (50) SAFE – an ITEA2 project D3.1.3 Revision chart and history log Version Date Reason 0.1 2012-09-19 Initialization of document 0.2 2012-10-18 Input to Introduction Section 4 0.3 2012-11-07 Input to Overview on ISO Section 5 0.4 2012-11-16 Input to EAST-ADL Overview Section 7 0.5 2012-11-16 Input to Overview on ISO Section 5 0.6 2012-11-23 Input to Methodology Section 6 0.7 2012-11-29 Further Input to Methodology Section 6 0.8 2012-12-20 Editing and Input in various sections 0.9 2013-02-13 Input to SAFE Meta-model Contribution Section 8 0.10 2013-02-15 Introduction of Patterns Section 6.3 0.11 2013-03-08 Editing and Input in various sections 0.12 2013-03-24 First version ready for review 0.13 2013-03-26 Incorporation of first review comments 0.14 2013-03-27 Updated version reviewed.
    [Show full text]
  • Systems Theoretic Process Analysis (STPA): a Bibliometric and Patents Analysis
    ORIGINAL ARTICLE Systems Theoretic Process Analysis (STPA): a bibliometric and patents analysis Modelo teórico - Sistêmico de Análise de Processos (STPA): uma análise bibliométrica e de patentes Sarah Francisca De Souza Borges1 , Marco Antônio Fontoura De Albuquerque1 , Moacyr Machado Cardoso Junior1 , Mischel Carmen Neyra Belderrain1 , Luís Eduardo Loures Da Costa1 1Área de Gestão Tecnológica do Programa de Ciências e Tecnologias Espaciais – CTE/G, Instituto Tecnológico De Aeronáutica - ITA, São José dos Campos, SP, Brasil. E-mail: [email protected]; [email protected]; [email protected]; [email protected]; [email protected] How to cite: Borges, S. F. S., Albuquerque, M. A. F., Cardoso Junior, M. M., Belderrain, M. C. N. & Costa, L. E. L. (2021). Systems Theoretic Process Analysis (STPA): a bibliometric and patents analysis. Gestão & Produção, 28(2), e5073. https://doi.org/10.1590/1806-9649-2020v28e5073 Abstract: The Systemic Theoretical Process Analysis (STPA) model is used for hazard analysis and accident prevention, based on systemic thinking and the identification of causal scenarios, created by Professor Nancy Leveson of the Institute of Technology of Massachusetts (MIT). The purpose of this article is to perform a bibliometric and patent analysis of the STPA model. Since bibliometry is an important tool in the analysis of scientific production, this method is used as a descriptive statistic, for the purposes of this study, the concepts of Goffman's Epidemic Theory were highlighted, under a mainly qualitative analysis, for a study of decline and ascent scientific method. For the bibliometric analysis, the main page of Professor Nancy Leveson was used in MIT's Web site, besides the Web of Science, Mendeley, ResearchGate, Village of Engineering and Scientific Electronic Library Online (SciELO).
    [Show full text]
  • Manuscript Instructions/Template
    Safety Analysis in Early Concept Development and Requirements Generation1 Nancy G. Leveson Massachusetts Institute of Technology 77 Massachusetts Ave, Cambridge MA 02139 617-258-0505 http://[email protected] [email protected] Copyright © 2018 by Nancy G. Leveson. Published and used by INCOSE with permission. Abstract. This paper shows how a new hazard analysis technique, STPA (System Theoretic Process Analysis), can be used to generate high-level safety requirements early in the concept development phase that can then assist in the design of the system architecture. These general, system-level requirements can be refined using STPA as decisions are made. The process goes hand-in-hand with design and the rest of the lifecycle as STPA can be used to provide information to assist in decision- making throughout the development and even operations phases. STPA also fits into a model-based engineering process as it works on a model of the system (which is also refined as design decisions are made) although that model is different than the architectural models usually proposed for model- based system engineering today. The process promotes traceability throughout the development process so decisions and designs can be changed with minimum requirements for redoing previous analyses. Finally, while this paper describes the approach with respect to safety, it can be applied to any emergent system property. Early Concept Exploration and Development Early concept development, shown at the top left of the V-model (Figure 1), is the first step in the usual system engineering process. While the label may differ in variants of the model, this stage includes such activities as stakeholder and user analysis (needs analysis), customer requirements generation, regulatory requirements review, feasibility studies, concept and tradespace exploration, and establishment of criteria for evaluation of the evolving and final design.
    [Show full text]
  • Download from an Unknown Website Cannot Be Said to Be ‘Safe’ Just Because It Happens Not to Harbor a Virus
    A direct path to dependable software The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters. Citation Jackson, Daniel. “A direct path to dependable software.” Commun. ACM 52.4 (2009): 78-88. As Published http://dx.doi.org/10.1145/1498765.1498787 Publisher Association for Computing Machinery Version Author's final manuscript Citable link http://hdl.handle.net/1721.1/51683 Terms of Use Attribution-Noncommercial-Share Alike 3.0 Unported Detailed Terms http://creativecommons.org/licenses/by-nc-sa/3.0/ A Direct Path To Dependable Software Daniel Jackson Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology What would it take to make software more dependable? Until now, most approaches have been indirect: some practices – processes, tools or techniques – are used that are believed to yield dependable software, and the argument for dependability rests on the extent to which the developers have adhered to them. This article argues instead that developers should produce direct evidence that the software satisfies its dependability claims. The potential advantages of this approach are greater credibility (since the ar- gument is not contingent on the effectiveness of the practices) and reduced cost (since development resources can be focused where they have the most impact). 1 Why We Need Better Evidence Is a system that never fails dependable? Not necessarily. A dependable system is one you can depend on – that is, in which you can place your reliance or trust. A rational person or organization only does this with evidence that the system’s benefits outweigh its risks.
    [Show full text]
  • Appendix J Software Safety
    FAA System Safety Handbook, Appendix J: Software Safety December 30, 2000 Appendix J Software Safety SOFTWARE SAFETY...............................................................................................................................1 J.0 SOFTWARE SAFETY DURING LIFE CYCLE PHASES................................................................2 J-1 FAA System Safety Handbook, Appendix J: Software Safety December 30, 2000 J.0 Software Safety During Life Cycle Phases The safety process should support a structured program life cycle model that incorporates both the system design and engineering process and the software acquisition process. Prominent software life cycle models include the waterfall and spiral methodologies. Although different models may carry different lifecycle emphasis, the adopted model should not affect the safety process itself. For discussion purposes only, this enclosure adopts a waterfall model (subject to IEEE/IEA Standard for Information Technology-software life cycle processes No. 12207.) For brevity, only the development phase of the Standard is addressed in terms of the relationship to software safety activities. J.1 Safety Critical Software Development A structured development environment and an organization using state-of-the-art methods are prerequisites to developing dependable safety critical software. The following requirements and guidelines are intended to carry out the cardinal safety rule and its corollary that no single event or action shall be allowed to initiate a potentially hazardous event. The system, upon detection of an unsafe condition or command, shall inhibit the potentially hazardous event sequence and originate procedures/functions to bring the system to a predetermined “safe” state. The purpose of this section is to describe the software safety activities that should be incorporated into the software development phases of project development. The software safety information that should be included in the documents produced during these phases is also discussed.
    [Show full text]
  • Machine Learning Testing: Survey, Landscapes and Horizons
    1 Machine Learning Testing: Survey, Landscapes and Horizons Jie M. Zhang*, Mark Harman, Lei Ma, Yang Liu Abstract—This paper provides a comprehensive survey of techniques for testing machine learning systems; Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing. Index Terms—machine learning, software testing, deep neural network, F 1 INTRODUCTION The prevalent applications of machine learning arouse inherently follows a data-driven programming paradigm, natural concerns about trustworthiness. Safety-critical ap- where the decision logic is obtained via a training procedure plications such as self-driving systems [1], [2] and medical from training data under the machine learning algorithm’s treatments [3], increase the importance of behaviour relating architecture [8]. The model’s behaviour may evolve over to correctness, robustness, privacy, efficiency and fairness. time, in response to the frequent provision of new data [8]. Software testing refers to any activity that aims to detect While this is also true of traditional software systems, the the differences between existing and required behaviour [4]. core underlying behaviour of a traditional system does not With the recent rapid rise in interest and activity, testing typically change in response to new data, in the way that a has been demonstrated to be an effective way to expose machine learning system can.
    [Show full text]