Overview of Software Reliability Engineering
Total Page:16
File Type:pdf, Size:1020Kb
SENG 637 Dependability, Reliability & Testing of Software Systems Chap ter 1: OiOverview Department of Electrical & Computer Engineering, University of Calgary B.H. Far ([email protected]) http://www.enel.ucalgary.ca/People/far/Lectures/SENG637/ [email protected] 1 Contents Shorter version: How to avoid these? [email protected] 2 Contents Longer version: What is this course about? What factors affect software quality? What is software reliability? What is software reliability engineeringengineering?? What is software reliability engineering process? [email protected] 3 About This Course … The topics discussed include: Concepts and relationships Analytical models and supporting tools Techniques for software reliability improvement, including: Fault avoidance, fault elimination, fault tolerance Error detection and repair FilFailure detect ion and retracti on Risk management [email protected] 4 Terminology & Scope Failures Treats Faults Errors Availability Reliability Safety Attributes Dependability Confidentiality Integrity Maataabtyintainability The ability of a system to deliver Fault prevention Means service that can Fault tolerance FlFault remova l justifiably be Fault forecasting trusted. Reliability Block Diagram Models Fault Tree model Reliability Graph [email protected] 5 Software Reliability Software Reliability Concept Model Process Reliability Single Multiple SRE Availability Failure Failure Process Failure rate Model Model MTTF Failure density Etc. Reliability Growth Model [email protected] 6 Software Testing Software Testing Techniques Process Other: Certification Reliability SRE Black-box Test Growth Process WhiteWhite--boxbox Test Alpha Beta BigBig--bangbang Stress Etc. [email protected] 7 Question to Ask Do I really need to take this course? AddAnswer depend on you! Take this course if you want to avoid these in your career as a software designer , tester and quality controller: Bug Fix [email protected] 8 At The End … What is software reliability engineering (SRE)? Why SRE is important? How does it affect software quality? What are the main factors that affect the reliability of software? Is SRE equivalent to software testing? What makes SRE different from software testing? How can one determine how often will the software fail? How can one determine the current reliability of the software under development? How can one determine whether the ppgroduct is reliable enough to be released? Can SRE methodology be applied to the current ways of software development such as component-based and agile development? What are challenges and difficulties of applying SRE? What are current research topics of SRE? [email protected] 9 Chapter 1 Section 1 FSftFrom Software Quality lit t o Software Reliability Engineering [email protected] 10 What is Quality? QQyppuality popular view: – Something “good” but not quantifiable – Something luxury and classy QliQuality prof essi onal li view: – Conformance to requirement (Crosby, 1979) – Fitness for use (Juran, 1970) SENG521 [email protected] 11 Quality: Various Views AthtiAesthetic DlDeveloper View View Customer View [email protected] 12 What is Software Quality? Conformance to requirement The requirements are clearly stated and the product must conform to it What is Specified? Any deviation from the requirements is regarddded as a dfdefect What A good quality product contains fewer defects SW Fitness for use Does? What user Fit to user expectations: meet user’s needs Needs? A good quality product provides better user satisfaction Both Dependable computing system SENG521 [email protected] 13 Definition: Software Quality ISO 8402 definition of QUALITY: The totality of features and characteristics of a product or a service that bear on its ability to satisfy stated or implied needs Reliability and Maintainability are two majjpor components of QyQuality SENG521 [email protected] 14 Quality Model: ISO 9126 Characteristics Attributes 1. Functionality Suitability Interoperability Accuracy Compliance Security 2. Reliability Maturity Recoverability Fault tolerance Crash frequency 3. Usab ilit y UUdesadabnderstandability Learnab ility Operab ility 4. Efficiency Time behaviour Resource behaviour 5. Maintainability Analyzability Stability Changeability Testa bility 6. Portability Adaptability Installability Conformance Replacability [email protected] 15 Quality Model – Structure SW Qua lity User oriented Quality Quality Quality Factor 1 Factor 2 ... Factor n Quality Quality Quality Quality Criterion Criterion Criterion ... Criterion Software oriented 1 2 3 m Measures [email protected] Example: Attribute Expansion Design by measurable Quality objective objectives: Incremental design is evaluated to check whether Availability the goal for each increment User friendliness was achieved. % of planned Days on job to System uptime learn task supplied By new system Worst: 95% Worst: 7 days Best: 99% Best: 1 day [email protected] 17 What Affects Quality? SENG521 [email protected] 18 What Affects Software Quality? Time: Meeting the project deadline. Quality Reaching the market at the right time. Cost: Cost Time Meeting the anticipated project costs. Quality (reliability): WkififWorking fine for th e designated period on People Technology the designated system. [email protected] 19 Quality vs. Project Costs Cost distribution for a typical software project Product Integration Design and test Release Programming Design Programming Testing What i s wrong with this p ic ture ? SENG521 [email protected] 20 Total Cost Distribution Maintenance is responsible for more that 60% of total cost for a typical software project Product Design Questions: 1) How to Programming build quality into a system? Maintenance 2) How to Integrat i on assess quality and test of a system? Developing better quality system will contribute to lowering maintenance costs SENG521 [email protected] 21 1) How to Build Quality into a System? Developing better quality systems requires: Establishing Quality Assurance (QA) programs Establishing Reliability Engineering (SRE) process SENG521 [email protected] 22 2) How to Assess Quality of a System? Relevant to both pre- release and post- Quality release Assess ment Pre-release: SRE, certification, standards ISO9001 Post-release: evaluati on, valid ati on, RAM SENG521 [email protected] 23 How Do We Assess Quality? AdAd--hochoc (trial and error) approach! Systematic approach SENG521 [email protected] 24 Pre-release Quality Software Facts: inspection and • About 20% of the software projects are canceled. (missed testing schedules, etc.) • About 84% of software projects Methods: are incomplete when released SRE (need patch, etc). • Almost all of the software projects Certification costs exceed initial estimations. Standards (cost overrun) ISO9001, 9126, 25000 SENG521 [email protected] 25 Fatal Software Examples Fatal software related incidents [Gage & McCormick 2004] Date Casualties Detail 2003 3 Software failure contributes of power outage across North-eastern U.S. and Canada. 2001 5 Panamanian cancer patients die following overdoses of radiation, determined by the use of faulty software. 2000 4 Crash of marine corps osprey tilt-rotor aircraft, partially blamed on software anomaly. 1997 225 Radar that could have prevented Korean jet crash hobbled by software problem. 1995 159 Amer ican a iliirlines j jtdet, descen ding itinto CliClCali, Columbi a crash es i itnto a mountain. A cause was that the software presented insufficient and conflicting information to the pilots, who got lost. 1991 28 Software problem prevents Patriot missile battery from picking up SCUD missile, which hits US Army barracks in Saudi Arabia. [email protected] 26 Cost of a Defect … Require- Functional System Field Design Coding ments Test Test Use 50 % Fault 40 % Origin 10 % Fault 50 % Detection 7 % 3 % 5 % 25 % 10 % 20 KDM Cost per Fault 12 KDM 1 KDM 1 KDM 1 KDM 6 KDM 1 KDM = 1,000 Deutsch Marks CMU. Software Engineering Institute [email protected] 27 A Central Question In spite of having many development methodologies, central questions are: 1. Can we remove all bugs before release? 2. How often will the software fail? [email protected] 28 Two Extremes Craftsman SE: fast, cheap, buggy Cleanroom SE: slow, expensive, zero defect Is there a middle solution? Craftsman YES! Cleanroom Software Is there Software Using Software Develop- a middle Develop- Reliability ment solution? ment Engineering (SRE) Process [email protected] 29 Can We Remove All Bugs? Size Faiiilure potential Failure removal rate Faiiilure Density [function points] [development] [at release] 1 1.85 95% 0.09 10 2.45 92% 0.20 100 3.68 90% 0.37 1000 5.00 85% 0.75 10000 7.60 78% 1.67 100000 9.55 75% 2.39 Av erage 5025.02 86% 0910.91 Defect potential and density are expressed in terms of defects per function point The answer is usually NO! [email protected] 30 What Can We Learn from Failures? Time Between Failure vs. ith Failure 1000 900 Does this plot make 800 any sense to you? 700 600 urs 500 oo H 400 300 200 100 0 1 112131415161718191 ith Failure Failure Time [email protected] 31 How to Handle Defects? Table below gives the time between failures for a software system: Failure no . 1 2 3 4 5 6 7 8 9 10 Time since last failure (hours) 64856911141619 What can we learn from this data? System reliability? Approximate number of bugs in the system? Approximate time to remove remaining bugs? [email protected] 32 What to Learn from Data? The inverses of the inter-failure times are the failure intensity (= failure per unit of time) data points Error no. 1 2 3 4 5 6 7 8 9 10 Time since last failure 6 4 8 5 6 9 11 14 16 19 (hours) Failure intensity 0.166 0.25 0.125 0.20 0.166 0.111 0.09 0.071 0.062 0.053 [email protected] 33 What to Learn from Data? Mean-time-to-failures MTTF (or average failure rate) MTTF = (6+4+8+5+6+9+11+14+16+19)/10 = 9. 8 hour System reliability for 1 hour of operation t 1 Ret eMTTF e9.8 0.90299 Fitting a straight line to the graph in (a) would show an x- intercept of about 15 . Using this as an estimate of the total number of original failures, we estimate that there are still five bugs in the software.