SENG 637 Dependability, Reliability & Testing of Software Systems SRE Dep loymen t (Chapter 10) Department of Electrical & Computer Engineering, University of Calgary B.H. Far （[email protected]） http://www. enel. ucalgary.ca/People/far/Lectures/SENG637/ [email protected] 1 Contents Quality in requirements phase Quality in design & implementation, testing & release phases SfSoftware Qua lity Assurance (SQA) an d So ftware Reliability Engineering (SRE) Quality, test and data plans Roles and responsibilities Sample quality and test plan Defect reporting procedure Best practices of SRE Quality in post-release and maintenance phase [email protected] 2 Quality vs. Project Costs Cost distribution for a typical software pro jec t Product Integration Design and test Programming 3 Total Cost Distribution Product Design Questions: Programming How to build quality into a system? Maintenance How to Integration andtd test assess quality of a system? Developing better quality system will contribute to lowering maintenance costs 4 Quality in Software Development Process Q. How to include quality concerns in the process? Architectural analysis Quality attributes Software Reliability Software Quality Method: ATAM, CBAM, etc. Engineering (SRE) Assurance (SQA) Requirement & Design & Test & Release AhittArchitecture IlImplemen ttitation Maintenance Software QQyuality Assessment Method: RAM, etc. [email protected] 5 Chapter 10 Section 1 SfSoftware QliQuality: Requirements and Architecture phase [email protected] 6 Quality Challenges Modern software systems are required to meet several quality attributes such as: modifiability, performance, security, interoperability, portability, reliability, etc. Questions for any particular system: What precisely do these quality attributes mean? Can a system bldbe analyzed to didetermine diddesired qua liilities? How soon can such an analysis occur? How do you know if the design is suitable without having to build the system first? SW Architecture Evaluation / Assessment! [email protected] 7 Evaluating SW Architecture Determining whether an architecture satisfies its requ iremen ts oft en i nvol ves: Being very explicit about what the requirements (functional & non-functional) are and how they are reflected in the architecture Understanding where one has to make trade-offs between different design alternatives Applying analysis wherever possible to determine the consequences of an architectural choice Mediating between desires of different stakeholders To achieve these goals an architectural evaltiluation process i s need dded [email protected] 8 SW Architecture Evaluation IfInforma l/dl / ad-hoc architec tura l eva luati on Pros? QQpuick and Cheap Cons? … and Dirty? Incomplete? Unreliable? … Unrepeatable? Poorly documented? [email protected] 9 SW Architecture Evaluation Are there better methods than ad-hoc evaluation? The answer is “YES”: SAAM (Software Architecture Analysis Method) Scenario-based evaluation ATAM (Architecture Tradeoff Analysis Method) Scenario-based evaluation with focus on trade -offs SACAM (Software Architecture Comparison Method) Business goal-driven comparison of architecture alternatives CBAM (Cost-Benefit Analysis Method) Focus on economic aspects etc. [email protected] 10 References Software Architecture Technology Initiative of the SEI: http://www.sei.cmu.edu/architecture/ ATAM: Method for Architecture Evaluation (2000), Rick Kazman, Mark Klein, Paul Clements, Technical Report, CMU/SEI-2000-TR-004. CBAM: M aki ng A rchi tecture D esi gn D eci si ons: A n Economic Approach (2002), Rick Kazman, Jai Asundi, Mark Klein, Technical Report, CMU/SEI- 2002-TR-035. nd Software Architecture in Practice, 2 ed., Len Bass, Paul Clements,,, Rick Kazman, Addison-Wesley, 2003. Evaluating Software Architectures: Methods and Case Studies, Paul Clements, Rick Kazman, Mark Klein, Addison-Wesley, 2001. [email protected] 11 Chapter 10 Section 2 SfSoftware QliDi&Quality: Design & Implementation, Testing & Release Phases [email protected] 12 What is Reliable Software? Reliable software products are those that run correctly and consistently, have fewer remaining defects, handle abnormal situation properly, and need less installation effort The remaining defects should not affect the normal behaviour and the use of the software, they will not do any destructive things to system and its hardware or software environment, and rarely be evident to the users DliDeveloping re liblftliable software requi res: Establishing Software Quality System (SQS) and Software Quality Assurance (SQA) programs Establishing Software Reliability Engineering (SRE) process [email protected] 13 Software Quality System (SQS) Goals: Bu ilding qualit y into the software from the beggginning Keeppging and tracking quality in the software throughout the software life cycle ThTechnol ogy John W. Horch: Practical Guide to Software Quality Management [email protected] 14 Software Quality Assurance (SQA) Software quality Assurance (SQA) is a planned and systematic approach to ensure that both software process and software product conform to the established standards, processes, and procedures. The goals of SQA are to improve software quality by monitoring both software and the development process to ensure full compliance with the established standards and procedures. Steps to establish an SQA program Get the top management’s agreement on its goal and support. Identify SQA issues, write SQA plan, establish standards and SQA functions, implement the SQA plan and evaluate SQA program. [email protected] 15 SRE: Process & Plans Requirement & Design & Test Architecture Implementation Define Necessary Reliability Develop Operational SRE Profile Proc PfTtPrepare for Test Apply Execute Failure Test Data time Quality Test Data Plan Plan Plan There may be many Test and Data (measurement) plans for various parts of the same project [email protected] 16 Defect Handling: Without & With SQS Defect reppg,g,orting, tracking, and closure p rocedure Defect reports DB SCN: software change notice STR: software trouble report John W. Horch: Practical Guide to Software Quality Management [email protected] 17 SRE: Who is Involved? Senior management Test coordinator (manager) Data coordinator (manager) Customer or user [email protected] 18 SRE: Management Concerns Perception and specification of a customer’s real needs. Translilation of specifi ifiication i nto a conf ormi ng d diesign. Maintaining conformity throughout the development processes. Product and sub-product demonstrations which provide convincing indications of the product and project having met their requirements. Ensuring that the tests and demonstrations are designed and controlled, so as to be both achievable and manageable. [email protected] 19 Roles & Responsibilities /1 Test Coordinator (Manager): Test coordinator is expected to ensure that every specific statement of intent in the product requirement, specification and design, is matched by a well designed (cost-effective, convincing, self-reporting, etc.) test, measurement or demonstration. Data Coordinator (Manager) : Data coordinator ensures that the physical and administrative structures fdfor data co llillection exi st and are d ocumented dih in the quali lilty plan, recei ves and validates the data during development, and through analysis and communication ensures that the meaning of the information is known to all, in time, for effective application. [email protected] 20 Roles & Responsibilities /2 Customer or User: Actively encouraging the making and following of detailed quality plans for the products and projects. Requiring access to previous quality plans and their recorddded outcomes bfbefore accept ing t he figures an d methods quoted in the new plan. Enquiring into the sources and validity of synthetics and formulae used in estimating and planning . Appointing appropriate personnel to provide authoritative responses to queries from the developer and a managed interface to the developer. Receiving and reviewing reports of significant audits, reviews, tests and demonstrations. Making any queries and objections in detail and in writing, at the earliest possible time. [email protected] 21 Quality Plans /1 The most promising mechanisms fiidiifor gaining and improving predictability and controllability of software qualities are quality Test plan and its subsidiary documents, including test plans Plan and data (measurement) plans. Quality The creation of the quality plan Plan can be instrumental in raising project effectiveness and in ppgpreventing expensive and time- Data consuming misunderstandings Plan during the project, and at release/acceptance time. [email protected] 22 Quality Plan /2 Quality plan and quality record, provide guidelines fitdtllithfllifor carrying out and controlling the followings: Requirement and specification management. Development processes . Documentation management. Design evaluation. Product testing. SRE related Data collection and interpretation. activities Acceptance and release processes. [email protected] 23 Quality Plan /3 Quality planning should be made at the very earliest point in a project, preferably before a final decision is made on feasibility, and before a software development contract is signed. Quality plan should be devised and agreed between all the concerned parties: senior management, software development management (both administrative and technical) , software development team, customers, and any involved general support functions such as resource