Institute for Software Technology tugraz Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria Summer Term 2019 B. Aichernig Quality Assurance in Software Development 1 / 20 Institute for Software Technology tugraz Agenda 1 Organisation 2 Contents 3 Quality of SW B. Aichernig Quality Assurance in Software Development 2 / 20 Institute for Software Technology tugraz Quality Assurance I Lecture: Tue 11:15 – 12:00 (i12) I Exercise: Tue 12:00 – 12:45 (i12) I Optional: Fri 14:15–15:00 (i12) I Registration via tugonline – 8.3. strict! I Student assistants: I Dominik Augustin I Jorrit Stramer B. Aichernig Quality Assurance in Software Development 3 / 20 Institute for Software Technology tugraz Additional Resources I WWW: http://www.ist.tugraz.at/qs.html I Newsgroup: tu-graz.lv.qs I Email: Subject: [QS19] I [email protected] B. Aichernig Quality Assurance in Software Development 4 / 20 Institute for Software Technology tugraz Marking I Marking if registered on 8.3. I Written exam (Klausur) : 25.6. (50 %). I Three practical tasks (groups of 3): I Task 1: 12.3.–2.4. (16 %) I Task 2: 2.4.–7.5. (16 %) I Task 3: 7.5–4.6. (18 %) I Exam and exercises count each 50% I Positive if > 50% total points and I > 30% on exam and I > 30% on exercises in total I Prüfungsabbruch: the registration to a group for the exercises counts as item of work. B. Aichernig Quality Assurance in Software Development 5 / 20 Institute for Software Technology tugraz Marking (cont.) Grading key: 50,01% - 62,50%: genügend 62,51% - 75,00%: befriedigend 75,01% - 87,50%: gut 87,51% - 100,0%: sehr gut Nachklausur: I Date: 9.7., 11:00 I After excused absence at exam I e.g. due to illness with sick certificate (ärztlicher Bestätigung) I work is no excuse (take leave!) I For all negative who have been excused at exam! B. Aichernig Quality Assurance in Software Development 6 / 20 Institute for Software Technology tugraz Plan Datum VO UE Tue 5.3. Organisation, Introduction Tue 12.3. Test Coverage Ausgabe Task 1 Tue 19.3. Symbolic and Concolic Execution Tue 26.3. Specifications 1 Tue 2.4. Tool Demo Abgabe Task 1, Ausgabe Task 2 Tue 9.4. Specifications 2 Osterferien Tue 30.4. Property- & Model-based Testing Tue 7.5. Tool Demo Abgabe Task 2, Ausgabe Task 3 Tue 14.5. Tue 21.5. Input-Output Conformance Tue 28.5. (Model-based) Mutation Testing Tue 4.6. Equivalence Classes and Test- Abgabe Task 3 Sequence Generation Di nach Pfingsten Tue 18.6. Decision Tables, White-box Test- ing, Reviews Tue 25.6. Written exam B. Aichernig Quality Assurance in Software Development 7 / 20 Institute for Software Technology tugraz What we will not cover? Quality management: I Process management, e.g. Scrum I Process improvement, e.g. ISO 9000 I Test management, e.g. IBM Rational Quality Manager I Version control, e.g. svn, git I ... B. Aichernig Quality Assurance in Software Development 8 / 20 Institute for Software Technology tugraz General Aims of the Course I Getting familiar with quality assurance techniques I Raising the awareness for quality of software I Practising fault-based thinking I Being able to distinguish process-oriented and technical quality assurance I Challenging common quality criteria I Realising the relations between the foundations of software engineering and quality assurance. I Learning the use and foundations of advanced testing tools. B. Aichernig Quality Assurance in Software Development 9 / 20 Institute for Software Technology tugraz Specific Learning Targets and Skills I Getting familiar with a professional IDE I Writing good tests :( I Generating good tests :) I Specifying test oracles I Using and understanding state-of-the-art test case generators I Test coverage metrics "Don’t write test cases, generate them!" (John Hughes) B. Aichernig Quality Assurance in Software Development 10 / 20 Institute for Software Technology tugraz Specific Learning Targets and Skills I Getting familiar with a professional IDE I Writing good tests :( I Generating good tests :) I Specifying test oracles I Using and understanding state-of-the-art test case generators I Test coverage metrics "Don’t write test cases, generate them!" (John Hughes) B. Aichernig Quality Assurance in Software Development 10 / 20 Institute for Software Technology tugraz Course Contents I Introduction and motivation I Challenges of software testing I Coverage: Measuring the quality of test cases I Testing with Concolic Execution I Contracts as test oracles I Model-based Testing I Mutation Testing I Reviews I (Proof-based Development) I Tools (exercise): I MS Visual Studio Enterprise 2015 I MS IntelliTest (Pex) I .NET Code Contracts I FsCheck B. Aichernig Quality Assurance in Software Development 11 / 20 Institute for Software Technology tugraz Course Contents I Introduction and motivation I Challenges of software testing I Coverage: Measuring the quality of test cases I Testing with Concolic Execution I Contracts as test oracles I Model-based Testing I Mutation Testing I Reviews I (Proof-based Development) I Tools (exercise): I MS Visual Studio Enterprise 2015 I MS IntelliTest (Pex) I .NET Code Contracts I FsCheck B. Aichernig Quality Assurance in Software Development 11 / 20 Institute for Software Technology tugraz Quality? I Today, often a sufficient level of I correctness I efficiency I costs cannot be guaranteed. I Vision: Development methods for SW with warranty. B. Aichernig Quality Assurance in Software Development 12 / 20 Institute for Software Technology tugraz Quality? I Today, often a sufficient level of I correctness I efficiency I costs cannot be guaranteed. I Vision: Development methods for SW with warranty. B. Aichernig Quality Assurance in Software Development 12 / 20 Marketing with App Institute for Software Technology tugraz Limited Warranties I Cisco: “Cisco does not warrant that the Software will operate uninterrupted or error-free or that all errors will be corrected. In addition, Cisco does not warrant that the Software or any equipment, system or network on which the Software is used will be free of vulnerability to intrusion or attack." Limited Warranty and Disclaimer, (c) Disclaimer; effective: May 22, 2017: (https://www.cisco.com/c/en/us/about/legal/ cloud-and-software/end_user_license_agreement.html) I Skydrop Sprinkler Controller:“This warranty does not cover consumable parts, including batteries, unless damage is due to defects in materials or workmanship of the Product, or software (even if packaged or sold with the product).” (https://www.skydrop.com/warranty/) B. Aichernig Quality Assurance in Software Development 13 / 20 Institute for Software Technology tugraz Limited Warranties I Cisco: “Cisco does not warrant that the Software will operate uninterrupted or error-free or that all errors will be corrected. In addition, Cisco does not warrant that the Software or any equipment, system or network on which the Software is used will be free of vulnerability to intrusion or attack." Limited Warranty and Disclaimer, (c) Disclaimer; effective: May 22, 2017: (https://www.cisco.com/c/en/us/about/legal/ cloud-and-software/end_user_license_agreement.html) I Skydrop Sprinkler Controller:“This warranty does not cover consumable parts, including batteries, unless damage is due to defects in materials or workmanship of the Product, or software (even if packaged or sold with the product).” (https://www.skydrop.com/warranty/) Marketing with App B. Aichernig Quality Assurance in Software Development 13 / 20 Institute for Software Technology tugraz Bugs Part of engineering jargon for many decades: I Moth trapped in relay of Mark II (Hopper 1946) I Little faults and difficulties (Edison 1878): I Software bugs Relay #70 Panel F (moth) in relay. First actual case of bug being found. B. Aichernig Quality Assurance in Software Development 14 / 20 Institute for Software Technology tugraz Bugs Part of engineering jargon for many decades: I Moth trapped in relay of Mark II (Hopper 1946) I Little faults and difficulties (Edison 1878): I Software bugs Relay #70 Panel F (moth) in relay. First actual case of bug being found. B. Aichernig Quality Assurance in Software Development 14 / 20 Institute for Software Technology tugraz Bugs Definition A software bug is the common term Part of engineering jargon for many used to describe an decades: error, flaw, mistake, failure, or Moth trapped in relay of Mark II I I fault in a computer program or (Hopper 1946) system Little faults and difficulties I that produces an incorrect or (Edison 1878): I unexpected result, I Software bugs I or causes it to behave in unintended ways. (Wikipedia 2012) B. Aichernig Quality Assurance in Software Development 14 / 20 Institute for Software Technology tugraz Some Bugs Become Famous! I Ariane 5 test flight (1996) I out of control due to software failure I controlled destruction! I Loss of I money and time I satellites I research (TU Graz) I Dijkstra (EWD 1036): I call it error, not bug I a programmer created it B. Aichernig Quality Assurance in Software Development 15 / 20 Institute for Software Technology tugraz Some Bugs Become Famous! I Ariane 5 test flight (1996) I out of control due to software failure I controlled destruction! I Loss of I money and time I satellites I research (TU Graz) I Dijkstra (EWD 1036): I call it error, not bug I a programmer created it B. Aichernig Quality Assurance in Software Development 15 / 20 Algorithm was proven correct! I Programming Pearls [Bentley86, Bentley00] I assuming infinite integers :( Institute for Software Technology tugraz Some Bugs Hide for a Long Time! Binary search bug in Java 1 public static I JDK 1.5 library (2006) 2 int binarySearch(int[] a,int key) 3 { I out of boundary access of 4 int low = 0; large arrays 5 int high = a.length - 1; 6 7 while (low <= high) { I due to integer overflow 8 int mid = (low + high) / 2; I 9 years undetected 9 int midVal = a[mid]; 10 11 if (midVal < key) 12 low = mid + 1; 13 else if (midVal > key) 14 high = mid - 1; 15 else 16 return mid;// key found 17 } 18 return -(low + 1);// key not found 19 } “Beware of bugs in the above code; I have only proved it correct, not tried it.” [Knuth77] B.