Quality Assurance in Software Development Qualitätssicherung in Der Softwareentwicklung

Home , Software testing

Institute for Software Technology tugraz

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung

A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig

Institute for Software Technology Graz University of Technology Austria

Summer Term 2019

B. Aichernig Quality Assurance in Software Development 1 / 20 Institute for Software Technology tugraz Agenda

1 Organisation

2 Contents

3 Quality of SW

B. Aichernig Quality Assurance in Software Development 2 / 20 Institute for Software Technology tugraz Quality Assurance

I Lecture: Tue 11:15 – 12:00 (i12)

I Exercise: Tue 12:00 – 12:45 (i12)

I Optional: Fri 14:15–15:00 (i12)

I Registration via tugonline – 8.3. strict! I Student assistants:

I Dominik Augustin

I Jorrit Stramer

B. Aichernig Quality Assurance in Software Development 3 / 20 Institute for Software Technology tugraz Additional Resources

I WWW: http://www.ist.tugraz.at/qs.html

I Newsgroup: tu-graz.lv.qs

I Email: Subject: [QS19]

I [email protected]

B. Aichernig Quality Assurance in Software Development 4 / 20 Institute for Software Technology tugraz Marking

I Marking if registered on 8.3.

I Written exam (Klausur) : 25.6. (50 %).

I Three practical tasks (groups of 3):

I Task 1: 12.3.–2.4. (16 %)

I Task 2: 2.4.–7.5. (16 %)

I Task 3: 7.5–4.6. (18 %)

I Exam and exercises count each 50%

I Positive if > 50% total points and

I > 30% on exam and

I > 30% on exercises in total

I Prüfungsabbruch: the registration to a group for the exercises counts as item of work.

B. Aichernig Quality Assurance in Software Development 5 / 20 Institute for Software Technology tugraz Marking (cont.)

Grading key: 50,01% - 62,50%: genügend 62,51% - 75,00%: befriedigend 75,01% - 87,50%: gut 87,51% - 100,0%: sehr gut

Nachklausur:

I Date: 9.7., 11:00

I After excused absence at exam

I e.g. due to illness with sick certiﬁcate (ärztlicher Bestätigung)

I work is no excuse (take leave!)

I For all negative who have been excused at exam!

B. Aichernig Quality Assurance in Software Development 6 / 20 Institute for Software Technology tugraz Plan

Datum VO UE Tue 5.3. Organisation, Introduction Tue 12.3. Test Coverage Ausgabe Task 1 Tue 19.3. Symbolic and Concolic Execution Tue 26.3. Specifications 1 Tue 2.4. Tool Demo Abgabe Task 1, Ausgabe Task 2 Tue 9.4. Specifications 2 Osterferien Tue 30.4. Property- & Model-based Testing Tue 7.5. Tool Demo Abgabe Task 2, Ausgabe Task 3 Tue 14.5. Tue 21.5. Input-Output Conformance Tue 28.5. (Model-based) Mutation Testing Tue 4.6. Equivalence Classes and Test- Abgabe Task 3 Sequence Generation Di nach Pfingsten Tue 18.6. Decision Tables, White-box Test- ing, Reviews Tue 25.6. Written exam

B. Aichernig Quality Assurance in Software Development 7 / 20 Institute for Software Technology tugraz What we will not cover?

Quality management:

I Process management, e.g. Scrum

I Process improvement, e.g. ISO 9000

I Test management, e.g. IBM Rational Quality Manager

I Version control, e.g. svn, git

I ...

B. Aichernig Quality Assurance in Software Development 8 / 20 Institute for Software Technology tugraz General Aims of the Course

I Getting familiar with quality assurance techniques

I Raising the awareness for quality of software

I Practising fault-based thinking

I Being able to distinguish process-oriented and technical quality assurance

I Challenging common quality criteria

I Realising the relations between the foundations of software engineering and quality assurance.

I Learning the use and foundations of advanced testing tools.

B. Aichernig Quality Assurance in Software Development 9 / 20 Institute for Software Technology tugraz Speciﬁc Learning Targets and Skills

I Getting familiar with a professional IDE

I Writing good tests :(

I Generating good tests :)

I Specifying test oracles

I Using and understanding state-of-the-art test case generators

I Test coverage metrics "Don’t write test cases, generate them!"

(John Hughes)

B. Aichernig Quality Assurance in Software Development 10 / 20 Institute for Software Technology tugraz Speciﬁc Learning Targets and Skills

I Getting familiar with a professional IDE

I Writing good tests :(

I Generating good tests :)

I Specifying test oracles

I Using and understanding state-of-the-art test case generators

I Test coverage metrics "Don’t write test cases, generate them!"

(John Hughes)

B. Aichernig Quality Assurance in Software Development 10 / 20 Institute for Software Technology tugraz Course Contents

I Introduction and motivation

I Challenges of software testing

I Coverage: Measuring the quality of test cases

I Testing with Concolic Execution

I Contracts as test oracles

I Model-based Testing

I Mutation Testing

I Reviews

I (Proof-based Development) I Tools (exercise):

I MS Visual Studio Enterprise 2015

I MS IntelliTest (Pex)

I .NET Code Contracts

I FsCheck

B. Aichernig Quality Assurance in Software Development 11 / 20 Institute for Software Technology tugraz Course Contents

I Introduction and motivation

I Challenges of software testing

I Coverage: Measuring the quality of test cases

I Testing with Concolic Execution

I Contracts as test oracles

I Model-based Testing

I Mutation Testing

I Reviews

I (Proof-based Development) I Tools (exercise):

I MS Visual Studio Enterprise 2015

I MS IntelliTest (Pex)

I .NET Code Contracts

I FsCheck

B. Aichernig Quality Assurance in Software Development 11 / 20 Institute for Software Technology tugraz Quality?

I Today, often a suﬃcient level of

I correctness

I eﬃciency

I costs cannot be guaranteed.

I Vision: Development methods for SW with warranty.

B. Aichernig Quality Assurance in Software Development 12 / 20 Institute for Software Technology tugraz Quality?

I Today, often a suﬃcient level of

I correctness

I eﬃciency

I costs cannot be guaranteed.

I Vision: Development methods for SW with warranty.

B. Aichernig Quality Assurance in Software Development 12 / 20 Marketing with App

Institute for Software Technology tugraz Limited Warranties

I Cisco: “Cisco does not warrant that the Software will operate uninterrupted or error-free or that all errors will be corrected. In addition, Cisco does not warrant that the Software or any equipment, system or network on which the Software is used will be free of vulnerability to intrusion or attack." Limited Warranty and Disclaimer, (c) Disclaimer; eﬀective: May 22, 2017: (https://www.cisco.com/c/en/us/about/legal/ cloud-and-software/end_user_license_agreement.html)

I Skydrop Sprinkler Controller:“This warranty does not cover consumable parts, including batteries, unless damage is due to defects in materials or workmanship of the Product, or software (even if packaged or sold with the product).” (https://www.skydrop.com/warranty/)

B. Aichernig Quality Assurance in Software Development 13 / 20 Institute for Software Technology tugraz Limited Warranties

B. Aichernig Quality Assurance in Software Development 13 / 20 Institute for Software Technology tugraz Bugs

Part of engineering jargon for many decades:

I Moth trapped in relay of Mark II (Hopper 1946)

I Little faults and diﬃculties (Edison 1878):

I Software bugs Relay #70 Panel F (moth) in relay. First actual case of bug being found.

B. Aichernig Quality Assurance in Software Development 14 / 20 Institute for Software Technology tugraz Bugs

Part of engineering jargon for many decades:

I Moth trapped in relay of Mark II (Hopper 1946)

I Little faults and diﬃculties (Edison 1878):

I Software bugs Relay #70 Panel F (moth) in relay. First actual case of bug being found.

B. Aichernig Quality Assurance in Software Development 14 / 20 Institute for Software Technology tugraz Bugs

Definition A software bug is the common term Part of engineering jargon for many used to describe an decades: error, flaw, mistake, failure, or Moth trapped in relay of Mark II I I fault in a computer program or (Hopper 1946) system Little faults and difficulties I that produces an incorrect or (Edison 1878): I unexpected result, I Software bugs I or causes it to behave in unintended ways. (Wikipedia 2012)

B. Aichernig Quality Assurance in Software Development 14 / 20 Institute for Software Technology tugraz Some Bugs Become Famous!

I Ariane 5 test ﬂight (1996)

I out of control due to software failure

I controlled destruction!

I Loss of

I money and time

I satellites

I research (TU Graz)

I Dijkstra (EWD 1036):

I call it error, not bug

I a programmer created it

B. Aichernig Quality Assurance in Software Development 15 / 20 Institute for Software Technology tugraz Some Bugs Become Famous!

I Ariane 5 test ﬂight (1996)

I out of control due to software failure

I controlled destruction!

I Loss of

I money and time

I satellites

I research (TU Graz)

I Dijkstra (EWD 1036):

I call it error, not bug

I a programmer created it

B. Aichernig Quality Assurance in Software Development 15 / 20 Algorithm was proven correct!

I Programming Pearls [Bentley86, Bentley00]

I assuming inﬁnite integers :(

Institute for Software Technology tugraz Some Bugs Hide for a Long Time! Binary search bug in Java 1 public static I JDK 1.5 library (2006) 2 int binarySearch(int[] a,int key) 3 { I out of boundary access of 4 int low = 0; large arrays 5 int high = a.length - 1; 6 7 while (low <= high) { I due to integer overﬂow 8 int mid = (low + high) / 2; I 9 years undetected 9 int midVal = a[mid]; 10 11 if (midVal < key) 12 low = mid + 1; 13 else if (midVal > key) 14 high = mid - 1; 15 else 16 return mid;// key found 17 } 18 return -(low + 1);// key not found 19 }

“Beware of bugs in the above code; I have only proved it correct, not tried it.” [Knuth77]

B. Aichernig Quality Assurance in Software Development 16 / 20 Institute for Software Technology tugraz Some Bugs Hide for a Long Time! Binary search bug in Java 1 public static I JDK 1.5 library (2006) 2 int binarySearch(int[] a,int key) 3 { I out of boundary access of 4 int low = 0; large arrays 5 int high = a.length - 1; 6 7 while (low <= high) { I due to integer overﬂow 8 int mid = (low + high) / 2; I 9 years undetected 9 int midVal = a[mid]; 10 11 if (midVal < key) 12 low = mid + 1; Algorithm was proven correct! 13 else if (midVal > key) 14 high = mid - 1; I Programming Pearls 15 else [Bentley86, Bentley00] 16 return mid;// key found 17 } I assuming inﬁnite integers 18 return -(low + 1);// key not found :( 19 } “Beware of bugs in the above code; I have only proved it correct, not tried it.” [Knuth77]

B. Aichernig Quality Assurance in Software Development 16 / 20 Institute for Software Technology tugraz Some Bugs Hide for a Long Time! Binary search bug in Java 1 public static I JDK 1.5 library (2006) 2 int binarySearch(int[] a,int key) 3 { I out of boundary access of 4 int low = 0; large arrays 5 int high = a.length - 1; 6 7 while (low <= high) { I due to integer overﬂow 8 int mid = (low + high) >>> 1; I 9 years undetected 9 int midVal = a[mid]; 10 11 if (midVal < key) 12 low = mid + 1; Algorithm was proven correct! 13 else if (midVal > key) 14 high = mid - 1; I Programming Pearls 15 else [Bentley86, Bentley00] 16 return mid;// key found 17 } I assuming inﬁnite integers 18 return -(low + 1);// key not found :( 19 } “Beware of bugs in the above code; I have only proved it correct, not tried it.” [Knuth77]

B. Aichernig Quality Assurance in Software Development 16 / 20 Institute for Software Technology tugraz Risks for the Public due to SW

The Risks Digest

http://catless.ncl.ac.uk/Risks/

B. Aichernig Quality Assurance in Software Development 17 / 20 Institute for Software Technology tugraz Edsger W. Dijkstra

I In academia, in industry, and in the commercial world, there is a widespread belief that computing science as such has been all but completed and that,

I consequently, computing has matured from a theoretical topic for the scientists to a practical issue for the engineers, the managers, and the entrepreneurs. [...]

I ... then he characterises the software crisis and concludes:

I I would therefore like to posit that computing’s central challenge, "How not to make a mess of it," has not been met.

B. Aichernig Quality Assurance in Software Development 18 / 20 Institute for Software Technology tugraz Edsger W. Dijkstra (cont.)

I On the contrary, most of our systems are much more complicated than can be considered healthy, and are too messy and chaotic to be used in comfort and conﬁdence.

I The average customer of the computing industry has been served so poorly that he expects his system to crash all the time, and we witness a massive worldwide distribution of bug-ridden software for which we should be deeply ashamed. (Communications of the ACM, Mar 2001)

B. Aichernig Quality Assurance in Software Development 19 / 20 Institute for Software Technology tugraz Why?

I A possible explanation: Quality assurance requires a certain degree of (healthy) redundancy

I → extra costs! (≥ 50% for critical systems)

I Examples of redundancy:

I process management

I documentation

I pair programming

I modelling

I test cases

I redundant components in fault-tolerant systems (accepted)

B. Aichernig Quality Assurance in Software Development 20 / 20 Institute for Software Technology tugraz Why?