WP4:WP4: TestingTesting toolstools andand methodologiesmethodologies
István Forgács 4D SOFT
www.eu-etics.org
INFSOM-RI-026753 Team
• 4D SOFT’s staff – some changes – István Forgács - responsible for – leading WP4 – 4D Soft’s own methods and tools – Key scientific issues – Cooperate with the coordinator, visiting PMBs, etc. – Anna Bánsághi, – Study and evaluate testing tools – Katalin Gergely, (Thalmeiner Zsolt) – Study and evaluate testing methods – Cooperates with Univ. of Wisconsin staff – Andor Dirner – DILIGENT connection – Gyöngyi Kispál – Financial issues – Klára Tauszig – Administrative issues
INFSOM-RI-026753 A4.1 requirements
A4.1 Collect requirements for unit, functional and regression testing. There is a large number of research papers on the test adequacy criteria of unit testing. However, there are no related tools in the practice for most of them. We are going to collect applicable and practical requirements based on our knowledge, experience and the testing tools currently available and widely used such as JUnit. The task involves the requirements and applicability criteria for test planning, dynamic testing and the testing tools.
INFSOM-RI-026753 Work has been done
• General requirements for test automation – Studying the literature especially the book of Fewster and Graham: Software Test Automation – Key result: test automation should be based on good test methods • Requirements for test planning – Easy to learn, – Easy to use – Easy to maintain – Can be automated – Can be generally used – Needs more invention and less manual work
INFSOM-RI-026753 Methods
• Methods have been studied – Method for the whole testing process – V model – Test planning – Category-partitioning (detailed description tomorrow) – Syntax testing – can be automated, checks correct and incorrect syntax – Equivalence Partitioning – input domain is partitioned into sub domains so that one sub domain behaves uniquely w.r.t. the errors – widely used – disadvantages • Doesn't test every input • No guidelines for choosing inputs • Heuristic based • very limited focus
INFSOM-RI-026753 Methods
• Methods have been studied – Test planning – Boundary Value Analysis – together with the equivalence partitioning – Well-known requirements that boundaries and close to the boundaries should be tested – The boundaries of both valid and invalid partitions are considered – Disadvantages • Does not test all possible inputs • Does not test dependencies between combinations of inputs – Error guessing – the experience of the tester is used to postulate what faults might occur, and to design tests specifically to expose them. – These three are used together.
INFSOM-RI-026753 Methods
• Methods have been studied – Test planning – Cause-Effect Graphing: old fashioned, not maintainable – State transition diagrams: widely used if the software contains states and different events cause the transition from one to another example ATM – Random testing – random input for test cases. The distribution (normal, uniform, etc.) to be used in the generation of random input values shall be based on the expected operational distribution of inputs. – Ad hoc testing unfortunately it’s widely used – Fuzz input testing - lots of arbitrary, mainly invalid input – Smoke testing: is a cursory examination of all of the basic components of a software system to ensure that they work – Equivalence testing this is the best method if you have two different software aiming to do the same things
INFSOM-RI-026753 methods
• Unit, component and integration testing – Black box methods – Test planning methods – EBIT (tomorrow) – White box methods – Test bed – Test driver – missing callers – Test stubs – missing callees – Coverage analyses – All statement criterion – All branch criterion – most widely used minimum requirement – MC/DC criterion – Data flow criteria – better but rarely applied – Reduced path testing – cycles are tested with 0,1 and more iterations
INFSOM-RI-026753 methods
– Scripting languages – Capture and play back techniques – Static analysis – Dependence analysis (slicing) – Metrics – Exception handling analysis – Null pointer analysis – Non-initialised variable analysis
INFSOM-RI-026753 methods
• Regression testing – Dynamic regression testing – Full regression testing – Selective regression testing – Selection modification-traversing tests – Classifying regression tests – At least one output has been influenced – Influences through predicates – No influences exist – If necessary transforming tests to be effective – Static regression testing – Original and modified sources are available – Dependence analysis considering the modifications – Selecting the non-expected influences
INFSOM-RI-026753 A4.1 Collect requirements for tools
• Testing tool selection – Requirements can only be collected based on existing tools – We studied tools for different purposes – Test planning - No specification-based tools have been considered studying the specification, detailed design, etc. documents for DILIGENT – CatGen – Unit testing – Junit – GJTester – Functional, regression – vTest – Static analysis – INDUS
INFSOM-RI-026753 A4.1 Collect requirements for tools • Testing tool selection – Acceptance testing – AutAT (web appl) – Accessibility testing – CSE Lite (web appl test tool) – A-Prompt (web appl test tool for people with disabilities) – PowerTools HTTP proxy – PowerTools Connection Test – PowerTools HTTP Request Editor – PowerTools Expression Test – PowerTools Encode/Decode – Performance testing (load, stress) – OpenSTA – TestMaker – Grinder – Jmeter – LoadRunner – Analog – Coverage – JavaCov INFSOM-RI-026753 Immediate future plan
• More testing tools to study – Functional – Qengine (performance) – Enterprise Web Test (stress, reliability) – AppPerfect WebTest (load) – JFunc – Conformance – SuiteRunner (unit) – Regression, smoke – ITP – Integration – iValidator – Performance – TPTP
INFSOM-RI-026753 Immediate future plan
• More testing tools to study – Component – ASP Component Test – Code analyser – PMD – Coverage – Jcoverage – Unit – AppPerfect DevTest
INFSOM-RI-026753 Immediate future plan
• We have finished studying testing methods • Requirement analysis – Starting points – Methods available – Testing tools available – Testing research – Test books for testing – Setting requirements – Weighting requirements – Validating and comparing requirements
INFSOM-RI-026753 Deliverables
• We haven’t any scheduled deliverables yet • Future deliverables in this year – D4.1 - Distributed test execution system (prototype) in PM06. Lead participant: UoW – D4.2 - Requirements for unit, functional and regression testing in PM06. Lead participant: 4DS
INFSOM-RI-026753 metrics
• Number of testing tools downloaded and studied – Plan 20-25 current status 20
• Number of test methods studied – Plan 20-30 current status 23
• Number of quality metrics identified for test methods and tool selection – Plan 15-20 current status 0
INFSOM-RI-026753 Future plan
• Selection of the testing tools and methodologies based on the requirements derived in the previous part • Working out special aid on how to implement and use these tools/methods with clear examples explaining the main difficulties of the application.
INFSOM-RI-026753