An Approach to Quality Engineering of TTCN-3 Test Specifications
Total Page:16
File Type:pdf, Size:1020Kb
Int J Softw Tools Technol Transfer (2008) 10:309–326 DOI 10.1007/s10009-008-0075-0 SPECIAL SECTION ON ADVANCES IN TEST AUTOMATION – THE EVOLUTION OF TTCN-3 An approach to quality engineering of TTCN-3 test specifications Helmut Neukirchen · Benjamin Zeiss · Jens Grabowski Published online: 6 May 2008 © Springer-Verlag 2008 Abstract Experience with the development and mainte- 1 Introduction nance of large test suites specified using the Testing and Test Control Notation (TTCN-3) has shown that it is diffi- The Testing and Test Control Notation (TTCN-3) [16,22]is cult to construct tests that are concise with respect to quality a mature standard which is widely used in industry and stan- aspects such as maintainability or usability. The ISO/IEC dardisation to specify abstract test suites. Nowadays, large standard 9126 defines a general software quality model that TTCN-3 test specifications with a size of several ten thou- substantiates the term “quality” with characteristics and sub- sand lines of code are developed [2,13–15]. Like any other characteristics. The domain of test specifications, however, large software, such large test specifications tend to have requires an adaption of this general model. To apply it to quality problems. The roots of these quality problems are specific languages such as TTCN-3, it needs to be instanti- manifold, for example inexperienced test developers [2]or ated. In this paper, we present an instantiation of this model as software ageing [39]. Usually, statements on quality defi- well as an approach to assess and improve test specifications. ciencies of test suites are made in a subjective manner. How- The assessment is based on metrics and the identification of ever, to obtain a dependable quality assurance for TTCN-3 code smells. The quality improvement is based on refactor- test suites, an impartial quality assessment for TTCN-3 test ing. Example measurements using our TTCN-3 tool TRex specifications is desirable. Hence, a model for test suite demonstrate how this procedure is applied in practise. quality is required. In this article, a method and a tool for quality engineer- Keywords Test specification · TTCN-3 · Quality model · ing of TTCN-3 test suites are presented. To this aim, we Code smells · Metrics · Refactoring use an adaption of the ISO/IEC 9126 [26] software prod- uct quality model which is suitable for test specifications. For the automated quality assessment of TTCN-3 test suites, we apply metrics and code smells measuring attributes of the quality characteristics that constitute the test specifica- tion quality model. Metrics are a common means to quan- B. Zeiss is supported by a Ph.D. scholarship from Siemens AG, tify properties of software. More sophisticated deficiencies Corporate Technology. in source code structures which cannot be identified by using metrics require a pattern-based analysis. These patterns in H. Neukirchen (B) · B. Zeiss · J. Grabowski source code are described by code smells. For the improve- Software Engineering for Distributed Systems Group, ment of test suites, we use refactoring. A refactoring is “a Institute for Computer Science, University of Göttingen, Lotzestr. 16–18, 37083 Göttingen, Germany change made to the internal structure of software to make it e-mail: [email protected] easier to understand and cheaper to modify without chang- B. Zeiss ing its observable behavior”[20]. This means, a refactoring e-mail: [email protected] is a behaviour preserving transformation which is able to J. Grabowski improve internal source code quality. For an automation of e-mail: [email protected] the complete quality assessment and improvement process, 123 310 H. Neukirchen et al. Fig. 1 The ISO/IEC 9126-1 External and Internal model for internal and external Quality quality Functionality Reliability Usability Efficiency Maintainability Portability Characteristics Understand- Suitability Analysability Adaptability ability Maturity Time Behaviour Accuracy Changeability Installability Learnability Fault Tolerance Resource Interoperability Stability Co-Existence Operability Utilisation Recoverability Security Testability Replaceability Attractiveness Efficiency Reliability Compliance Functionality Compliance Usability Maintainability Portability Subcharacteristics Compliance Compliance Compliance Compliance we have developed the TTCN-3 Refactoring and Metrics tool internal quality is obtained by reviews of specification docu- TRex which is available as open-source software. ments, checking models, or by static analysis of source code. This article is structured as follows: Sect.2 describes the External quality refers to properties of software interacting ISO/IEC 9126 software product quality model and intro- with its environment. In contrast, quality in use refers to the duces its adaptation to the domain of test specification. Sub- quality perceived by an end user who executes a software sequently, Sect. 3 surveys metrics and smells and presents product in a specific context. TTCN-3 specific metrics and code smells. An application AsshowninFig.1, ISO/IEC 9126 defines the same generic of metrics and smells for the quality assessment of TTCN-3 model for modelling internal and external quality. This specifications is demonstrated in Sect.4. In Sect. 5, the TRex generic quality model can then be instantiated as a concrete tool for assessing and improving TTCN-3 test specifications model for internal or external quality by using different sets is presented. Finally, a summary and an outlook are given. of metrics. The model itself is based on the six characteris- tics functionality, reliability, usability, efficiency, maintain- ability, and portability. Each characteristic is structured into 2 Quality of test specifications further subcharacteristics. The model of quality in use is based on the characteristics Quality models are needed to evaluate and set goals for the effectiveness, productivity, safety, and satisfaction and does quality of a software product. The ISO/IEC standard 9126 not elaborate on further subcharacteristics. In the further parts [26] defines a general quality model for software products of ISO/IEC 9126, metrics are defined which are intended to that requires an instantiation for each concrete target envi- be used to measure the properties of the (sub)characteristics ronment (e.g. a programming language). In this section, we defined in Part 1. The provided metrics are quite abstract briefly introduce ISO/IEC 9126 and present an adaption to which makes them applicable to various kinds of software the domain of test specifications [56]. products, but they cannot be applied without further refine- ment. 2.1 Software quality (ISO/IEC 9126) The actual process of assessing the quality of a software product is not part of ISO/IEC 9126. It is defined in the ISO/IEC 9126 [26] is an international multipart standard pub- ISO/IEC standard 14598 [25]: the assessment requires the lished by the International Organization for Standardization weighting of the different (sub)characteristics and the selec- (ISO) and the International Electrotechnical Commission tion of appropriate metrics. (IEC). It is based on earlier attempts for defining software quality [5,30] and presents a software product quality model, 2.2 Test specification quality quality characteristics, and related metrics. Part 1 of ISO/IEC 9126 contains a two-part quality model: Our quality model for test specification is an adaptation of the first part of the quality model is applicable for model- ISO/IEC 9126 to the domain of test specification. While the ling the internal and external quality of a software product, ISO/IEC 9126 model deals with internal quality, external whereas the second part is intended to model the quality in quality, and quality in use, the remainder of this article will use of a software product. These different quality models are only address internal quality characteristics. Due to our par- needed to be able to assess the quality of a software prod- ticipation in standardisation, our primary interest is the qual- uct at different stages of the software life cycle. Typically, ity of abstract test specifications. 123 An approach to quality engineering of TTCN-3 test specifications 311 Fig. 2 The test specification Test Specification quality model Quality Test Effectivity Reliability Usability Efficiency Maintainability Portability Reusability (Functionality) (Reliability) (Usability) (Efficiency) (Maintainability) (Portability) ( — ) Characteristics Test Repeatability Understand- ( — ) ability Test Coverage (Understand- (Suitability) Maturity ability) Time Behaviour Analysability Coupling (Maturity) (Time (Analysability) ( — ) Test Learnability Behaviour) Correctness Fault-Tolerance (Learnability) Changeability Adaptability Flexibility (Accuracy) (Fault- Resource (Changeability) (Adaptability) ( — ) Tolerance) Operability Utilisation Fault- (Operability) (Resource Stability Portability Comprehen- Revealing Security Utilisation) (Stability) Compliance sibility Capability ( — ) Test (Portability ( — ) Subcharacteristics ( — ) Evaluability Efficiency Maintainability Compliance) Recoverability ( — ) Compliance Compliance Reusability Test Effectivity (Recoverability) (Efficiency (Maintainability compliance Compliance Usability Compliance) Compliance) ( — ) (Functionality Reliability Compliance Complicance) Compliance (Usability (Reliability Compliance) Compliance) Bold text: Quality characteristic ( Text in parentheses ): Corresponding characteristic in ISO/IEC 9126-1 ( — ): No corresponding