Integrating Quality Models and Static Analysis for Comprehensive Quality Assessment Klaus Lochmann Lars Heinemann Technische Universität München, Garching b. München, Germany {lochmann,heineman}@in.tum.de ABSTRACT (1) Code analysis tools provide a large range of metrics To assess the quality of software, two ingredients are avail- that can be used as indicators for the quality of a software able today: (1) quality models defining abstract quality product. However, these metrics focus on very special as- characteristics and (2) code analysis tools providing a large pects of the source code and it is therefore difficult to use variety of metrics. However, there exists a gap between these them for obtaining a comprehensive overview of the overall two worlds. The quality attributes defined in quality models quality of a software system. are too abstract to be operationalized. On the other side, (2) Quality models like ISO 25010 [2] and others [3, 7, the aggregation of the results of static code analysis tools re- 6] define high-level quality attributes, which are commonly mains a challenge. We address these problems by defining a used to characterize the quality of software. However, these quality model based on an explicit meta-model. It allows to quality models are too abstract to be operationalized for the operationalize quality models by defining how metrics cal- quality assessment of a software system. culated by tools are aggregated. Furthermore, we propose a Problem. There exists a gap between abstract quality new approach for normalizing the results of rule-based code characteristics defined in quality models on the one hand analysis tools, which uses the information on the structure and isolated quality analysis tools on the other hand. The of the source code in the quality model. We evaluate the problem of obtaining quality assessments using static code quality model by providing tool support for both develop- analysis can be split into two parts: ing quality models and conducting automatic quality assess- (1) The aggregation of the results of heterogenous static ments. Our results indicate that large quality models can be code analysis tools to an overall quality assessment of a built based on our meta-model. The automatic assessment software product remains a challenge. For tools that ad- shows a high correlation between the automatic assessment dress this challenge, adequate user assistance is required for and an expert-based ranking. defining meaningful and comprehensible aggregation speci- fications. Especially for rule-based static code analysis tools that produce rule violation messages (like FindBugs1 and Categories and Subject Descriptors PMD2) associated with code locations (called by us find- D.2.8 [Software Engineering]: Metrics|Product metrics; ings) rather than metric numbers, little work exists on how D.2.9 [Software Engineering]: Management|Software to aggregate them. Besides the simplistic defect density Quality Assurance (SQA) (number of findings per lines of code) approaches exist that allow to specify arbitrary mathematical expressions to ag- General Terms gregate the results. There is no systematic approach for obtaining meaningful and comprehensible aggregation spec- Measurement ifications for rule-based static code analysis tools. (2) An additional challenge is the large number of rules Keywords provided by static code analysis tools. Organizing them and Quality Models, Software Metrics, Static Code Analysis working with them is not possible without a mechanism to structure and classify them in a comprehensible manner. 1. INTRODUCTION Contribution. In this paper we present a quality model that addresses both problems. An explicit meta-model de- Motivation. To effectively manage software costs it is fines a comprehensive structure for quality models. This essential to assess the quality of software. For this, two structure allows us to build adequate tool-support for man- major ingredients are available today. aging quality models growing very large. Regarding the problem of aggregating the results of rule- based code analysis tools, we propose a new approach for Permission to make digital or hard copies of all or part of this work for normalizing them. This approach is tightly coupled to the personal or classroom use is granted without fee provided that copies are quality model. The quality model provides the necessary not made or distributed for profit or commercial advantage and that copies background information on the structure of the source code, bear this notice and the full citation on the first page. To copy otherwise, to in order to enable the user to define meaningful aggregations. republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 1http://findbugs.sourceforge.net WETSoM ’11, May 24, 2011, Waikiki, Honolulu, HI, USA 2 Copyright 2011 ACM 978-1-4503-0593-8/11/05 ...$10.00. http://pmd.sourceforge.net We evaluate the approach by providing tool support for 3. QUALITY MODEL both developing quality models and for conducting auto- The quality model pursuits three goals: (1) to structure matic quality assessments. A case study shows the suitabil- rules of static analysis tools in a comprehensible way, (2) to ity of the quality model to cover a large number of rules define their influence on quality characteristics of the soft- from two rule-based static analysis tools. To evaluate the ware product, and (3) to define comprehensible and under- meaningfulness of the produced quality assessments we ap- standable aggregation formulas for automatically calculating ply the automatic assessment based on the quality model quality assessments of a software product. to 17 open source systems. Our results are analyzed for The quality meta-model presented here is a variant of the their diversification and for their consistency with an inde- model developed in the project Quamoco8. This variant pendent expert-based assessment. The results indicate that aims for clear and understandable aggregation specifications. the diversification is satisfactory and that there is a high correlation between the two assessment results. 3.1 Structure The quality model relies on a product model of software, 2. RELATED WORK which describes the entities a software product consists of. The related work can be split into three parts: The idea of using a product model as a backbone of the qual- (1) Quality models were proposed to structure and under- ity model has been proposed in similar forms in literature [5, stand the multi-faceted concept of software quality [3, 5, 6]. 4]. In our quality model, we model the parts of the software These models define quality by decomposing it into more product as Entities. Between entities we define a special- manageable quality attributes. However, these quality at- ization relation is-a and a decomposition relation part-of. tributes are too abstract to be directly measurable and do Since we model the rules of static code checkers, the enti- not allow for an operationalization in terms of a quality as- ties usually refer to the source code of the software. Typical sessment of real software systems. In contrast, our model entities include Class and Expression, whereby Expression allows to define quality from abstract characteristics down is further refined by Relational expression and Arithmeti- to concrete measures. cal expression, which are in an is-a relation to Expression. (2) Code analysis tools determine software metrics and qual- The main concept of the quality model is a Property. As ity defects like coding guideline violations or bug patterns the name suggests, it describes a property of the software (e. g., FindBugs1, PMD2, Checkstyle3 ). Although these product. In the literature, the term property is defined as tools are highly valuable within their specific scope, it re- an attribute which is used to characterize an object. The mains unclear how to deduct from the results of them a notion of an object that is characterized is central to all comprehensive assessment of the overall quality of a soft- definitions. In our quality model, we define a Property as ware system. an Attribute of an Entity. In the following we use the (3) Dashboard tools integrate multiple quality analyses and notation [attribute@entity] for properties. allow to aggregate and visualize their results in a comprehen- In our quality model, we use entities that character- sive quality dashboard (e. g., Sonar4, XRadar5, QALab6). ize the entire product, to express the \-ilities" of the However, in these tools the metrics are either not explicitly ISO 25010 [2]; for example the \maintainability" is expressed linked to a quality model or they use a fixed quality model as [Maintainability@Product]. Other attributes are used for and thus do not allow for project-specific definition of soft- parts of the product, for example \Correctness", resulting in ware quality. [Correctness@Arithmetical expression]. The approach of Schackmann et al. [9] connects measure- Properties can be refined, using the Refines relation. If ment tools with a quality model. The quality model is de- a property B refines a property A, it means that B is more fined in their tool, which requires the user to define the special regarding the entity it characterizes than A. More functions for aggregating the measurement values. How- precisely, if a property B refines a property A, then the ever, their approach allows arbitrary functions, and gives entity of B must be in a is-a relation to the entity of A. For therefore no help to the user in specifying meaningful ones. A= [a@e1] and B= [a@e2], written formally that is: 7 The research project Squale develops a quality model ref is-a based on existing standards (e. g., ISO 9126) and tool sup- [a@e1] − [a@e2] ) e1 − e2 (1) port that allows to aggregate metrics from third party anal- The refinement relation is used to create a hierarchy of ysis tools to high level quality factors.