Examining the Performance of Java Static Analyzers Kevin Daimi and Shadi Banitaan Department of Mathematics, Computer Science and Software Engineering University of Detroit Mercy, 4001 McNichols Road, Detroit, MI 48221 {daimikj, banitash}@udmercy.edu Kathy Liszka Department of Computer Science The University of Akron Akron, Ohio 44325-4003 [email protected] ABSTRACT tools often uncover true but trivial bugs and some details about violations throughout the development Static Analysis refers to the analysis of computer lifecycle of software. They further added that there is programs prior to executing them to reveal potential little published information regarding the evaluation of problems that need to be fixed before executing the these tools to verify their claims. It is understandable programs. In this paper, five static analyzers for Java that companies may prohibit the publication of any programs will be examined and compared using three experimental data for commercial tools. However, Java programs, which are randomly selected from a publishing such data for open source tools should not be collection available on the Internet. a problem. Keywords During software development, it is valuable to obtain early estimates of the defect density of software Static Analyzer, Java, Evaluation, Software Engineering components to further improve the quality of software. Such estimates identify fault-prone areas of code requiring further testing. It is valuable to collect early I. INTRODUCTION estimates of fault density for software components throughout the process of software development. Static analysis is becoming a critical component for Nagappan et al. [13] presented an empirical software development. Currently, many software methodology for the early projection of pre-release developers are appreciating the advantages of using defect density based on the outcomes of static analysis static analyzers to improve software. Static analyzers tools. With the aid of two different static analysis tools, function through using techniques from program the discovered defects were used to predict the actual analysis, model checking, and automated deduction [3]. pre-release defect density for Windows Server 2003. Static analysis tools can also be used to automate the They concluded that there was a strong positive process of identifying violations of security rules [12]. correlation between the static analysis list of defects and the pre-release defect density list obtained through Despite the popularity of static analysis tools for actual testing. There are a number of approaches for software flaws discovery, experimental assessments of static analysis. Static analysis by Abstract Interpretation the correctness and merits of the output of these tools [15] is one such approach. The authors indicated that are lacking. Ayewah et al. [2] examined the types of this approach offers a considerable assurance and warnings generated and the classification of warnings evidence needed for supporting its findings. They into false positives, trivial bugs and serious bugs for demonstrated that static analysis must be able to scale FindBugs, a static analysis tool for Java programs. and report few false positives without calling for expert They stipulated some perception into why static analysis interference. As mentioned above, public information on evaluating evaluative rules that have been extracted during a given static analyzers is scarce. An interesting study by Ware execution [15]. This tool is equipped with a default set et al. [17] focused on evaluating the degree to which of rules which can be used to reveal common eight static analysis tools can isolate violations of a development bugs. PMD also supports custom broad set of coding heuristics for increasing the quality analyses by allowing users the opportunity to develop and security of Java SE code. They revealed that a their own (new) evaluative rules. It scans Java source significant number of security violations were not code looking for potential problems including empty detected by any tool. The resulting vulnerabilities can try, catch, finally, and switch statements, dead code, easily lead to various attacks. Note that three of the suboptimal code, overcomplicated expressions, and tools used in this study; CheckStyle, Findbugs, and duplicate code. It can be integrated with JDeveloper, PMD are further analyzed in our study below. Eclipse, JEdit, JBuilder, BlueJ, CodeGuide, In this paper, four open source and one commercial NetBeans/Sun Java Studio Enterprise/Creator, IntelliJ static analysis tools are evaluated. Three levels of IDEA, TextPad, Maven, Ant, Gel, JCreator, and evaluation including general features, performance, and Emacs. Copeland [6] indicated that Junit tests can be capabilities are exercised. For this purpose, three kept in good order by using PMD]. random programs available online, are used. To study the performance of each tool on unearthing various C. ESC/Java2 fault/violations categories and sub-categories, violations were temporarily inserted into these programs. The Extended Static Checker for Java version 2 Outcomes of these evaluations are summarized in (ESC/Java2) is a programming tool that endeavors to various tables. discover common run-time errors in JML-annotated Java programs by static analysis of the program code II. STATIC ANALYSIS TOOLS OVERVIEW and its formal annotations. It allows users the flexibility to control the extent and types of checking that The static analysis tools for Java studied in this paper ESC/Java2 implements by annotating Java programs are briefly described below. with specifically formatted comments called pragmas [8]. This implies that the ESC/Java2 tool tries to A. FindBugs unearth common run-time errors in Java programs at compile time [10]. The approach used in ESC/Java2 FindBugs is an open source static analysis tool that digs comprises a range of techniques for statically checking into class or JAR files looking for potential problems the correctness of various program constraints. through matching Java bytecodes against a list of known Extended static checking usually deploys an automated bug patterns [9]. The current version of FindBugs theorem prover [7]. ESC/Java2 can be integrated with (2.0.2) requires JRE (or JDK) 1.5.0 or later to operate. the Mobius Program Verification Environment, used as However, it can analyze programs compiled for any a command-line tool with a simple Swing GUI front- version of Java, from 1.0 to 1.8. It is capable of end, or added as an Eclipse plugin. identifying over 250 potential types of errors. FindBugs D. CheckStyle uses real bugs in software, extracts a bug pattern from those bugs, and develops possible detectors that can Checkstyle is an open source development tool, which efficiently pinpoint that bug pattern. In other words, it aims to help programmers write Java code that follows is based on the concept of bug patterns [5]. The process some coding standard [4]. It automates the process of is evaluated by trying the recommended detector on checking Java code resulting in coding standard various test cases for that bug pattern [11]. In FindBugs, enforcement. Checkstyle is highly configurable and can bugs are ranked from 1-20, and grouped into the following categories: scariest (rank 1-4), scary (rank 5- support many coding standards. A number of sample 9), troubling (rank 10-14), and of concern (rank 15-20). configuration files are supplied for well-known conventions, such as Sun Code Conventions. It provides a flexible way for developers to share Historically, Checkstyle’s main functionality evolved information and define and install plugins. It can be around checking code layout concerns, but since its integrated with Eclipse, Maven, NetBeans, Hudson, and internal architecture was modified starting in version 3, IntelliJ. more checks for other purposes have been added. Currently, Checkstyle provides checks that uncover a B. PMD number of issues including class design problems, duplicate code, or bug patterns like double checked PMD is an open-source, rule-based, static source code locking. It supports loading a configuration from URL analyzer that analyzes Java source code based on reference and can be integrated with Eclipse, IntelliJ IDEA, NetBeans, BlueJ, tIDE, Emacs JDE, Jedit, Vim IV. TOOLS RFORMANCE EVALUATION Editor, Maven, and QALab. Having analyzed the five tools based on the three E. AppPerfect Java Code Test (AppPerfect) criterions; violations, run-time, and memory, a deep- rooted evaluation will be carried out to reveal the actual performance of each tool with regards to various fault AppPerfect Java Code Test is a commercial static Java categories. For this purpose, various faulty codes are code analysis tool aimed at automating Java code temporary injected in the three programs. The fault review and enforcing good Java coding practices [1]. categories that will be used for this evaluation involves AppPerfect Code Test analyzes both Java and Java data faults, control faults, interface faults, measurement Server Pages (JSP) source code using a large set of Java faults, duplicate code, and code convention violations. coding rules extracted from experts in the Java Each of these categories is further divided into programming field. These rules are grouped into a subcategories. Detailed analysis is provided in tables 4- number of functional areas such as security, 9 below. In these tables, “Y” indicates that the tool is optimization, and portability. AppPerfect analyzes