IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 44, NO. 5, MAY 2018 429 Detecting Overly Strong Preconditions in Refactoring Engines Melina Mongiovi , Rohit Gheyi, Gustavo Soares, Marcio Ribeiro, Paulo Borba, and Leopoldo Teixeira Abstract—Refactoring engines may have overly strong preconditions preventing developers from applying useful transformations. We find that 32 percent of the Eclipse and JRRT test suites are concerned with detecting overly strong preconditions. In general, developers manually write test cases, which is costly and error prone. Our previous technique detects overly strong preconditions using differential testing. However, it needs at least two refactoring engines. In this work, we propose a technique to detect overly strong preconditions in refactoring engines without needing reference implementations. We automatically generate programs and attempt to refactor them. For each rejected transformation, we attempt to apply it again after disabling the preconditions that lead the refactoring engine to reject the transformation. If it applies a behavior preserving transformation, we consider the disabled preconditions overly strong. We evaluate 10 refactorings of Eclipse and JRRT by generating 154,040 programs. We find 15 overly strong preconditions in Eclipse and 15 in JRRT.Our technique detects 11 bugs that our previous technique cannot detect while missing 5 bugs. We evaluate the technique by replacing the programs generated by JDOLLY with the input programs of Eclipse and JRRT test suites. Our technique detects 14 overly strong preconditions in Eclipse and 4 in JRRT. Index Terms—Refactoring, overly strong preconditions, automated testing, program generation Ç 1INTRODUCTION EFACTORING is the process of changing a program to Nevertheless, refactoring engines may have overly weak Rimprove its internal structure while preserving its and overly strong preconditions [12], [13], [14]. Overly weak observable behavior [1], [2], [3]. Refactorings can be applied preconditions allow incorrect transformations to be applied, manually, which may be time consuming and error prone, while overly strong preconditions prevent developers from or automatically by using a refactoring engine, such as applying behavior preserving transformations. Neither of Eclipse [4], NetBeans [5], and JastAdd Refactoring Tools these problems are desirable. Vakilian and Johnson [15] (JRRT) [6], [7], [8], [9]. These engines contain a number of have shown that programmers prefer that the refactoring refactoring implementations, such as Rename Class, Pull engine does not reject a refactoring application, even if they Up Method, and Encapsulate Field. For correctly applying need to manually fix some problems afterwards. a refactoring, and thus, ensuring behavior preservation, the Some approaches have proved the correctness of some refactoring implementations might need to consider a num- refactorings for a subset of Java [16], [17], [18]. They ensure ber of preconditions, such as checking whether a method or that the resulting programs compile and that the transforma- field with the same name already exists in a type. However, tions preserve the program behavior. However, none of them defining and implementing preconditions is a nontrivial proved that such preconditions are minimal. As a result, those task. Proving the correctness of the preconditions with res- approaches may reject useful transformations. In practice, pect to a formal semantics of complex languages such as developers are concerned about testing whether their refactor- Java, constitutes a challenge [10]. ing implementations have overly strong preconditions. For In practice, refactoring engine developers may imple- example, we find that 32 percent of the assertions used in ment the refactoring preconditions based on their experi- the test suites of 10 refactoring implementations of Eclipse ence, some previous work [11], or formal specifications [6]. (30 percent) and 10 of JRRT (36 percent) are concerned with detecting overly strong preconditions. Moreover, we find more than 40 bugs related to overly strong preconditions in M. Mongiovi, R. Gheyi, and G. Soares are with the Department of Com- the Eclipse’s bug tracker. More than 50 percent of them were puting and Systems, Federal University of Campina Grande, Campina Grande, PB 58429-900, Brazil. E-mail: [email protected], already fixed. Testing refactoring engines is not trivial since it {rohit, gsoares}@dsc.ufcg.edu.br. requires complex inputs, such as programs and an oracle to M. Ribeiro is with the Computing Institute, Federal University of Alagoas, define the correct resulting program. In general, developers Maceio, AL 57072-900, Brazil. E-mail: [email protected]. manually select the input programs according to their experi- P. Borba and L. Teixeira are with the Informatics Center, Federal Univer- sity of Pernambuco, Recife, PE 50732-970, Brazil. ence, which is costly, and thus it may be difficult to establish a E-mail: {phmb, lmt}@cin.ufpe.br. good test suite considering all language constructs. Moreover, Manuscript received 15 Apr. 2016; revised 19 Mar. 2017; accepted 5 Apr. even if they establish useful inputs, they may not have a sys- 2017. Date of publication 11 Apr. 2017; date of current version 22 May 2018. tematic technique to detect the overly strong preconditions. Recommended for acceptance by M. Kim. Vakilian and Johnson [15] use refactoring alternate paths to For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference the Digital Object Identifier below. identify usability problems related to overly strong precondi- Digital Object Identifier no. 10.1109/TSE.2017.2693982 tions in refactoring engines. They discover usability problems 0098-5589 ß 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. 430 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 44, NO. 5, MAY 2018 by analyzing the interactions of users that had problems with 35.72 h to detect all overly strong preconditions of JRRT and tools in general. They manually inferred usability problems Eclipse, respectively. Moreover, the technique takes on aver- from the detected critical incidents. age a few seconds to find the first overly strong precondition Our previous work uses Differential Testing [19] to auto- in JRRT and 17.41 min in Eclipse. matically identify transformations rejected by refactoring DP and DT techniques are complementary in terms of bug engines due to overly strong preconditions (DT tech- detection. The DP technique finds 11 bugs not detected by nique) [13]. It automatically generates a number of programs DT, and the DT technique finds 5 bugs not detected by DP as test inputs using JDOLLY, a Java program generator [12]. (false-negatives). In addition, the DP technique does not Next, it applies the same refactoring to each test input using require using another refactoring engine with the same refac- two different implementations, and compares both results. torings to compare the results. So whenever possible, devel- The technique uses SAFEREFACTOR [20], [21] to automatically opers can run the DP technique and after fixing the detected evaluate whether a transformation preserves the program bugs, they run the DT technique to search for more bugs. behavior. SAFEREFACTOR automatically evaluates whether two We also perform another study, where we use programs versions of a program have the same behavior by automati- from Eclipse and JRRT refactoring test suites as input for DP cally generating test cases only for the common methods and DT techniques instead of the JDOLLY generated programs. impacted by the change. To use this technique, developers These programs are used in test cases that expect the refactor- need access to at least two refactoring engines. However, it ing engine to prevent refactoring application. We assess the can only be used if both refactoring engines implement the same refactoring implementations evaluated before using same refactoring. both techniques. We detect 23 overly strong preconditions In this work, we propose Disabling Preconditions (DP), a (17 of them are new) previously undetected by the develop- new technique to detect overly strong preconditions in refac- ers. In addition to these bugs, we find some false-positives toring implementations by disabling preconditions. Hereafter due to the equivalence notion of SAFEREFACTOR. The develop- we refer to disabling preconditions as the process of prevent- ers did not find these overly strong preconditions because ing to report messages to the user, raised by the preconditions. they may not have a systematic strategy to detect them, even A message is reported when a precondition is unsatisfied. We with useful input programs in their test suite. Additionally, automatically generate a number of programs as test inputs, they may not have an automated oracle to check behavior using JDOLLY. For each generated program, we attempt to preservation, such as SAFEREFACTOR. apply the transformation using the refactoring implementa- In summary, the main contributions of this article are the tion that is being tested. When the refactoring implementation following: rejects a transformation, it reports a message to the user a new technique to detect overly strong precondi- describing the problem. For each kind of message, we identify code fragments related to the precondition that yields the tions