Java Decompiler Diversity and its Application to Meta-decompilation ∗ Nicolas Harrand , César Soto-Valero, Martin Monperrus and Benoit Baudry
KTH Royal Institute of Technology, SE-100 44 Stockholm, Sweden
ARTICLEINFO ABSTRACT
Keywords: During compilation from Java source code to bytecode, some information is irreversibly lost. In other Java bytecode words, compilation and decompilation of Java code is not symmetric. Consequently, decompilation, decompilation which aims at producing source code from bytecode, relies on strategies to reconstruct the information reverse engineering that has been lost. Different Java decompilers use distinct strategies to achieve proper decompilation. source code analysis In this work, we hypothesize that the diverse ways in which bytecode can be decompiled has a direct impact on the quality of the source code produced by decompilers. In this paper, we assess the strategies of eight Java decompilers with respect to three quality indica- tors: syntactic correctness, syntactic distortion and semantic equivalence modulo inputs. Our results show that no single modern decompiler is able to correctly handle the variety of bytecode structures coming from real-world programs. The highest ranking decompiler in this study produces syntacti- cally correct, and semantically equivalent code output for 84%, respectively 78%, of the classes in our dataset. Our results demonstrate that each decompiler correctly handles a different set of bytecode classes. We propose a new decompiler called Arlecchino that leverages the diversity of existing decompil- ers. To do so, we merge partial decompilation into a new one based on compilation errors. Arlecchino handles 37.6% of bytecode classes that were previously handled by no decompiler. We publish the sources of this new bytecode decompiler.
1. Introduction Overall, the ideal decompiler is one that transforms all inputs into source code that faithfully reflects the original In the Java programming language, source code is com- code: the decompiled code 1) can be recompiled with a Java piled into an intermediate stack-based representation known compiler and 2) behaves the same as the original program. as bytecode, which is interpreted by the Java Virtual Ma- However, previous studies that compared Java decompilers chine (JVM). In the process of translating source code to [15, 23] found that this ideal Java decompiler does not ex- bytecode, the compiler performs various analyses. Even if ist, because of the irreversible data loss that happens dur- most optimizations are typically performed at runtime by the ing compilation. In this paper, we perform a comprehensive just-in-time (JIT) compiler, several pieces of information re- assessment of three aspects of decompilation: the syntactic siding in the original source code are already not present in correctness of the decompiled code (the decompiled code the bytecode anymore due to compiler optimization [28, 27]. can recompile); the semantic equivalence with the original For example the structure of loops is altered and local vari- source (the decompiled code passes all tests); the syntac- able names may be modified [17]. tic similarity to the original source (the decompiled source Decompilation is the inverse process, it consists in trans- looks like the original). We evaluate eight recent and notable forming the bytecode instructions into source code [31]. De- decompilers on 2041 Java classes, making this study one or- compilation can be done with several goals in mind. First, der of magnitude larger than the related work [15, 23]. it can be used to help developers understand the code of the Next, we isolate a subset of 157 Java classes that no state- libraries they use. This is why Java IDEs such as IntelliJ of-the-art decompiler can correctly handle. The presence and Eclipse include built-in decompilers to help developers
arXiv:2005.11315v1 [cs.SE] 21 May 2020 of generics and wildcards is a major challenge that prevents analyze the third-party classes for which the source code is successful decompilation. Meanwhile, we note that different not available. In this case, the readability of the decompiled decompilers fail for diverse reasons. This raises the oppor- code is paramount. Second, decompilation may be a pre- tunity to merge the results of several incorrect decompiled liminary step before another compilation pass, for example sources to produce a version that can be recompiled. We call with a different compiler. In this case, the main goal is that this process meta-decompilation. Meta-decompilation is a the decompiled code is syntactically correct and can be re- novel approach for decompilation: 1) it leverages the natu- compiled. Some other applications of decompilation with ral diversity of existing decompilers by merging the results slightly different criteria include clone detection [34], mal- of different decompilers 2) it is able to provide decompiled ware analysis [46,7] and software archaeology [35]. sources for classes that no decompiler in isolation can han- ∗ Corresponding author dle. We implement this approach in a novel meta-decompiler [email protected] (N. Harrand); [email protected] (C. called Arlecchino. [email protected] Soto-Valero); (M. Monperrus); Our results have important implications: 1) for all users [email protected] (B. Baudry) ORCID(s): 0000-0002-2491-2771 (N. Harrand); of decompilation, our paper shows significant differences be- 0000-0003-0541-6411 (C. Soto-Valero); 0000-0003-3505-3383 (M. tween decompilers and provide well-founded empirical evi- Monperrus); 0000-0002-4015-4640 (B. Baudry) dence to choose the best ones; 2) for researchers in decom-
Harrand et al.: Preprint submitted to Elsevier Page 1 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
pilation, our results show that the problem is not solved; 1 class Utils { 3) for authors of decompilers, our experiments have iden- 2 private static final int RADIX = 16; 3 3 static int digit16(final byte b) throws tified bugs in their decompilers ( have already been fixed, DecoderException { and counting) and our methodology of semantic equivalence 4 final int i = Character.digit((char) b, RADIX); modulo inputs can be embedded in the QA process of all de- 5 if (i == -1) { compilers in the world. 6 throw new DecoderException("Invalid URL encoding: nota valid digit(radix"+ In summary, this paper makes the following contribu- RADIX +"):" + b); tions: 7 } 8 return i; • an empirical comparison of eight Java decompilers ba- 9 } sed on 2041 real-world Java classes, tested by 25019 10 } test cases, identifying the key strengths and limitations Listing 1: Source code of Java class correspondig to of Java bytecode decompilation; org.apache.commons.codec.net.Utils. • meta-decompilation, a novel approach to decompila- 1 class org.apache.commons.codec.net.Utils { tion that leverages decompilers diversity to improve 2 static int digit16(byte) throws decompilation effectiveness; org.apache.commons.codec.DecoderException; _ 3 0: ILOAD 0//Parameter byteb • a tool and a dataset for future research on Java decom- 4 1: I2C 2: BIPUSH 16 pilers publicly available at https://github.com/ 5 6 4: INVOKESTATIC #19//Character.digit:(CI)I castor-software/decompilercmp _ 7 7: ISTORE 1//Variable inti _ 8 8: ILOAD 1 _ 9 9: ICONST m1 _ 2. Background 10 10: IF ICMPNE 37 11 //org/apache/commons/codec/DecoderException In this section, we present an example drawn from 12 13: NEW #17 commons-codec the Apache library. We wish to illus- 13 16: DUP trate information loss during compilation of Java source 14 17: NEW #25//java/lang/StringBuilder 20: DUP code, as well as the different strategies that byte- 15 16 //"Invalid URL encoding: nota valid digit(radix code decompilers adopt to cope with this loss when 16):" they generate source code from bytecode. Listing 1 17 21: LDC #27 shows the original source code of the utility class 18 //StringBuilder."
Harrand et al.: Preprint submitted to Elsevier Page 2 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
syntactic distortion 1 class Utils { decompiled results. This metric, called , 2 private static final int RADIX = 16; allows to measure the differences that go beyond variable 3 static int digit16(byte b) throws DecoderException { names. The description of the metric is as follows: 4 int i = Character.digit((char)b, 16); Definition 2. Syntactic distortion. The minimum number 5 if(i == -1) { of atomic edits required to transform the AST of the original 6 throw new DecoderException("Invalid URL encoding: nota valid digit(radix 16): source code of a program into the AST of the corresponding " + b); decompiled version of it. 7 } else{ 8 return i; In the general case, determining if two program are se- 9 } 10 } mantically equivalent is undecidable. For some cases, the 11 } decompiled sources can be recompiled into bytecode that is equivalent to the original, modulo reordering of the constant Listing 3: Decompilation result of Listing 2 with pool. We call these cases strictly equivalent programs. We Fernflower. measure this equivalence with a bytecode comparison tool 2 1 class Utils named Jardiff. 2 { Inspired by the work of [25] and [47], we check if the de- 3 static int digit16(byte b) compiled and recompiled program are semantically equiva- 4 throws DecoderException 5 { lent modulo inputs. This means that for a given set of inputs, 6 int i = Character.digit((char)b, 16); the two programs produce equivalent outputs. In our case, 7 if(i == -1) we select the set of relevant inputs and assess equivalence 8 throw new DecoderException((new StringBuilder()).append("Invalid URL based on the existing test suite of the original program. encoding: nota valid digit(radix Definition 3. Semantic equivalence modulo inputs. We 16):").append(b).toString()); call a decompiled program semantically equivalent modulo 9 else 10 return i; inputs to the original if it passes the set of tests from the orig- 11 } inal test suite. 12 private static final int RADIX = 16; } 13 In the case where the decompiled and recompiled pro- Listing 4: Decompilation result of Listing 2 with Dava. gram produce non-equivalent outputs, that demonstrates that the sources generated by the decompiler express a differ- Besides, this does not include the different formatting ent behavior than the original. As explained by Hamilton choices made by the decompilers such as new lines place- and colleagues [15], this is particularly problematic as it can ment and brackets usage for single instructions such as if mislead decompiler users in their attempt to understand the and else. original behavior of the program. We refer to theses cases as deceptive decompilation results. 3. Decompiler evaluation methodology Definition 4. Deceptive decompilation: Decompiler output that is syntactically correct but not semantically equivalent In this section, we introduce definitions, metrics and re- to the original input. search questions. Next, we detail the framework to compare decompilers and we describe the Java projects that form the 3.2. Research Questions set of case studies for this work. We elaborated five research questions to guide our study 3.1. Definitions and Metrics on the characteristics of modern Java decompilers. The value of the results produced by decompilation RQ1: To what extent is decompiled Java code syn- varies greatly depending on the intended use of the gener- tactically correct? In this research question, we investigate ated source code. In this work, we evaluate the decompilers’ the effectiveness of decompilers for producing syntactically capacity to produce a faithful retranscription of the original correct and hence recompilable source code from bytecode sources. Therefore, we collect the following metrics. produced by the javac and ecj compilers. RQ2: To what extent is decompiled Java code seman- Definition 1. Syntactic correctness. The output of a de- tically equivalent modulo inputs? Le and colleagues [25] compiler is syntactically correct if it contains a valid Java propose to use equivalence modulo inputs assessment as a program, i.e. a Java program that is recompilable with a way to test transformations that are meant to be semantic Java compiler without any error. preserving (in particular compilation). In this research ques- When a bytecode decompiler generates source code that tion, we adapt this concept in the context of decompilation can be recompiled, this source code can still be syntactically testing. In this paper we rely on the existing test suite instead different from the original. We introduce a metric to measure of generating inputs. the scale of such a difference according to the abstract syn- 2https://github.com/scala/jardiff tax tree (AST) dissimilarity[9] between the original and the
Harrand et al.: Preprint submitted to Elsevier Page 3 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
Original Decompiled
decompiler produces (b) syntactically correct code. a Syntactic distortion Then, we measure (c) the textual difference between b the original and the recompiled bytecode. If there are Syntactic correctness
Source code none, the decompiler produced semantically equiva- lent code, otherwise we cannot assess anything in that 1 2 3 regard yet. Compile Decompile Recompile 4. Run the test cases on the recompiled bytecode. If the
tests are successful, we mark the source as passTests for the given triplet, showing that the decompiler pro- 0111 c 0111
Bytecode 1001 Bytecode difference 1001 duces (d) semantically equivalent code modulo inputs. 0110 0110
Original Recompiled If one of these steps fails we do not perform the follow- 4 ing steps and consider all the resulting metrics not available. Test As decompilation can generate a program that does not stop, we set a 20 minutes timeout for the execution of the test suite d Semantic equivalence modulo inputs (the original test suites run under a minute on the hardware used for this experiment, a Core i5-6600K with 16GB of Figure 1: Java decompiler assessment pipeline with four evalu- RAM). ation layers: syntactic distortion, bytecode difference, syntactic The tests used to assess the semantic equivalence modulo correctness, and semantic equivalence modulo input. inputs are those of the original project that cover the given Java file.3 We manually excluded the tests that fail on the original project (either flaky or because versioning issue). RQ3: To what extent do decompilers produce decep- The list of excluded tests is available as part of our experi- tive decompilation results? In this research question, we ments. investigate the cases where we observe semantic differences between the original source code and the outputs of the de- 3.4. Study Subjects compilers. Decompilers. Table 1 shows the set of decompilers un- RQ4: What is the syntactic distortion of decompiled der study. We have selected Java decompilers that are (i) code? Even if decompiled bytecode is ensured to be syntac- freely available, and (ii) have been active in the last two tically and semantically correct, syntactic differences may years. We add Jode in order to compare our results with remain as an issue when the purpose of decompilation is hu- a legacy decompiler, and because the previous survey by man understanding. Keeping the decompiled source code Hamilton and colleagues considers it to be one of the best free of syntactic distortions is essential during program com- decompilers [15]. prehension, as many decompilers can produce human un- The column VERSION shows the version used (some de- readable code structures. In this research question, we com- compilers do not follow any versioning scheme). We choose pare the syntactic distortions produced by decompilers. the latest release if one exists, if not the last commit avail- RQ5: To what extent do the successes and failures of able the 09-05-2019. The column STATUS indicates the date decompilers overlap? In this research question we inves- of the last commit or "Active" if the last commit was more tigate the intersection of classes for which each decompiler recent than 30 days. The column #COMMITS represents the produce semantically equivalent modulo input sources. number of commits in the decompiler project, in cases where the decompiler is a submodule of a bigger project (e.g. Dava 3.3. Study Protocol and Fernflower) we count only commits affecting the sub- Figure 1 represents the pipeline of operations conducted module. The column #LOC is the number of lines of code in on every Java source file in our dataset. For each triplet
Harrand et al.: Preprint submitted to Elsevier Page 4 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
Table 1 Table 2 Characteristics of the studied decompilers. Characteristics of the projects used to evaluate decompilers. Decompiler Version Status #Commits #LOC Project name Java version #Classes #Tests #LOC CFR [2] 0.141 Active 1433 52098 Bukkit 1.6 642 906 60800 Dava [19] 3.3.0 2018-06-15* 14 22884 Commons-codec 1.6 59 644 15087 Fernflower [20] NA** Active 453 52118 Commons-collections 1.5 301 15067 62077 JADX [38] 0.9.0 Active 970 55335 Commons-imaging 1.5 329 94 47396 JD-Core [6] 1.0.0 Active NA*** 36730 Commons-lang 1.8 154 2581 79509 Jode [16] 1.1.2-pre1 2004-02-25* NA*** 30161 DiskLruCache 1.5 3 61 1206 Krakatau [39] NA** 2018-05-13* 512 11301 JavaPoet* 1.6 2 60 934 Procyon [40] 0.5.34 Active 1080 122147 Joda time 1.5 165 4133 70027 . * Date of last update. Jsoup 1 5 54 430 14801 JUnit4 1.5 195 867 17167 ** Not following any versioning scheme. Mimecraft 1.6 4 14 523 *** CVS not available at the date of the present study. Scribe Java 1.5 89 99 4294 Spark 1.8 34 54 4089 Java 5 and beyond and javac, shares its test suite with CFR. DcTest** 1.5 − 1.8 10 9 211 Total 2041 25019 378121 Fernflower [20] is the decompiler embedded in IntelliJ IDE. Krakatau (*) Formerly named JavaWriter. [39] up to Java 7 does not currently support Java 8 (**) Examples collected from previous decompilers evaluation. or invokedynamic. JD-Core [6] is the engine of JD-GUI. It supports Java 1.1.8 to Java 12.0. The version we study through bytecode instrumentation with a tool named yajta.8 in this work is the first version released since the complete rewrite of JD-Core. While older versions were based on a 4. Experimental Results simple bytecode pattern recognition engine, JD-Core now includes a CFG analysis layer. JADX [38] is a decompiler 4.1. RQ1: (syntactic correctness) To what extent is that originally targeted dex files (bytecode targeting the an- decompiled Java code syntactically correct? droid platform) but can also target class files, as in our exper- This research question investigates to what extent the iments. Dava [19] is a decompiler built on top of the Soot source code produced by the different decompilers is syntac- Framework [42]. It does not target Java bytecode produced tically correct, meaning that the decompiled code compiles. by any specific compiler nor from any specific language, but We also investigate the effect of the compiler that produces produces decompiled sources in Java. Soot supports byte- the bytecode on the decompilation results. To do so, in this code and source code up to Java 7. Jode [16] is a legacy section, we focus on the 1887 classes that compile with both decompiler that handles Java bytecode up to Java 1.4. javac and ecj. Projects. In order to get a set of real world Java projects Figure 2 shows the ratio of decompiled classes that are to evaluate the eight decompilers, we reuse the set of projects syntactically correct per pair of compiler and decompiler. of Pawlak and colleagues[32]. To these 13 projects we added The horizontal axis shows the ratio of syntactically correct a fourteenth one named DcTest made out of examples col- output in green, the ratio of syntactically incorrect output in lected from previous decompiler evaluations [15, 23].6 Ta- blue, and the ratio of empty output in red (an empty output ble 2 shows a summary of this dataset: the Java version in occurs, e.g. when the decompiler crashes). The vertical axis which they are written, the number of Java source files, the shows the compiler on the left and decompiler on the right. number of unit tests as reported by Apache Maven, and the For example, Procyon, shown in the last row, is able to pro- number of Java lines of code in their sources. duce syntactically correct source code for 1609 (85.3%) class As different Java compilers may translate the same files compiled with javac, and produce a non-empty syntacti- sources into different bytecode representations [5]7, we ex- cally incorrect output for 278 (14.7%) of them. On the other periment with the two most used Java compilers: javac hand, when sources are compiled with ecj, Procyon gener- and ecj (versions 1.8.0_17 and 13.13.100, respectively). ates syntactically correct sources for 1532 (81.2%) of class We compiled all 14 projects with both compilers (except files and syntactically incorrect for 355 (18.8%) sources. In commons-lang which failed to build with ecj). Our dataset other words, Procyon is slightly more effective when used includes 3928 bytecode classes, 1887 of which compiled against code compiled with javac. It is interesting to notice with ecj, and 2041 compiled with javac. As we study the in- that not all decompiler authors have decided to handle er- fluence of the compiler, in RQ1, we limit our datasets to the ror the same way. Both Procyon and Jode’s developers have 1887 classes that compiled with both compilers. As seman- decided to always return source files, even if incomplete (for tic equivalence modulo inputs is based on test suites, for RQ2 our dataset). Additionally, when CFR and Procyon detect a and RQ3 we focus on the classes that contain code executed method that they cannot decompile properly, they may re- by test suites: 2397 classes generated by the two compilers. place the body of the method by a single throw statement These classes covered by the test suites exclude interfaces and comment explaining the error. This leads to syntacti- as they do not contain executable code. Most enum declara- cally correct code, but not semantically equivalent. tions fall under the same category. Test coverage is assessed The ratio of syntactically correct decompiled code 85.7% Procyon javac 6http://www.program-transformation.org/Transform/ ranges from for on inputs (the best), JavaDecompilerTests down to 44% for Krakatau on ecj (the worst). All decom- 7https://www.benf.org/other/cfr/ 8https://github.com/castor-software/yajta eclipse-differences.html
Harrand et al.: Preprint submitted to Elsevier Page 5 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
Category Recompile Doesn't recompile Empty 1 //Bytecode 2 NEW Lang$LangRule javac 83.4% 12.9% 3.7% CFR 3 DUP ecj 73.9% 26% 0.1% 4 ALOAD 10//pattern Dava javac 44.4% 46.9% 8.6% 5 NEW HashSet ecj 44.5% 48.3% 7.2% 6 DUP Fernflower javac 67.9% 28.4% 3.7% 7 ALOAD 11//langs
ecj 67.7% 29.4% 3% 8 INVOKESTATIC Arrays.asList invokespecial HashSet.
JADX 9 javac 68.7% 27.4% 3.8% 10 ILOAD 12// accept ecj 70.6% 29.2% 0.2% ACONST_NULL
JD − Core 11 javac 69.8% 24.9% 5.2% Compiler 12 -aaINVOKESPECIAL Lang$LangRule.
javac 65.2% 34.8% Jode 14 +aaINVOKESPECIAL Lang$LangRule.
Krakatau 16 javac 44.8% 47.9% 7.3% //Decompiled sources ecj 44% 52% 4% 17 //Usage of private static inner class LangRule
Procyon 18 javac 85.7% 14.3% constructor's wrapper generated by the compiler ecj 81.2% 18.8% 19 -aanew LangRule(pattern, new HashSet
Harrand et al.: Preprint submitted to Elsevier Page 6 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
parameters. Since the value of this parameter is never used, 1 -aaaIFEQ L2 null is passed as additional parameter when the wrapper is 2 +aaaIFNE L2 called. ecj does almost the same thing, but the additional 3 +aaaGOTO L0 Lang$LangRule Procyon 4 + L2 parameter is of type . is able to ALOAD 5 javac ecj 5 reverse the transformation correctly, but not the 6 INVOKESTATIC Lang$LangRule.access$100 one. It decompiles the ecj version literally, conserving the (LLang$LangRule;)Z IFEQ L3 null parameter. Yet, the synthetic wrapper does not exist in 7 8 ALOAD 3 the decompiled sources. Consequently, the decompiled code 9 ALOAD 5 that refers to an absent constructor is syntactically incorrect. 10 INVOKESTATIC Lang$LangRule.access$200 This example illustrates the decompilation challenge intro- (Lang$LangRule;)Ljava/util/Set; -aaaINVOKEINTERFACE Set.retainAll (LCollection;)Z duced by synthetic elements that are generated by a com- 11 12 +aaaINVOKEVIRTUAL HashSet.retainAll (LCollection;)Z piler. Note that synthetic elements in bytecode do carry a 13 (itf) flag indicating their nature. But, (i) it does not change the 14 POP difficulty of reversing an unforeseen pattern and (ii) this flag 15 -aaaGOTO L2 +aaaGOTO L0 could be abused by an obfuscator as it does not change the 16 semantic of the bytecode but rather gives indication to drive Listing 7: Excerpt of bytecode from class org/apache/ potential tools modifying the bytecode. In the case of Pro- commons/codec/language/bm/Lang.class, cyon and CFR, their common test suite did not include code compiled with javac and decompiled with CFR: Lines in generated by ecj at the time of this experiment. Since then, red, marked with a -, are in the original bytecode, while the author of CFR has updated the test suite common to CFR lines in green, marked with a +, are from the recompiled and Procyon with test covering bytecode from ecj10 sources. We have shown that the compiler impact decompilation effectiveness for two reasons: (i) it changes the oracle for 4.2. RQ2: (semantic equivalence) To what extent syntactic correctness; (ii) it produces different bytecode is decompiled Java code semantically structure that decompilers might not expect. equivalent modulo inputs? 2 To assess the significance of this impact, we use a test To answer this research question, we focus on the 2397 on the ratio of classfiles decompiled into syntactically cor- javac javac ecj class files, regrouping bytecode generated by both and rect source code depending on the compiler, versus . ecj, that are covered by at least one test case. This ex- The compiler variable has an impact for three decompilers 99% cludes all types which contain no executable code, such as and no impact for the remaining five, with confidence interfaces. When decompilers produce sources that compile, level. The test rejects that the compiler has no impact on the we investigate the semantic equivalence of the decompiled decompilation syntactic correctness ratio for CFR, Procyon JD-Core 10−14 0.00027 0.006444 source and their original. To do so, we split recompilable and (p-value , and ). For outputs in three categories: (i) semantically equivalent: the the five other decompilers we do not observe a significant code is recompiled into bytecode that is strictly identical to difference between javac and ecj (p-values: Dava 0.15, Fer- nflower 0.47 JADX 0.17 Jode 0.50 Krakatau 0.09 the original (modulo reordering of the constant pool, as ex- , , , and ). plained in Section 3.1), (ii) semantically equivalent modulo Note that beyond syntactic correctness, the compiler may inputs: the output is recompilable and passes the original impact the correctness of the decompiled code, this is dis- project’s test suite (i.e. we cannot prove that the decom- cussed in more details in Section 4.3. semantically Procyon CFR piled code is semantically different), and (iii) To sum up, and are the decompilers different: the output is recompilable but it does not pass the that score the highest on syntactic correctness. The three original test suite (deceptive decompilation, as explained in decompilers ranking the lowest are Jode, Krakatau and Dava Definition4). . It is interesting to note that those three are no longer Let us first discuss an example of semantic equivalence actively maintained. of decompiled code. Listing 7 shows an example of bytecode that is different when decompiled-recompiled but equivalent Answer to RQ1: No single decompiler is able to produce 85.7% modulo inputs to the original. Indeed, we can spot two dif- syntactically correct sources for more than of class ferences: the control flow blocks are not written in the same files in our dataset. The implication for decompiler users order (L2 becomes L0) and the condition evaluated is re- is that decompilation of Java bytecode cannot be blindly versed (IFEQ becomes IFNEQ), which leads to an equivalent applied and does require some additional manual effort. control flow graph. The second difference is that the type of Only few cases make all decompilers fail, which suggests a variable originally typed as a Set and instantiated with an that using several decompilers in conjunction could help HashSet has been transformed into a variable typed as an to achieve better results. HashSet, hence once remainAll is invoked on the variable INVOKEINTERFACE becomes directly INVOKEVIRTUAL. 10Commit: https://github.com/leibnitz27/cfr_tests/ commit/b4f0b01e34a953a1fd57e52f508c9c02c58e6dee Discus- This example illustrates how bytecode may be changed (here sion: https://github.com/leibnitz27/cfr/issues/50 with a slightly different but equivalent control flow graph and a change in the precision of the type information) yet still be
Harrand et al.: Preprint submitted to Elsevier Page 7 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
Equivalence Modulo Input Strict
2397
2000 78% 71% 591 60% 599 59% 57% 327 48% 1000 853 835 # Classes 565 32% 30% 1278 1114 1032 589 537 582 573 596 173 187 0
CFR JADX Jode Dava Procyon JD−Core Krakatau Fernflower Decompiler Figure 3: Equivalence results for each decompiler on all the 2397 classes of the studied projects covered by at least one test, (both compilers combined). Figure 4: Deceptive decompilation results per decompiler. From left to right, deceptive decompilation occurring after Table 3 javac compilation, ecj and both. Cause of non-equivalence for each decompiler.
to blindly trust the behavior of decompiled code. Procyon CFR Fernflower JADX JD-Core Jode Dava Krakatau #Deceptive 33 22 21 78 44 142 36 97 #!Recompile 495 662 941 911 994 1094 1599 1576 4.3. RQ3: (bug finding) To what extent do #Failures 528 684 962 989 1038 1236 1635 1673 decompilers produce deceptive decompilation results? semantically equivalent modulo inputs. As explained by Hamilton and colleagues [15], while Now we discuss the results globally. Figure 3 shows the a syntactically incorrect decompilation output may still be recompilation outcomes of decompilation regarding seman- useful to the user, syntactically correct but semantically dif- tic equivalence for the 2397 classes under study. The hori- ferent output is more problematic. Indeed, this may mis- zontal axis shows the eight different decompilers. The ver- lead the user by making her believe in a different behaviour deceptive tical axis shows the number of classes decompiled success- than the original program has. We call this case decompilation fully. Strictly equivalent output is shown in blue, equiva- (as explained in Definition4). When such lent classes modulo input are shown in orange. For example, cases occur, since the decompiler produces an output that is CFR (second bar) is able to correctly decompile 1713 out of semantically different from what is expected, they may be 2397 classes (71%), including 1114 classes that are recom- considered decompilation bugs. pilable into strictly equivalent bytecode, and 599 that are re- Figure 4 shows the distribution of bytecode classes that compilable into equivalent bytecode modulo inputs. When are deceptively decompiled. Each horizontal bar groups de- the decompiler fails to produce an output that is semantically ceptive decompilation per decompiler. The color indicates equivalent modulo inputs, it is either because the decompiled which compiler was used to produce the class file triggering sources were not syntactically correct and did not recompile, the error. In blue is the number of classes leading to a de- javac or because they did recompile but did not pass the original compilation error only when compiled with , in green ecj test suite (deceptive decompilation). Table 3 gives this in- only when compiled with , and in pink is the number of formation for each decompiler. classes triggering a decompilation error with both compil- The three decompilers that are not actively maintained ers. The sum of these classes is indicated by the total on the any more (Jode, Dava and Krakatau) handle less than 50% right side of each bar. Note that the bars in Figure 4 represent of the cases correctly (recompilable and pass tests). On the the number of bug manifestations, which are not necessarily other hand, Procyon and CFR have the highest ratio of equiv- distinct bugs: the same decompiler bug can be triggered by alence modulo inputs of 78% and 71%, respectively. different class files from our benchmark. Also, Figure 4 plots the same classes referred as Deceptive in Table 3, but in Answer to RQ2: The number of classes for which the Table 3 classes leading to a deceptive decompilation for both decompiler produces equivalent sources modulo input compilers are counted twice. Jode varies significantly from one decompiler to another. The Overall, is the least reliable decompiler, with 83 Fer- result of decompilation is usually not strictly identical decompilation bug instances in our benchmark. While nflower to the original source code. Five decompilers generate produces the least deceptive decompilations on our 13 CFR equivalent modulo input source code for more than 50% benchmark ( ), it is interesting to note that produces 14 of the classes. For end users, it means that the state of only one more deceptive decompilation ( ) but that cor- the art of Java decompilation does not guarantee seman- responds to fewer bugs per successful decompilation. This CFR tically correct decompilation, and care must be taken not makes the most reliable decompiler on our benchmark. We manually inspected 10 of these bug manifestations.
Harrand et al.: Preprint submitted to Elsevier Page 8 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
1 public final class Bukkit { 1 public static setServer(Lorg/bukkit/Server;)V 2 private static Server server; 2 -aaaGETSTATIC org/bukkit/Bukkit.server : 3 [...] 3 -aaaLorg/bukkit/Server; 4 public static void setServer(Server server) { 4 +aaaALOAD 0 5 -aaaaaif (Bukkit.server != null){ 5 IFNULL L0 6 +aaaaaif (server != null){ 6 NEW java/lang/UnsupportedOperationException 7 throw new UnsupportedOperationException( 7 DUP 8 "Cannot redefine singleton Server"); 8 ATHROW 9 } 9 L0 10 -aaaaaBukkit.server = server; 10 ALOAD 0 11 +aaaaaserver = server; 11 -aaaPUTSTATIC org/bukkit/Bukkit.server : 12 [...] 12 -aaaLorg/bukkit/Server; 13 } 13 +aaaASTORE 0 ALOAD 0 org.bukkit.Bukkit 14 Listing 8: Exerpt of differences in 15 INVOKEINTERFACE org/bukkit/Server.getLogger original (in red marked with a -) and decompiled with ()Ljava/util/logging/Logger; (itf) JADX sources (in green marked with a +). 16 NEW java/lang/StringBuilder Listing 9: Exerpt of bytecode from class org/bukkit/ 2 of them were already reported by other users. We reported Bukkit.class compiled with javac: Lines in red, the other 8 to the authors of decompilers.11 The sources marked with a -, are in the original bytecode, while lines in of errors include incorrect cast operation, incorrect control- green, marked with a +, are from the recompiled sources flow restitution, auto unboxing errors, and incorrect refer- (decompiled with JADX). ence resolution. Below we detail two of these bugs. 1 protected StringBuffer applyRules(final Calendar 4.3.1. Case study: incorrect reference resolution calendar, final StringBuffer buf) { -aaaaareturn (StringBuffer) applyRules(calendar, org.bukkit.Bukkit 2 We analyze the class from the 3 -aaaaaaaaaaaaaaaaaaaa(Appendable) buf); Bukkit project. An excerpt of the original Java source code 4 +aaaaareturn this.applyRules(calendar, buf); is given in Listing 8. The method setServer implements 5 } Bukkit.server 6 a setter of the static field . This is an im- 7 private B applyRules(final plementation of the common Singleton design pattern. In Calendar calendar, final B buf) {...} the context of method setServer, server refers to the pa- Listing 10: Excerpt of differences in FastDatePrinter rameter of the method, while Bukkit.server refers to the original (in red marked with a -) and decompiled with static field of the class Bukkit. Procyon sources (in green marked with a +). When this source file is compiled with javac, it pro- org/bukkit/Bukkit.class duces a file containing the 4.3.2. Case study: Down cast error bytecode translation of the original source. Listing 9 shows an excerpt of this bytecode corresponding to the setServer Listing 10 illustrates the differences between the orig- org/apache/commons/lang3/time/ method (including lines are filled in red, while excluding inal sources of FastDatePrinter lines are filled in green) and the decompiled sources produced Procyon When using the JADX decompiler on org/bukkit/- by . The line in red is part of the original, while the Bukkit.class it produces decompiled sources, of which line in green is from the decompiled version. In this exam- applyRules an excerpt is shown in Listing 8. In this example, the de- ple, method is overloaded, i.e. it has two im- StringBuffer compiled code is not semantically equivalent to the origi- plementations: one for a parameter and one Appendable Appendable nal version. Indeed, inside the setServer method the ref- for a generic parameter ( is an in- StringBuffer erences to the static field Bukkit.server have been sim- terface that implements). The implementa- StringBuffer buf Appendable plified into server which is incorrect in this scope as the tion for down casts into , Appendable parameter server overrides the local scope. In the byte- calls the method handling and casts the result StringBuffer code of the recompiled version (Listing 9, including lines are back to . In a non-ambiguous context, it is Appendable filled in green), we can observe that instructions accessing perfectly valid to call a method which takes and writing the static field (GETSTATIC, PUTSTATIC) have arguments on an instance of a class that implements that been replaced by instructions accessing and writing the local interface. But in this context, without the down cast to Appendable variable instead (ALOAD, ASTORE). , the Java compiler will resolve the method call applyRules When the test suite of Bukkit runs on the recompiled to the most concrete method. In this case, this applyRules StringBuffer bytecode, the 11 test cases covering this code fail, as the first will lead for to call itself in- access to setServer will throw an exception instead of nor- stead of the other method. When executed, this will lead to mally initializing the static field Bukkit.server. This is an infinite recursion ending in a StackOverflowError. There- Procyon clearly a bug in JADX. fore, in this example, changes the behaviour of the decompiled program and introduces a bug in it. 11https://github.com/castor-software/decompilercmp/ tree/master/funfacts
Harrand et al.: Preprint submitted to Elsevier Page 9 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
0.08 Procyon 1 public class Foo { 2 public int foo(int i, int j) { 0.15 Krakatau 3 while(true){ try{ 0.09 4 Jode 5 while (i < j) i = j++ / i; 0.05 6 +aaaaareturn j; JD−Core 7 } catch (RuntimeException re) { 0.07 8 i = 10; JADX
Decompiler 9 -aaaaacontinue; 0.08 Fernflower 10 } 11 -aaabreak; 0.16 Dava 12 } 13 -aareturn j; 0.05 CFR 14 } 15 } 0.0 0.1 0.2 0.3 0.4 0.5 distanceToOriginal/nbNodesOriginal Listing 11: Excerpt of differences in Foo original (in red Figure 5: Distribution of differences between the original and marked with a -) and decompiled with Fernflower (in green the decompiled source code ASTs. Green diamonds indicate marked with a +) sources. average. source language nor the compiler used to produce the byte- Answer to RQ3: Our empirical results indicate that no code it decompiles. [28] This partly explains the choice of decompiler is free of deceptive decompilation bugs. The its author to not reverse some optimizations made by Java developers of decompilers may benefit from the equiva- compilers (See example introduced in Section 2.). lent modulo input concept to find bugs in the wild and Listing 11 shows the differences on the resulting source extend their test base. Two bugs found during our study code after decompiling the Foo class from DcTest with Fer- have already been fixed by the decompiler authors, and nflower. As we can observe, both Java programs represent three others have been acknowledged. a semantically equivalent program. Yet, their ASTs contain substantial differences. For this example, the edit distance is 3∕104 MOVE return 4.4. RQ4: (ASTs difference) What is the syntactic as it contains three tree edits: the node, and DELETE the break node and the continue node (the distortion of decompiled code? original source’s AST contained 104 nodes). The quality of decompilation depends not only on its Note that some decompilers perform some transforma- syntactic correctness and semantic equivalence but also on tions on the sources they produce on purpose to increase how well a human can understand the behaviour of the de- readability. Therefore, it is perfectly normal to observe compiled program. The code produced by a decompiler may some minimal syntactic distortion, even for decompilers be syntactically and semantically correct but yet hard to read producing readable sources. But as our benchmark is for a human. In this research question, we evaluate how far composed of non obfuscated sources, it is expected that a the decompiled sources are from the original code. We mea- readable output will not fall too far from the original. sure the syntactic distortion between the original and the de- compiled sources as captured by AST differences (Definition Answer to RQ4: 2). All decompilers present various de- Figure 5 shows the distribution of syntactic distortion grees of syntactic distortion between the original source present in syntactically correct decompiled code, with one code and the decompiled bytecode. This reveals that violin plot per decompiler. The green diamond marks the all decompilers adopt different strategies to craft source average syntactic distortion. For example, the syntactic dis- code from bytecode. We propose a novel metric to quan- tortion values of the Jode decompiler have a median of 0.05, tify the distance between the original source code and its average of 0.09, 1st-Q and 3rd-Q of 0.01 and 0.11, respec- decompiled counterpart. Also, decompiler users can use tively. In this figure, lower is better: a lower syntactic dis- this analysis when deciding which decompiler to employ. tortion means that the decompiled sources are more similar to their original counterparts. 4.5. RQ5: (Decompiler Diversity) To what extent CFR JD-Core and introduce the least syntactic distor- do the successes and failures of decompilers tion, with high proportion of cases with no syntactic dis- overlap? tortion at all (as we exclude renaming). Their median and average syntactic distortion are close to 0.05, which corre- In the previous research questions, we observe that dif- sponds to 5 edits every 100 nodes in the AST of the source ferent decompilers produce source code that varies in terms program. On the other extreme, Dava and Krakatau intro- of syntactic correctness, semantic equivalence and syntac- duce the most syntactic distortion with average of 16 (resp. tic distortion. Now, we investigate the overlap in successes 15) edits per 100 nodes. They also have almost no cases for and failures of the different decompilers considered for this which they produce sources with no syntactic distortion. It study. is interesting to note that Dava makes no assumptions on the Figure 6 shows a Venn Diagram of semantically equiv-
Harrand et al.: Preprint submitted to Elsevier Page 10 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
Table 4 Summary results of the studied decompilers Decompiler #Recompilable #PassTest #Deceptive CFR 3097 (0.79) 1713 (0.71) 22 Dava 1747 (0.44) 762 (0.32) 36 Fernflower 2663 (0.68) 1435 (0.60) 21 JADX 2736 (0.70) 1408 (0.59) 78 JD-Core 2726 (0.69) 1375 (0.57) 44 Jode 2569 (0.65) 1161 (0.48) 142 Krakatau 1746 (0.44) 724 (0.30) 97 Procyon 3281 (0.84) 1869 (0.78) 33 Union 3734 (0.95) 2240 (0.93) 342 Total 3928 (1.00) 2397 (1.00)-
both the type of parent and the type required by the super constructor can be resolved, but, it does make it more chal- lenging to decompilers in practice. A manual analysis of these classes shows common is- sues among the studied decompilers. (i) Generics is a fea- ture that causes many decompilers to fail in particular when combined with ternary operators, wildcards or type bounds. Figure 6: Venn diagram of syntactically and semantically Another example of such a case is detailed in Section 5. (ii) equivalent modulo inputs decompilation results. As mentioned in Section 4.1 and Section 4.3, compilers pro- ducing the bytecode do play a role. In particular, synthetic 1 protected static class KeySetIterator
Harrand et al.: Preprint submitted to Elsevier Page 11 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
Answer to RQ5: The classes for which each decompi- 1 public class YamlConfiguration extends FileConfiguration { ler produce semantically equivalent source code modulo _ 2 protected static final String COMMENT PREFIX ="# input do not overlap completely. For 6 out of 8 decompil- "; 1 _ ers, there exists at least class for which the decompiler 3 protected static final String BLANK CONFIG = "{} is the only one to produce semantic equivalence modulo 4 "; inputs sources. In theory, a union of the best features of private final DumperOptions yamlOptions; 2240 2397 93% 5 each decompiler would cover out of the ( ) 6 private final Representer yamlRepresenter; classes of the dataset. This suggests to combine multiple 7 private final Yaml yaml; decompilers to improve decompilation results. 8 9 public YamlConfiguration() 10 { 11 DumperOptions r7; 5. Meta Decompilation 12 YamlRepresenter r8; 13 YamlConstructor r9; In this section, we present an original concept for decom- 14 Yaml r10; pilation. 15 BaseConstructor r11; 16 r7 = new DumperOptions(); 5.1. Overview 17 yamlOptions = r7; 18 r8 = new YamlRepresenter(); In 1995, Selberg et al. [37] noticed that different web 19 yamlRepresenter = r8; search engines produced different results for the same in- 20 r9 = new YamlConstructor(); r11 = (BaseConstructor) r9; put query. They exploited this finding in a tool called 21 22 r10 = new Yaml(r11, yamlRepresenter, METACRAWLER, which delegates a user query to various yamlOptions); search engines and merges the results. This idea of combin- 23 yaml = r10; ing diverse tools that have the same goal has been explored 24 } [...] since then. For example, Blair and Somayaji [11] explore 25 26 } how a genetic algorithm can recombine related programs at org.bukkit.configuration. the object file level to produce correct variants of C pro- Listing 13: Excerpt of file.YamlConfiguration Dava grams. Persaud et al. [33] combines cryptographic libraries decompiled with . together for software security. Chen et al. [4] rely on vari- 1 public class YamlConfiguration extends ous fuzzers to build an ensemble based fuzzer that gets better FileConfiguration { _ performance and generalization ability than that of any con- 2 protected static final String BLANK CONFIG = "{}\n"; stituent fuzzer alone. _ 3 protected static final String COMMENT PREFIX ="# In this paper, we apply a similar approach to improve ";
Java decompilation. Each decompiler has its strengths and 4 private final Yaml yaml = weaknesses, and the subset of JVM bytecode sequences they 5 new Yaml(new YamlConstructor(), correctly handle is diverse (cf Section 4.5). Therefore, our 6 this.yamlRepresenter, this.yamlOptions); 7 private final DumperOptions yamlOptions = new idea is to combine decompilers in a meta-decompiler. DumperOptions(); Arlecchino In this paper, we propose a tool called , that 8 private final Representer yamlRepresenter = new implements such a ‘meta-decompilation’ approach. Ar- YamlRepresenter(); [...] lecchino merges partially incorrect decompilation results 9 10 } from diverse decompilers in order to produce a correct one. Listing 14: Excerpt of org.bukkit.configuration. 5.2. Example file.YamlConfiguration decompiled with JADX. The class org.bukkit.configuration.file. YamlConfiguration of the project Bukkit is an example static field BLANK_CONFIG is correctly initialized with of a class file that is incorrectly handled by both JADX "{}\n", but the initialization of yaml, yamlOptions and Dava. While both decompilers produce syntactically and yamlRepresenter are conducted out of order, incorrect Java code for this class, the error that prevents which lead to a compilation error as yamlOptions and successful recompilation is not located at the same place in yamlRepresenter are still null when yaml is initialized. both decompiled classes. Intuitively, one can see that Dava’s solution could be fixed by replacing lines 3 and 4 with line 2 from JADX’s solu- Listing 13 shows an excerpt of the decompiled sources tion. This is an example of successful meta-decompilation, produced by Dava for YamlConfiguration. The static merging the output of two decompilers. _ field BLANK CONFIG is initialized with an incorrect string Figure 7 illustrates how two erroneous decompilations literal that contains a non escaped line return. When attempt- can be merged into one that is correct, when the error is not ing to recompile these sources, javac produces an unclosed located at the same place. This figure represents different string literal error for both line 3 and 4. versions of the abstract syntax tree (AST) of a Java class. Listing 14 shows an excerpt of the decompiled The root node corresponds to the class itself, while its chil- sources produced by JADX for the same class. The
Harrand et al.: Preprint submitted to Elsevier Page 12 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
Data: bytecode A bytecode file, Decompilation #1 Decompilation #2 Decompilers A set of decompilers Result: The decompiled java sources corresponding 1 Solutions ← {} Class Class 2 F ragmentStore ← {} 3 foreach dc ∈ Decompilers do 4 solution ← AST (decompile(dc, bytecode)) 5 F ragments ← fragmentsOf(solution) A B C D A B C’ 6 foreach f ∈ F ragments do 7 if ¬problem(f) ∧ signature(f) ∉ Store then 8 F ragmentStore ← F ragmentStore signature f → f Fragment Store Meta Decompilation ∪ { ( ) } 9 end Signature Fragment 10 end A A Class 11 Solutions ← Solutions ∪ {solution} B B 12 foreach s ∈ Solutions do 13 if completable(s, F ragmentStore) then C C 14 if C’ A B C D recompile(complete(s, F ragmentStore)) D D then 15 return print(s) Figure 7: Meta decompilation: Merger of different partial de- ; 16 else compilation. Node in blue with plain border originates from 17 remove(s, F ragmentStore) Decompiler #1, nodes in yellow with dashed border originate from Decompiler #2. Borderless nodes in red contains compi- 18 end lation errors. 19 end 20 end 21 end dren represent type members of the class. A type member Algorithm 1: Meta decompilation procedure. is either a method, a field, a nested type (class, or enum), or a static initialization block. Decompilation #1 rep- resents the AST of the sources produced by one decompi- The bytecode file is passed to the decompiler d. An AST ler, it includes 4 type members (A,B,C and D), and one is built from the decompiled sources (line 4). While build- compilation error located in B. Decompilation #2 rep- ing the AST, the compilation errors and their location are resents the sources produced by a different decompiler for gathered (if any) and the type members containing errors the same class. It contains only 3 type members (A,B an are annotated as such. A class abstract syntax tree includes C’) and one compilation error located in C’. The fragment a node for the class itself as the root, as well as children store is a dictionary containing an error free AST fragment representing class information (super class, super interfaces, for each type member when such a fragment exists. Meta formal type parameters) and type members. Type members Decompilation shows an example of error free AST that include fields, methods, constructors, inner classes, enum can be built based on Decompilation #1 and the store that values, and static blocks. These type members’ source lo- combines AST fragments from both decompilations. Note cations are recorded and compared with the compiler er- that different decompilers may produce sources that do not ror locations. If an error is located between a type member exactly contain the same type members. This is illustrated start and end location, the type member is annotated as er- here by Decompilation #2 not having a type member D rored. For example, the element corresponding to the field _ and having a different signature for C. BLANK CONFIG is annotated as errored in Dava’s solution for YamlConfiguration. This annotated AST, that we call 5.3. Algorithm solution, is added to the set of remaining solutions. Algorithm 1 describes the process of meta decompila- Additionally, for all type members in the current solu- tion as implemented by Arlecchino. Arlecchino takes as in- tion, if the fragment store does not already contain an er- put a bytecode file, and an ordered list of bytecode decom- ror free fragment with the same signature, the type mem- pilers. The process starts with an empty set of solutions and ber is added to the fragment store (line 8). The signa- an empty fragment store of correct fragments. This fragment ture of a type member is a character string that identi- store is a dictionary that associates a type member signature fies it uniquely. For example, the signature of the field _ to a fragment of AST free of compilation error correspond- BLANK CONFIG is org.bukkit.configuration.file. _ ing to the type member in question. YamlConfiguration#BLANK CONFIG and the signature For each decompiler, the meta-decompilation goes of YamlConfiguration’s constructor is org.bukkit. through the following steps. configuration.file.YamlConfiguration().
Harrand et al.: Preprint submitted to Elsevier Page 13 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
Table 5 Table 6 Arlecchino results on classes with no correct decompilation Comparison of Arlecchino results with state of the art. from state of the art decompilers. Arlecchino Union #PassTest 59 (37.6%) 0 (0%) #Deceptive 11 (7.0%) 23 (14.6%) #!Recompile 87 (55.4%) 134 (85.4%) Decompiler #Recompilable #PassTest #Deceptive ASTDiff Total 157 (100%) 157 (100%) CFR 3097 (79%) 1713 (71%) 22 (1.27%) 0.05 Procyon 3281 (84%) 1869 (78%) 33 (1.74%) 0.08 Arlecchino 3479 (89%) 2087 (87%) 30 (1.42%) 0.06 Total 3928 (100%) 2397 (100%)-- Each solution in the set of solutions is checked for com- pletion with the current store (line 12). A solution is “com- 59 157 37.6% pletable” with the members in a given fragment store, if for out of ( ) classes. It produces deceptive de- 11 7.0% all the solution’s type members annotated with an error are compilation for ( ) classes and fails to produce recom- 87 157 55.4% present in the fragment store. Indeed, these type members’ pilable results for out of ( ) classes. The success Arlecchino AST can be replaced with an error free variant present in the case where produces correct output is when: 1) fragment store. If a solution is completable with the current at least one compiler is able to read the correct signature for fragment store, all its type members annotated as errored are all type members of a class and, 2) an error free decompila- replaced with a fragment from the fragment store. The so- tion exists for all of these type members. However, when no lution is then passed to the compiler to check if it compiles. decompiler is able to decompile a specific type member or If it does, it is printed, and the meta decompilation stops. If that no decompiler reads correctly the signature of all type not, the solution is removed from the set of solutions. As the members, no meta decompilation can be successful. These first solution that satisfies the oracle (syntactic correctness) results demonstrate that successful decompilation (in terms stops the process, and this oracle is imperfect, the order of of both syntactic correctness and semantic equivalence mod- Arlecchino the decompilers matters. More details are given in the fol- ulo inputs) can be found by for classes where no lowing section. other decompilers can. Ar- By attempting to repair each solution and its given set Table 6 shows the results obtained when running lecchino of type members with a minimum of transplanted fragments on the whole dataset presented in Section 3 and Procyon CFR Procyon from those available in the fragment store, Arlecchino does compares it with and . is the de- not favour any type member set. This allows Arlecchino to compiler that scores the highest in terms of syntactic correct- deal with cases where the different solutions do not contain ness as well as semantic equivalence modulo inputs, while CFR the same type members. This occurs with implicit construc- scores the lowest in deceptive decompilation rate and tor declarations such as the one present in Listing 14 with syntactic distortion. The first column indicates the num- YamlConfiguration. It also makes it possible to handle ber of classes for which each decompiler produced syntac- 3928 cases where element signatures might differ depending on tically correct sources, among the from the dataset. how type erasure is dealt with by each decompiler. And fi- The second column shows the number of classes for which nally, it handles cases where elements might not be in the each decompiler produced semantically correct modulo in- 2397 same order (and the order of type members is meaningful as puts sources among the classes covered by tests. The seen in Listing 14). third column indicates the number of deceptive decompi- lations produced by each decompiler. The percentage of 5.4. Experimental results about deceptive decompilation is computed with #Deceptive ∕ meta-decompilation (#Deceptive + #PassTests). The last column shows the The following section evaluates the effectiveness of Ar- median syntactic distortion in number of edits per nodes in lecchino. It is organized as follows. First, we gather the 157 the original AST. Arlecchino classes of our dataset for which no decompilers produced se- produced syntactically correct sources for 3479 89%) mantically equivalent modulo input sources and assess the classes ( . It produces semantically equivalent 2087 87% 30 results produced by Arlecchino. Second, we run Arlecchino modulo inputs sources for ( ) classes, and decep- Procyon Arlecchino on the complete dataset of classes in this study. We then tive decompilations. Compared with , 198 evaluate the results with regards to semantic equivalence produces syntactically correct sources for more classes, 218 modulo inputs. Finally, we study the origin of fragments semantically correct modulo inputs sources for more. 3 produced by Arlecchino and discuss the consequences on the It also produces less deceptive decompilations, and has CFR Ar- number of deceptive decompilations. a lower syntactic distortion. Compared with , lecchino 8 Table 5 shows the results of meta decompilation on the produces more deceptive decompilations but it 157 classes of our dataset that led to decompilation errors for produces semantically correct modulo inputs sources for 374 all decompilers in the study and were covered by at least one more classes. In percentage of deceptive decompi- Arlecchino test12. Arlecchino produces semantically equivalent results lation among recompilable decompilation, pro- duces 1.42% of deceptive decompilation which is lower than 12 3 of the 137 classes that led to syntactically incorrect outputs for all Procyon’s 1.74% but slightly higher than CFR’s 1.27%. decompilers are not covered by any tests.
Harrand et al.: Preprint submitted to Elsevier Page 14 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
100 used, meaning that there is no need for meta-decompilation. For 287 classes, Arlecchino combines the output of 2 de- 1922 compilers. It uses 3 and 4 decompilers for 20 classes and 2 75 Main Decompiler classes respectively. The color indicates which decompiler’s CFR (43) base solution was used. In the overwhelming majority, the Fernflower (13) Procyon 50 JD−Core (2) solution is used. Jode (2) % classes Figure 9 shows the distribution of transplanted frag- None (166) ments’ origin for the 309 classes where several decompil- Procyon (2171) 25 ers are used. For 252 classes, one or more fragments from 287 CFR Arlecchino 166 ’s solution were transplanted to build ’s so- 20 2 lution. 55 classes have fragments coming from Fernflower, 0 Ar- 0 1 2 3 4 and the rest of the distribution is negligible. Note that #Decompilers lecchino stops as soon as it finds an admissible solution. Figure 8: Distribution of the number of decompilers used by Thus, the order of decompilers when building a solution Arlecchino. largely impacts this distribution. Arlecchino produces a deceptive decompilation either 100 when the first recompilable solution of a given type mem- 75 252 ber is a deceptive one, or the assembly of different fragments 50 introduces an error. 25 55 In order to minimize these problems, Arlecchino uses % transplantation 1 9 10 6 0 Procyon as the first decompiler and orders the other decom- CFR Fernflower JADX JD−Core Jode Procyon Decompiler pilers by their deceptive decompilation rate. Therefore, most of the decompilers’ deceptive decompi- Figure 9: Distribution of the origin of transplanted fragments Procyon in Arlecchino results. lations are for the same classes as ’s one. In a lesser way, deceptive decompilation originating from type mem- bers decompiled with CFR affect Arlecchino when those type member are decompiled with syntactic errors by Pro- Overall, Arlecchino scores higher than all studied de- cyon. Note that, as no software is free of bugs, the imple- compilers in terms of semantic correctness as well as seman- mentation of Arlecchino could also add new sources of er- tic equivalence modulo inputs, and ranks second in decep- ror. In practice, as shown by Table 6, the number of decep- tive decompilation rate by a small margin. The rate of se- tive decompilations (30) corresponds to a better deceptive mantically equivalent decompilation modulo inputs is higher decompilation rate than all decompilers of this study except because Arlecchino produces, by design, more syntactically CFR. correct decompilations. On the other hand, the rate of de- ceptive decompilation is slightly higher than CFR, as Ar- 5.4.2. Case studies lecchino aggregates some of the deceptive decompilations Here we discuss two examples in details: one successful from all used decompilers. This is, to our knowledge, the and one failed meta decompilation. first implementation of this meta-decompilation approach. It demonstrates the validity of the approach and adds a new state of the art tool that practitioners can use to decompile Success: Request. Listing 15 shows the decompiled Java bytecode. sources for org.junit.runner.Request produced Note that Arlecchino also has its implementation flaws by Procyon. In this example, there are ambiguous and may fail where other decompilers may succeed. In par- references because two types share the same simply ticular, not all AST nodes transplantation produce syntac- qualified name: both org.junit.runners.model and tically correct code. But it may be used in conjunction of org.junit.internal.runners contain a type named other decompilers. The union of classes for which at least InitializationError, therefore the decompiled sources one decompiler (including Arlecchino) produces semanti- generated by Procyon lead to a compilation error. cally equivalent modulo inputs sources, presented in RQ5, Listing 16 shows the decompiled sources for org. now covers 2299 out of 2397 classes (96%) of our dataset. junit.runner.Request produced by CFR. These sources contain an error in the body of the static method 5.4.1. Remaining deceptive decompilations runner(Runner). Since this method contains an anony- In order to investigate deceptive decompilations pro- mous class, when the original sources are compiled, a syn- Arlecchino duced by , we need to investigate the origins of thetic field runner is created, by the compiler, for the the AST fragments used in each decompilation. anonymous class. This field contains the parameter runner Figure 8 shows the distribution of the number of decom- from the enclosing method. When CFR decompiles the byte- Arlecchino 2397 pilers used by for each of the classes of our code, it incorrectly replaces the statement that returns the pa- Arlecchino dataset for which we have tests. finds no solution rameter of the enclosing method by a statement that returns a for 166 classes. For 1922 classes, only one decompiler was
Harrand et al.: Preprint submitted to Elsevier Page 15 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
1 private final Closure iDefault; 1 import org.junit.runners.model.*; 2 2 import org.junit.internal.runners.*; 3 private SwitchClosure(final boolean clone, 3 public abstract class Request { 4 final Predicate[] predicates, 4 [...] 5 final Closure[] closures, 5 public static Request classes(final Computer 6 final Closure defaultClosure) { computer, final Class... classes) { 7 super(); 6 try{ 8 iPredicates = clone ? 7 final AllDefaultPossibilitiesBuilder FunctorUtils.copy(predicates) : builder = new predicates;
AllDefaultPossibilitiesBuilder(true); 9 iClosures = clone ? 8 final Runner suite = FunctorUtils.copy(closures) : closures;
computer.getSuite(builder, classes); 10 -aaaaaiDefault = (Closure) (defaultClosure 9 return runner(suite); 11 -aaaaaaaaa== null ? NOPClosure.
17 return new Request() { 18 @Override field that does not exist in the sources. This leads to a compi- 19 public Runner getRunner() { lation error when attempting to recompile. Since our report, 20 return runner; CFR 13 } ’s author has fixed this bug. 21 Procyon CFR 22 }; While both and ’s solutions contain 23 } an error, these errors are not located on the same 24 } type member. Hence, CFR’s fragment for the method Listing 15: Excerpt of org.junit.runner.Request classes(Computer, Class[]) is transplanted on decompiled with Procyon. Procyon’s solution. Since the pretty printer used by Ar- lecchino only lists imports at a type granularity, and CFR’s fragment contains references that are non-ambiguous, the 1 import org.junit.runners.model.InitializationError; combined solution is recompilable and semantically equiva- 2 lent modulo input. 3 public abstract class Request { 4 [...] Failure: SwitchClosure. 5 There are Java constructs for 6 public static Request classes(Computer computer, which all decompilers struggle. In these cases, all decom- Class ... classes) { pilers may produce an error on the same type member, and try{ 7 this leads to a failed meta-decompilation. The following ex- 8 AllDefaultPossibilitiesBuilder builder = new ample illustrates the problem of generic type lower bounds, AllDefaultPossibilitiesBuilder(true); which challenges all decompilers. 9 Runner suite = computer.getSuite(builder, Listing 17 shows an excerpt of the original sources classes); org.apache.commons.collections4.functors. return Request.runner(suite); for 10 SwitchClosure 11 } . The line highlighted in red is the orig- 12 catch (InitializationError e) { inal line. The line highlighted in green is the correspond- 13 throw new RuntimeException("Bug in saff's ing line as decompiled by Procyon, CFR and JD-Core. brain: Suite constructor, called as above, should always complete"); None of them is able to correctly reproduce the cast to Closure 14 } . This leads to a compilation er- 15 } ror as the method NOPClosure.
Harrand et al.: Preprint submitted to Elsevier Page 16 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
5.5. Discussion Internal validity. The internal threats are related to the In this section we discuss how alternative design deci- metrics employed, especially those used to compare the syn- sions might be applied for meta decompilation. In particu- tactic distortion and semantic equivalence modulo inputs be- lar, we discuss the use of different oracles to choose among tween the original and decompiled source code. Moreover, decompiled fragments and the order of decompilers in Al- the coverage and quality of the test suite of the projects under gorithm 1. study influences our observations about the semantic equiv- A benefit of embedding a compiler in the meta decompi- alence of the decompiled bytecode. To mitigate this threat, ler is that it allows to use many different oracles to pick we select a set of mature open-source projects with good test among the decompiled (and optionally recompiled) frag- suites as study subjects, and rely on state-of-the-art AST and ments. In this paper, we use the syntactic correctness assess- bytecode differencing tools. ment done by the compiler. But it would be possible to use External validity. other oracles. For certain decompilation use cases, such as The external threats refer to what extent source recovery, tests covering the original bytecode could the results obtained with the studied decompilers can be be available. In the case of reverse engineering, it is realistic generalized to other Java projects. To mitigate this threat, to assume that one has access to a set of inputs with known we reuse an existing dataset of Java programs which we outputs for the reversed program. They may be used as a test believe is representative of the Java world. Moreover, we suite. The challenge with this approach is the granularity of added a handmade project which is a collection of classes the oracle provided by test cases. For meta decompilation to used in previous decompilers evaluations as a baseline for work, the granularity of the oracle needs to be finer or equal further comparisons. ecj to the one of transplantation. In this work, we use errors Reliability validity. as we are able to map them to code fragments. This allows Our results are reproducible, the us to label a fragment as correct and incorrect. experimental pipeline presented in this study is publicly Another oracle can be based on the bytecode distance available online. We provide all necessary code to repli- cate our analysis, including AST metric calculations and between the decompiled-then-recompiled fragment and its 14 original bytecode counterpart. This could be considered as a statistical analysis via R notebooks. heuristic to minimize the likelihood of semantic differences between both fragments. In this work we measure bytecode 7. Related work distance with JarDiff, but SootDiff [5] could also be used, as its authors announce that it tolerates some control This paper is related to previous works on bytecode flow graph equivalent transformation. analysis, decompilation and program transformations. In Furthermore, depending on the metric that a decompi- this section, we present the related work on Java bytecode ler user favours, the order of the decompilers used through decompilers along these lines. meta decompilation may change. In this work we rank de- Kerbedroid [18] is the closest related work. The work fo- compilers according to the number of classes for which they cuses on decompilers for Android and starts from the same produce semantically equivalent modulo inputs sources. If observation as ours: decompilers perform differently with a user favours the rate of deceptive decompilation to be as varying applications due to the various strategies to han- low as possible, CFR could be put first. Inverting the order dle information lost in compilation. Kerbedroid is a meta- of Procyon and CFR for Arlecchino, on the 157 classes pre- decompiler that stitches together results from multiple de- sented in Section 4.5, yields only 38 decompiled classes that compilers. Our current work shares the same observation, are semantically equivalent modulo inputs. But it produces while contributing two key novel points. We perform an in- only 4 deceptive decompilations. depth assessment of the different strategies implemented in 8 decompilers, with respect to three quality attributes, in- Highlights about meta-decompilation: cluding equivalence modulo-input to compare the behavior To summarize, Arlecchino we have devised and implemented a novel approach to of decompiled bytecode. leverages the partial re- merge results from different decompilers, called meta- sults from 8 decompilers instead of 3 for Kerbedroid, which decompilation. This tool handles 59 of the 157 cases increases the coverage of various corner cases in the byte- (37.6%) previously not handled by any decompiler. code. Meta-decompilation is, to our knowledge, a radically new The evaluation of decompilers is closely related to the as- idea that has never been explored before. Our experi- sessment of compilers. In particular, Le et al. [25] introduce ments demonstrate the feasibility and effectiveness of the the concept of semantic equivalence modulo inputs to vali- idea. date compilers by analyzing the interplay between dynamic execution on a subset of inputs and statically compiling a program to work on all kind of inputs. Blackburn et al. [3] 6. Threats to Validity propose a set of benchmarking selection and evaluation methodologies, and introduces the DaCapo benchmarks, a In this section, we report about internal, external and re- 14https://github.com/castor-software/decompilercmp/ liability threats against the validity of our results. tree/master/notebooks
Harrand et al.: Preprint submitted to Elsevier Page 17 of 19 Java Decompiler Diversity and its Application to Meta-decompilation set of open source, client-side Java benchmarks. Naeem et ferent results to bypass the shortcomings of single decompil- al. [30] propose a set of software quality metrics aimed at ers. We called this approach ‘meta decompilation‘ and im- measuring the effectiveness of decompilers and obfuscators. plemented it in a tool called Arlecchino. Our experimental In 2009, Hamilton et al. [15] show that decompilation is pos- results show that Arlecchino can produce semantic equiva- sible for Java, though not perfect. In 2017, Kostelansky et lence modulo inputs sources for 37.6% of classes for which, al. [23] perform a similar study on updated decompilers. In previously, no single decompiler could. 2018, Gusarovs [14] performed a study on five Java decom- pilers by analyzing their performance according to different Acknowledgments handcrafted test cases. All those works demonstrate that Java bytecode decompilation is far from perfect. This work has been partially supported by the Wallen- Decompilers are disassemblers are closely related, and berg Autonomous Systems and Software Program (WASP) each pair of binary format, target language poses specific funded by Knut and Alice Wallenberg Foundation and by challenges (see Vinciguerra et al. [44] for C++, Khadra et the TrustFull project funded by the Swedish Foundation for al. [22] for ThumbISA, Grech et al. [13] for Ethereum byte- Strategic Research. code). With dissassembling, types must be reconstructing [41], as well as assignment chains [43]. As we do in this pa- References per, some researchers focus on reassembling disassembled binary code [45,8, 10]. [1] Amin, N., Tate, R., 2016. Java and scala’s type systems are unsound: The existential crisis of null pointers, in: Proceedings of the 2016 A recent trend in decompilation is to use neural networks ACM SIGPLAN International Conference on Object-Oriented Pro- [21, 26, 12]. For example, Katz et al. [21] present a technique gramming, Systems, Languages, and Applications, Association for for decompiling binary code snippets using a model based Computing Machinery, New York, NY, USA. p. 838–848. URL: on Recurrent Neural Networks, which produces source code https://doi.org/10.1145/2983990.2984004, doi:10.1145/ 2983990.2984004 that is more similar to human-written code and therefore . [2] Benfield, L., 2019. CFR. https://www.benf.org/other/cfr/. more easy for humans to understand. This a remarkable at- [Online; accessed 19-July-2019]. tempt at driving decompilation towards a specific goal. La- [3] Blackburn, S.M., Garner, R., Hoffmann, C., Khang, A.M., McKin- comis et al. [24] propose a probabilistic technique for vari- ley, K.S., Bentzur, R., Diwan, A., Feinberg, D., Frampton, D., Guyer, able name recovery. Schulte et al. [36] use evolutionary S.Z., Hirzel, M., Hosking, A., Jump, M., Lee, H., Moss, J.E.B., search to improve and recombine a large population of candi- Phansalkar, A., Stefanović, D., VanDrunen, T., von Dincklage, D., Wiedermann, B., 2006. The dacapo benchmarks: Java benchmarking date decompilations by applying source-to-source transfor- development and analysis, in: Proceedings of the 21st Annual ACM mations gathered from a database of human-written sources. SIGPLAN Conference on Object-Oriented Programming Systems, Miller and colleagues [29] model the uncertainty due to the Languages, and Applications, Association for Computing Machinery, information loss during compilation using probabilities and New York, NY, USA. p. 169–190. URL: https://doi.org/10. 1145/1167473.1167488 10.1145/1167473.1167488 propose a novel disassembly technique, which computes a , doi: . [4] Chen, Y., Jiang, Y., Ma, F., Liang, J., Wang, M., Zhou, C., probability for each address in the code space, indicating its Su, Z., Jiao, X., 2018. EnFuzz: Ensemble Fuzzing with likelihood of being a true positive instruction. Seed Synchronization among Diverse Fuzzers. arXiv e-prints , arXiv:1807.00182arXiv:1807.00182. [5] Dann, A., Hermann, B., Bodden, E., 2019. Sootdiff: Bytecode 8. Conclusion comparison across different java compilers, in: Proceedings of the In this work, we presented a fully automated pipeline to 8th ACM SIGPLAN International Workshop on State Of the Art in Program Analysis, Association for Computing Machinery, New assess Java bytecode decompilers with respect to their ca- York, NY, USA. p. 14–19. URL: https://doi.org/10.1145/ pacity to produce compilable, equivalent modulo-input, and 3315568.3329966, doi:10.1145/3315568.3329966. readable code. We assessed eight decompilers with a set of [6] Dupuy, E., 2019. Java Decompiler. https://http:// java-decompiler.github.io/ 2041 classes from 14 open-source projects compiled with . [Online; accessed 19-July- two different compilers. The results of our analysis show 2019]. [7] Ďurfina, L., Křoustek, J., Zemek, P., 2013. PsybOt Malware: A Step- that bytecode decompilation is a non-trivial task that still re- By-Step Decompilation Case Study, in: 20th Working Conference on quires human work. Indeed, even the highest ranking de- Reverse Engineering (WCRE), pp. 449–456. doi:10.1109/WCRE. compiler in this study produces syntactically correct output 2013.6671321. 84% [8] Emamdoost, N., Sharma, V., Byun, T., McCamant, S., 2019. Binary for of classes of our dataset and semantically equiva- 10. lent modulo inputs output for 78%. We extract 157 classes mutation analysis of tests using reassembleable disassembly. doi: 14722/bar.2019.23058. for which no decompiler produces semantically equivalent [9] Falleri, J.R., Morandat, F., Blanc, X., Martinez, M., Monperrus, M., sources. These classes illustrate how generics and, in partic- 2014. Fine-grained and Accurate Source Code Differencing, in: 29th ular, generic with wildcards and type bounds are challenging International Conference on Automated Software Engineering (ASE), ACM, New York, NY, USA. pp. 313–324. URL: http://doi. for all decompilers. Yet the Java language with its diver- acm.org/10.1145/2642937.2642982 10.1145/2642937. sity of compilers and versions makes room for many corner , doi: 2642982. cases that require extensive testing and development effort [10] Flores-Montoya, A., Schulte, E.M., 2019. Datalog disassembly. from decompilers authors. Meanwhile, the diversity of im- CoRR abs/1906.03969. URL: http://arxiv.org/abs/1906. plementation of these decompilers allows to merge their dif- 03969, arXiv:1906.03969.
Harrand et al.: Preprint submitted to Elsevier Page 18 of 19 Java Decompiler Diversity and its Application to Meta-decompilation
[11] Foster, B., Somayaji, A., 2010. Object-level recombination of com- [31] Nolan, G., 2004. Decompiler Design. Apress, Berkeley, CA. pp. 121– modity applications, in: Proceedings of the 12th annual conference 157. URL: https://doi.org/10.1007/978-1-4302-0739-9_ on Genetic and evolutionary computation, pp. 957–964. 5, doi:10.1007/978-1-4302-0739-9_5. [12] Fu, C., Chen, H., Liu, H., Chen, X., Tian, Y., Koushanfar, F., Zhao, J., [32] Pawlak, R., Monperrus, M., Petitprez, N., Noguera, C., Seinturier, 2019. Coda: An end-to-end neural program decompiler, in: Advances L., 2015. Spoon: A Library for Implementing Analyses and Trans- in Neural Information Processing Systems, pp. 3703–3714. formations of Java Source Code. Software: Practice and Experience [13] Grech, N., Brent, L., Scholz, B., Smaragdakis, Y., 2019. Gigahorse: 46, 1155–1179. URL: https://hal.archives-ouvertes.fr/ thorough, declarative decompilation of smart contracts, in: Interna- hal-01078532/document, doi:10.1002/spe.2346. tional Conference on Software Engineering, IEEE. pp. 1176–1186. [33] Persaud, B., Obada-Obieh, B., Mansourzadeh, N., Moni, A., So- [14] Gusarovs, K., 2018. An Analysis on Java Programming Language mayaji, A., 2016. Frankenssl: Recombining cryptographic libraries Decompiler Capabilities. Applied Computer Systems 23, 109–117. for software diversity, in: Proceedings of the 11th Annual Symposium [15] Hamilton, J., Danicic, S., 2009. An Evaluation of Current Java Byte- On Information Assurance., pp. 19–25. code Decompilers, in: 9th IEEE International Working Conference [34] Ragkhitwetsagul, C., Krinke, J., 2017. Using Compilation/Decompi- on Source Code Analysis and Manipulation (SCAM), pp. 129–136. lation to Enhance Clone Detection, in: 11th International Workshop doi:10.1109/SCAM.2009.24. on Software Clones (IWSC), pp. 1–7. doi:10.1109/IWSC.2017. [16] Hoenicke, J., 2019. JODE. http://jode.sourceforge.net/. 7880502. [Online; accessed 19-July-2019]. [35] Robles, G., Gonzalez-Barahona, J.M., Herraiz, I., 2005. An Empirical [17] Jaffe, A., Lacomis, J., Schwartz, E.J., Goues, C.L., Vasilescu, Approach to Software Archaeology, in: 21st International Conference B., 2018. Meaningful Variable Names for Decompiled Code: on Software Maintenance (ICSM), pp. 47–50. A Machine Translation Approach, in: 26th Conference on Pro- [36] Schulte, E., Ruchti, J., Noonan, M., Ciarletta, D., Loginov, A., 2018. gram Comprehension (ICPC), ACM, New York, NY, USA. pp. 20– Evolving exact decompilation, in: Shoshitaishvili, Y., Wang, R.F. 30. URL: http://doi.acm.org/10.1145/3196321.3196330, (Eds.), Workshop on Binary Analysis Research, San Diego, CA, USA. doi:10.1145/3196321.3196330. URL: http://www.cs.unm.edu/~eschulte/data/bed.pdf. [18] Jang, H., Jin, B., Hyun, S., Kim, H., 2019. Kerberoid: A practi- [37] Selberg, E., Etzioni, O., 1997. The metacrawler architecture for re- cal android app decompilation system with multiple decompilers, in: source aggregation on the web. IEEE Expert 12, 11–14. doi:10. Proceedings of the 2019 ACM SIGSAC Conference on Computer and 1109/64.577468. Communications Security, pp. 2557–2559. [38] skylot, 2019. JADX. https://github.com/skylot/jadx. [On- [19] Jerome Miecznikowski, Nomair A. Naeem, L.J.H., 2019. Dava. line; accessed 19-July-2019]. http://www.sable.mcgill.ca/dava/. [Online; accessed 19- [39] Storyyeller, 2019. Krakatau. https://github.com/ July-2019]. Storyyeller/Krakatau. [Online; accessed 19-July-2019]. [20] JetBrains, 2019. Fernflower. https://github.com/ [40] Strobel, M., 2019. Procyon. https://bitbucket.org/ JetBrains/intellij-community/tree/master/plugins/ mstrobel/procyon. [Online; accessed 19-July-2019]. java-decompiler/engine. [Online; accessed 19-July-2019]. [41] Troshina, K., Derevenets, Y., Chernov, A., 2010. Reconstruction of [21] Katz, D.S., Ruchti, J., Schulte, E., 2018. Using Recurrent Neural Composite Types for Decompilation, in: 10th IEEE Working Confer- Networks for Decompilation, in: 25th International Conference on ence on Source Code Analysis and Manipulation (SCAM), pp. 179– Software Analysis, Evolution and Reengineering (SANER), pp. 346– 188. doi:10.1109/SCAM.2010.24. 356. doi:10.1109/SANER.2018.8330222. [42] Vallée-Rai, R., Co, P., Gagnon, E., Hendren, L., Lam, P., Sundaresan, [22] Khadra, M.A.B., Stoffel, D., Kunz, W., 2016. Speculative disassem- V., 1999. Soot - a java bytecode optimization framework, in: Pro- bly of binary code, in: International Conference on Compliers, Ar- ceedings of the 1999 Conference of the Centre for Advanced Studies chitectures, and Sythesis of Embedded Systems (CASES), pp. 1–10. on Collaborative Research, IBM Press. p. 13. doi:10.1145/2968455.2968505. [43] Van Emmerik, M.J., 2007. Static Single Assignment for Decompila- [23] Kostelanský, J., Dedera, L., 2017. An Evaluation of Output from Cur- tion. University of Queensland. rent Java Bytecode Decompilers: Is it Android Which is Responsible [44] Vinciguerra, L., Wills, L., Kejriwal, N., Martino, P., Vinciguerra, for Such Quality Boost?, in: Communication and Information Tech- R., 2003. An Experimentation Framework for Evaluating Disassem- nologies (KIT), pp. 1–6. doi:10.23919/KIT.2017.8109451. bly and Decompilation Tools for C++ and Java, in: 10th Working [24] Lacomis, J., Yin, P., Schwartz, E.J., Allamanis, M., Goues, C.L., Neu- Conference on Reverse Engineering (WCRE), IEEE Computer Soci- big, G., Vasilescu, B., 2019. Dire: A neural approach to decompiled ety, Washington, DC, USA. pp. 14–. URL: http://dl.acm.org/ identifier naming. arXiv:1909.09029. citation.cfm?id=950792.951361. [25] Le, V., Afshari, M., Su, Z., 2014. Compiler Validation via Equiva- [45] Wang, S., Wang, P., Wu, D., 2015. Reassembleable disassembling, in: lence Modulo Inputs, in: 35th Conference on Programming Language 24th USENIX Security Symposium (USENIX Security 15), USENIX Design and Implementation (PLDI), ACM, New York, NY, USA. Association, Washington, D.C.. pp. 627–642. URL: https: pp. 216–226. URL: http://doi.acm.org/10.1145/2594291. //www.usenix.org/conference/usenixsecurity15/ 2594334, doi:10.1145/2594291.2594334. technical-sessions/presentation/wang-shuai. [26] Li, Z., Wu, Q., Qian, K., 2019. Adabot: Fault-Tolerant Java Decompi- [46] Yakdan, K., Dechand, S., Gerhards-Padilla, E., Smith, M., 2016. ler. Technical Report 1908.06748. arXiv. Helping Johnny to Analyze Malware: A Usability-Optimized De- [27] Lindholm, T., Yellin, F., Bracha, G., Buckley, A., 2014. The Java compiler and Malware Analysis User Study, in: IEEE Symposium on Virtual Machine Specification. Pearson Education. Security and Privacy (SP), pp. 158–177. doi:10.1109/SP.2016.18. [28] Miecznikowski, J., Hendren, L., 2002. Decompiling Java Bytecode: [47] Yang, Y., Zhou, Y., Sun, H., Su, Z., Zuo, Z., Xu, L., Xu, B., 2019. Problems, Traps and Pitfalls, in: Horspool, R.N. (Ed.), Compiler Con- Hunting for Bugs in Code Coverage Tools via Randomized Differen- struction, Springer Berlin Heidelberg, Berlin, Heidelberg. pp. 111– tial Testing, in: 41st International Conference on Software Engineer- 127. ing (ICSE), ACM. [29] Miller, K., Kwon, Y., Sun, Y., Zhang, Z., Zhang, X., Lin, Z., 2019. Probabilistic disassembly, in: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), IEEE. pp. 1187–1198. [30] Naeem, N.A., Batchelder, M., Hendren, L., 2007. Metrics for Mea- suring the Effectiveness of Decompilers and Obfuscators, in: 15th IEEE International Conference on Program Comprehension (ICPC), pp. 253–258. doi:10.1109/ICPC.2007.27.
Harrand et al.: Preprint submitted to Elsevier Page 19 of 19