Input Algebras

Rahul Gopinath · Hamed Nemati · Andreas Zeller CISPA Helmholtz Center for Information Security Saarbrucken,¨ Germany Email: {rahul.gopinath, hamed.nemati, andreas.zeller}@cispa.saarland

Abstract—Grammar-based test generators are highly efficient hjsoni ::= helti in producing syntactically valid test inputs and give their users precise control over which test inputs should be generated. helti ::= hobjecti|h arrayi|h stringi|h numberi Adapting a grammar or a test generator towards a particular | ‘true’ | ‘false’ | ‘null’ testing goal can be tedious, though. We introduce the concept of a hobjecti ::=‘ {’ hitemsi ‘}’ | ‘{}’ grammar transformer, specializing a grammar towards inclusion or exclusion of specific patterns: “The phone number must not hitemsi ::= hitemi|h itemi ‘,’ hitemsi start with 011 or +1”. To the best of our knowledge, ours is hitemi ::= hstringi ‘:’ helti the first approach to allow for arbitrary Boolean combinations harrayi ::=‘ [’ heltsi ‘]’ | ‘[]’ of patterns, giving testers unprecedented flexibility in creating targeted software tests. The resulting specialized grammars heltsi ::= helti|h elti ‘,’ heltsi can be used with any grammar-based fuzzer for targeted test hstringi ::=‘ "’ hcharsi ‘"’ generation, but also as validators to check whether the given specialization is met or not, opening up additional usage scenarios. hcharsi ::= hcharih charsi|  In our evaluation on real-world bugs, we show that specialized hchari ::=‘ [A-Za-z0-9]’ grammars are accurate both in producing and validating targeted hnumberi ::= hdigitsi inputs. Index Terms—testing, debugging, faults hdigitsi ::= hdigitih digitsi|h digiti hdigiti ::=‘ [0-9]’ I.INTRODUCTION Software test generators at the system level (commonly Fig. 1: JSON grammar (simplified) known as fuzzers) face the challenge of producing valid inputs that pass through syntactic checks to reach actual functionality. The problem is commonly addressed by having a seed, a properties—say, a SQL injection only in a particular context? set of known valid inputs which are then mutated to cover How about negation—say, any kind of input except for specific more behavior. For complex input languages, though, mutations elements? And how about their combination? In principle, still mostly exercise syntax error handling. We can obtain such features can be expressed in a context-free language, and better performance by using a language specification such as hence in a grammar—but these grammars would be nowhere a grammar to produce inputs. As such inputs are syntactically as compact and maintainable as the example in Fig.1. valid by construction, they reliably reach actual functionality In the past, the need for such control (also known as beyond parsing. The JSON grammar in Fig.1, for instance, taming) has been addressed by tweaking grammar-based test produces only syntactically valid JSON inputs and thus quickly generators—that is, adding special options (often ad hoc and covers JSON parser functionality. domain-specific) that address these concerns within the test Writing accurate input grammars can be a significant generator [4], [5], [6]. With the test generator being Turing- effort [1], which is why recent research [2], [3] has started complete, there is no limitation to what such options can do. extracting such grammars automatically from existing programs. However, they also mean that one is tied to the particular tool One less discussed advantage of language specifications, and its specific features. however, is how much control they grant their users over which In this paper, we address the problem of controlling grammar- inputs should be generated—actually, much more control than based test generators from the ground up. We introduce the for seed-based fuzzers. For our JSON grammar, for instance, a concept of a grammar transformer—a tool that takes a (simple) user could go and assign probabilities to individual productions. context-free grammar and specializes it towards a specific goal. If the input should contain, say, several null values, users can As a result, we obtain a specialized context-free grammar, assign a high probability to null productions. Likewise, users which can then be used to produce specialized inputs with any can customize the grammar. Adding an alternative expansion context-free grammar-based test generator [7], [8], [9], [10], "’); DROP TABLE STUDENTS; --" to the hstringi rule [11], [12], [13], [14], [15], [16], [17], [18], [19], [20]. Also, will quickly populate the generated JSON input with SQL the grammar can be used with any context-free parser [21], injections, for instance. [22], [23], [24] for checking existing inputs whether they meet But there’s a catch. Adding individual specific alternatives the specialization properties. or adjusting probabilities is easy. But how about contextual To specify specializations, we introduce a language that def jsoncheck(json): is the empty string. Hence, its outputs will always reach the if no_key_is_empty_string(json): process() function in Fig.2. fail(’one key must be empty’) To the best of our knowledge, this is the first work where if any_key_has_null_value(json): fail(’key value must not be null’) users of a test generator can specifically express the absence process(json) and conjunction of patterns in given contexts. Going beyond simple data languages like JSON, evocative patterns thus allow us to produce very targeted inputs for testing compilers and Fig. 2: jsoncheck() fails if any key value is null or if no interpreters—say, a series of while loops that contain no key is empty. assignments; or SQL queries containing inner joins, left joins, and right joins all in one. This ability to express conjunctions and negations, all within allows us to specify which features should be part of the the grammar, may be come as a surprise: As context-free specialized grammar and which ones should not. At the base grammars are not closed under conjunction or complement, of our language, we have evocative patterns (or patterns there is no way to transform a grammar into its negation. How- for short)—that form constraints over the values of specific ever, we show that our specialized grammars are closed under nonterminals. Such patterns take the form conjunction, disjunction, and complement. Hence, Boolean hnonterminali is value combinations of patterns result in algebras of specialized grammars. and are interpreted that there should be at least one instance Besides controlling test generation, our evocative expressions hnonterminali value of with value . For our JSON grammar, (which we call ewoks) can be used in various ways. For for instance, we can create specialized grammars example: • where at least one hitemi has a value of -1—that is, 1) When used in debugging, the evocative patterns can hitemi is hstringi: -1. The resulting grammar produces quickly generate test inputs that contain a given failure- JSON strings such as { "foo": -1, "bar": 33 }. inducing input, validating fixes. This works particularly • where at least one key should be named "zip", followed well for abstract failure-inducing inputs —strings in which by a hnumberi—that is, hitemi is "zip": hnumberi. The specific parts are generalized to nonterminals, abstracting resulting grammar would produce JSON strings such as over concrete values [25]—, which translate immediately { "foo": "bar", "zip": 45 }. into evocative patterns. • where at least one array should contain a null value— 2) When used in conjunction with abstract failure-inducing that is, heltsi is null. The resulting grammar would inputs which represent specific program behaviors, our produce JSON strings such as { "foo": "bar", evocative expressions can be used to generate inputs that "zip": true, "qux": [2, 1, null] }. represent complex program behavior, that can be checked For more complex specializations, these patterns can be for conformance. For example, one may mine abstract combined into Boolean formulas. These evocative expressions inputs that represent the coverage of specific portions of allows us to precisely specialize the produced inputs to match the program. The evocative patterns corresponding to such specific conditions. As an example, consider the function abstract inputs may be used to produce complex evocative jsoncheck() in Fig.2. The process() function is only expressions that correspond to inputs that (1) cover various reached if no key has a null value and at least one key is parts of the program, (2) does not cover specific parts the empty string. To avoid the first branch to failure (“one key of the program, or (3) any combination where some must be empty”), we can use the evocative pattern parts are guaranteed not to be covered, while other parts hitemi is "": helti (1) are guaranteed to be covered. That is, with evocative expressions, one can build complex oracles out of simpler This ensures that at least one item will have an empty key. oracles. Avoiding the second branch to failure (“key value must not be 3) When used with a parser, they can check whether a given null”), we have to specify the absence of a pattern. We do so input meets a specified evocative expression—for instance, by negation, expressing that no element matching the pattern to identify inputs that trigger a failure or vulnerability. should be generated: 4) Evocative expressions can serve as query languages for ¬hitemi is hstringi: null
(2) searching and capturing specific patterns in structured data. Any of the above patterns could be used in a query. To finally reach the process() function, both conditions must be 5) Going further, evocative expressions might serve as met. We obtain this by creating a conjunction of Equation (1) alternatives for regular expressions—especially as they and Equation (2): allow to express negation. Given their verbosity, structure, and named subexpressions, evocative expressions may be hitemi is "": helti
∧ ¬hitemi is hstringi: null
(3) easier to read, write, and maintain than regular expressions. The resulting grammar will produce null values, but never as 6) Evocative expressions can serve as configuration language the value of an hitemi; and ensure that at least one key name to adapt generic configurations towards specific goals. Features in software product lines, customizations of user Definition. A set of rules that describe the expansion of a interfaces, and permissions in access control lists could nonterminal or how the nonterminal is matched. be combined in arbitrary ways. Context-Free Grammar. The context-free grammar is com- This work opens the door towards giving developers simple, posed of a set of nonterminals and corresponding defini- yet powerful controls over what fuzzers produce, without re- tions that define the structure of the nonterminal. quiring them to modify fuzzer sources or rewrite grammars and Derivation. A terminal derives a string if the string contains other specs from scratch. Beyond fuzzing, usages of specialized only the symbols in the terminal. A nonterminal derives grammars also include 1) generating targeted failure-inducing a string if the corresponding definition derives the string. inputs for validating fixes; 2) parsing inputs, checking whether A definition derives the string if one of the rules in they contains a particular pattern (or combination thereof); the definition derives the string. A rule derives a string 3) querying patterns in data (possibly even replacing regular if the sequence of terms that make up the rule can expressions); 4) configuring and customizing systems towards derive the string, deriving one substring after another specific goals. contiguously (also called parsing). Generation is defined After introducing basic definitions (SectionII), the remainder complementarily [25]. of this paper is organized along its contributions: Derivation Tree. An ordered tree that describes how an input string is derived by the given start symbol. An abstract Abstract patterns. Section III introduces abstract patterns, derivation tree has some nodes marked as abstract, and which form the base for grammar specializations. abstract nodes may not have child nodes. Specializing grammars. In SectionIV, we introduce the Compatible Node. A node is compatible to another if both algorithm for specializing grammars to contain at least have the same nonterminal. A tree is compatible to another one instance of the pattern on any of the generated inputs. if both have compatible root nodes. Specialization algebras. SectionV shows how to combine Reachable Nonterminal. A nonterminal a is reachable from specializations into algebras using Boolean operators. another nonterminal b if a is reachable from any of the We provide (partially mechanized) proofs that grammar rules in the definition of b. A nonterminal is reachable from specializations are closed under disjunction, conjunction a rule if 1) that nonterminal is present in the rule or 2) that and negation. nonterminal is reachable from any of the nonterminals in Implementation and usage. SectionVI introduces Evogram, the rule. For example, harrayi is reachable from hjsoni a grammar transformer prototype that implements these helti, hobjecti, hitemsi, hitemi, heltsi and also itself. techniques, as well as usage scenarios for testing, debug- Subtree. For any node, a subtree is a tree rooted in any nonter- ging, and adaptation. minal reachable from that node. The characteristic node Evaluation. In Section VII, we show that the specialized of that subtree is its top most node and its nonterminal is grammars accurately produce and recognize inputs that the characteristic nonterminal. satisfy the given patterns. Related work. Comparing to related work (Section VIII), ours We also introduce the following new definitions, generalizing is the first approach that allows test generators to express beyond failure-inducing inputs to evocative inputs—that is, absence and conjunction of patterns in a given context. inputs that trigger a specific program behavior. In contrast to the closest related work [25], Evogram can Evocative Input. An input string is evocative or failure- produce and recognize inputs that contain failure-inducing inducing, if on execution of the program with the string as patterns in arbitrary contexts. input, the program fails in a particular fashion or displays As detailed in SectionIX, Evogram and all experiments are an expected behavior. available for reuse and replication. We close with conclusion Evocative Fragment. A fragment of an evocative input string and future work. is evocative or failure-inducing if some property of the fragment is the cause of the behavior or failure observed. II.DEFINITIONS That is, we can transplant the subtree corresponding to The following definitions are based on [25]. that fragment to a compatible node in another derivation Alphabet. The alphabet of the input language of a program tree, and the corresponding string reproduces the same is the set of all symbols accepted by the program. behavior. Input. A contiguous sequence of symbols from the alphabet that is passed to a given program. Terminal. A sequence of symbols from the alphabet. These III.ABSTRACT PATTERNS form the leaves of the derivation tree. Nonterminal. A symbol outside the alphabet whose expansion Our end goal is to generate specialized grammars that obey is defined in the grammar. given constraints. Our ongoing example will be the code Rule. A finite sequence of terms (two types of terms: terminals constraints from Fig.2, requiring 1) that at least one JSON and nonterminals) that describe an expansion of a given object has an empty string as key, and 2) no JSON object has nonterminal. null as a key value. In our pattern language, we can produce hjsoni hitemi hitemE0 i helti hstringi : helti hstringE1 i : helti hobjecti ” hcharsi ” ” hcharsE2 i ” {h itemi }   hstringi : helti (a) Abstract tree (b) Specialized nodes

” hcharsi ” hstringi Fig. 4: Abstract trees for hitemEi is "":helti

 ” hcharsi ”  hstringi as an abstraction of the corresponding input fragment. Fig. 3: Derivation tree for {"":""} When deriving an abstract pattern, a nonterminal also matches the corresponding abstraction. Such abstract patterns concisely and precisely specify what the following evocative expression from which a grammar that parts of the input are important, and the abstract pattern is a obeys the given constraints can be produced. recipe for minimal and complete evocative inputs. However, we are only interested in a smaller part of the corresponding hjsonE∧N i (4) abstract derivation tree. We are interested in the smallest subtree where hitemEi is "":helti (5) that when included in a larger input, induces the failure reliably. :null hitemN i is hstringi (6) B. Abstract Patterns to Evocative Patterns This expression is equivalent to the compound expression in So, what part of the derivation tree actually caused the Equation (3). Here, hitemEi (5) represents the constraint of failure? On inspection, one can see that a much smaller part Empty key, and hitemN i (6) represents the constraint of Null of the derivation tree under the node for hitemi is sufficient value. The expression hjsonE∧N i represents the specialized to induce the failure. That is, one can replace an hitemi grammar with the combined constraint of at least one empty node in any derivation tree with this node, and the resulting key, and no null values in any generated input.1 In the rest string will induce the failure. That is, the subtree given in of the paper, this forms the running example, and we see how Fig. 4a accurately captures the constraint. Such subtrees can be each component of this expression is derived. represented by their characteristic nonterminal, and the string How do we relate these expressions to grammars? The representation of the rest of the tree, using nonterminal symbols constraints are predicates on the derivation tree. That is where abstract nodes are present. We call these evocative hitemEi is "":helti represents the constraint that a derivation patterns. For example, hitemEi is "":helti is the evocative tree produced should contain an hitemi node with two children. pattern that represents the subtree given in Fig. 4a. The first should be an empty string, and the second could be Given an abstract pattern, one can also automatically obtain any helti. the characteristic node by simply looking for the smallest subtree that contains all terminal symbols in the abstract pattern. A. Finding Abstract Patterns in Derivation Trees We use such evocative patterns in our evaluation. However, the To derive the constraint E that represents a particular failure characteristic node obtained could likely be trimmed further to condition, we start with an evocative input that induces the produce a smaller subtree that can still induce failures reliably, expected failure: {"":""}. This will have a derivation tree in a larger variety of contexts. as given in Fig.3. Are all parts of the input equally needed to satisfy the constraint? Indeed, we can easily see that the IV. TRANSFORMING GRAMMARS key value helti of value ’""’ can be replaced with any other We now show how one can transform the underlying helti, and still induce the failure. That is, the following is an grammar such that any input generated from it would contain abstract representation of inputs that induces the same failure: at least one instance of the evocative pattern. The underlying {"":helti}. We note that this is exactly the abstract pattern non-specialized grammar is called the base grammar, and generated by DDSET [25]. the definitions, rules, and nonterminals in that grammar are That is, for any given derivation tree, the corresponding called base definitions, base rules and base nonterminals abstract pattern is the abstract string representation where respectively. Note that we require base grammars to be non- the string representation of nodes marked as abstract are ambiguous. That is, any given string should have only a single the corresponding nonterminals (called an abstraction). For derivation. Transforming the grammar means to specialize it, example, {hstringi:null} is an abstract pattern, and contains and this is accomplished by specializing each nonterminal, the corresponding definition, and each applicable rule in the 1While both evocative patterns have hitemi as the characteristic nontermi- definition. We describe specializations of nonterminals and nal in this example, there is no such requirement. That is, hjsonE∨I i where hnumberI i is 011hdigitsi is allowed. their corresponding definitions next. Specialized nonterminal. A specialized nonterminal is a base B. Constructing a reaching grammar nonterminal name followed by a specialization suffix. 1) Finding insertable positions: To be able to instantiate For example, given a nonterminal hitem i, the hitemi is N instances of the evocative fragment, one needs to be able to the corresponding base nonterminal (also called the base reach the characteristic nonterminal of the evocative pattern. representation of the specialized nonterminal) and the For example, to reach hitemi in the JSON grammar, one has to subscript N is the specialization representing the evocative start from one of the following nonterminal symbols: hjsoni, expression. Note that a terminal cannot be a specialized. helti, hobjecti, hitemsi, hitemi, harrayi, heltsi. Hence, the base representation of a terminal is always itself. A term is same kind as another term if both have For each rule in the grammar, we identify the nonterminal the same base representation. terms that can reach the nonterminal of the characteristic node. The subscript N is constructed as follows: An evocative For JSON, the following (*) are the insertable positions. pattern is defined as hjsoni ::= h*elti helti ::= h*objecti|h *arrayi hnonterminal subscripti is value hobjecti ::=‘ {’ h*itemsi ‘}’ ,’ where subscript refers to the particular constraint repre- hitemsi ::= h*itemi|h *itemi ‘ h*itemsi sented by value. We use a related notation for specializa- hitemi ::= hstringi ‘:’ h*elti tion. We use hnonterminalsubscripti to denote a specialized harrayi ::=‘ [’ h*eltsi ‘]’ nonterminal where subscript is a Boolean expression of heltsi ::= h*elti|h *elti ‘,’ h*eltsi the constraints acting on the given nonterminal. Note In the JSON grammar, hstringi cannot reach hitemi. Hence, it that in terms of the context-free grammar, this is just is unmarked. None of the other nonterminals have any rules another identifier. A context-free grammar containing such that can reach hitemi. specialized nonterminals can be used exactly like any other For the reaching grammar, we take each rule of a given context-free grammar. For us, however, it is a unique and definition, and identify the insertable positions in the rule. Next, expressive identifier that states the precise purpose. for each insertable position, we produce a copy of the rule with Specialized definition. A specialized definition can have rules the nonterminal at the identified position specialized with the with specialized nonterminals or some of the expansion evocative fragment name. The reaching definition correspond- rules may be removed from the corresponding base gram- ing to the new specialized nonterminal is the collection of such mar definition. The following is a specialized definition new rules. If the rule has no insertable positions, it cannot of hobject i, removing the empty object expansion, and E derive evocative fragments. Hence, such rules are discarded. includes specialized nonterminalshitems i and hitems i. E E1 If a nonterminal has no reaching rules, its definition is empty, hobject i ::=‘ {’ hitems i ‘}’ | ‘{’ hitems i ‘}’ E E E1 and it is discarded from the reaching grammar. The reaching Some of the specialized rules (such as above) grammar is a grammar composed of reaching nonterminals may have the same base representation rule – and their corresponding reaching definitions. The definition for hobjecti ::= ’{’ hitemsi ’}’ in the above example. Such nonterminals in reachable grammar for E is as follows a set of rules in a definition is called a ruleset for that hjson i ::= helt i base rule, and such rules have same kind. E E heltE i ::= hobjectE i|h arrayE i A. Translating evocative patterns to pattern grammars hobjectE i ::=‘ {’ hitemsE i ‘}’ hitemsE i ::= hitemE i ‘,’ hitemsi|h itemi ‘,’ hitemsE i We translate the abstract derivation tree rooted at charac- |h itemE i teristic node to a grammar capable of producing instances of hitemE i ::= hstringi ‘:’ heltE i the evocative fragment, using our JSON grammar (Fig.1) as harray i ::=‘ [’ helts i ‘]’ running example. To do this, we specialize the non-abstract E E nodes from the abstract tree with the evocative pattern name heltsE i ::= heltE i ‘,’ heltsi|h elti ‘,’ heltsE i|h eltE i and unique suffixes such that no two nonterminal symbols will reuse the same name and suffix. For example, given the abstract C. Connecting pattern grammar and reaching grammar tree Fig. 4a, nodes are specialized in Fig. 4b. Next, with the For the final grammar, we merge the pattern grammar and the characteristic nonterminal (hitem i here) as the start symbol, E0 reaching grammar, and use the start symbol from the reaching this tree is collapsed into a grammar, children of each node grammar. The connection is made at the reaching characteristic forming a single rule definition for their nonterminal. nonterminal (here, hitemEi). We merge the definition from hitem i ::= hstring i ‘:’ helti E0 E1 the characteristic nonterminal of the pattern grammar (here, hstringE1 i ::=‘ "’ hcharsE2 i ‘"’ hitemE0 i) to the reaching nonterminal of characteristic node. hcharsE i ::=  2 Note that hstringE1 i was previously defined and represents the Next, we will see how the characteristic symbol of pattern empty string. grammar (hitemE0 i) is connected to the rest of the grammar. hitemE i ::= hstringi ‘:’ heltE i|h stringE1 i ‘:’ helti hitemi hitemN0 i B. Conjunction of two terms The term from conjunction of two terms will only derive (any hstringi : helti hstringi : helt i N1 and all) substrings derived by both the operands, and is defined null null only for terms of the same kind. The conjunction between (a) Abstract tree (b) Specialized nodes two equal nonterminals is the same nonterminal, between > and a specialized nonterminal is the specialized nonterminal, Fig. 5: Abstract trees for hitem i is hstringi:null N between a nonterminal and its negation is ⊥, and between ⊥ and anything else is ⊥. For example, conjunction of harrayEi and harray i is harray i representing elements in harrayi V. ALGEBRAOF GRAMMAR SPECIALIZATIONS N E∧N derived by both harrayEi and harrayN i. Next, we discuss how arbitrary Boolean expressions can be composed from other grammar specializations. We call C. Disjunction of two terms these evocative expressions. Again, consider the expression The term from disjunction of two terms will only derive introduced in Equation (4) that represents a specialized gram- (any and all) substrings derived by either operands, and is only mar producing inputs that contain at least one empty key, and defined for terms of the same kind. The disjunction between does not contain a key value null. This will be our ongoing two equal nonterminals is the same nonterminal, between > example for specializations. and a specialized nonterminal is >, between a nonterminal We first look at negation of grammar specializations. In the and its negation is >, and between any nonterminal and ⊥ grammar, we indicate negation by an over line. To illustrate is the first nonterminal. For example, disjunction of harrayEi the algebra, we use the second pattern—at least one instance and harrayN i is harrayE∨N i representing elements in harrayi of a null key value in the input. The abstract pattern from derived by either harrayEi or harrayN i. which the expression is derived is {hstringi:null}, and the evocative pattern is hitemi is hstringi:null. The abstract tree D. Negation of a single rule is given in Fig. 5a, and its pattern tree in Fig. 5b. From this, Negation of a rule produces as many rules as there are the specialized grammar contains all the above nonterminal specialized nonterminals each with one specialized nonterminal definitions, as well as negated, and the rules from negation of a specialized rule

hjsonN i ::= heltN i will not be able to derive a string derived by the operand,

heltN i ::= hobjectN i|h arrayN i and vice versa. Say you have a rule heltEi ‘,’ heltsEi. To negate the rule, we take each specialized nonterminal, and harrayN i ::=‘ [’ heltsN i ‘]’ negate it one at a time. In this example, the result is two rules: hobjectN i ::=‘ {’ hitemsN i ‘}’ heltEi ‘,’ heltsEi and heltEi ‘,’ helts Ei. heltsN i ::= heltN i ‘,’ heltsi|h elti ‘,’ heltsN i|h eltN i 2 Proof: C A string derived by the operand could not have been hitemsN i ::= hitemN i ‘,’ hitemsi|h itemi ‘,’ hitemsN i |h item Ni derived by any of the new rules because each rule contains at least one nonterminal that will reject the corresponding hitemN i ::= hstringi ‘:’ heltN i|h stringi ‘:’ heltN i 1 derivation. If any new rule derived a string, at least one helt i ::=‘ null’ B N1 nonterminal in operand will reject it3. Now we come to the actual definition of Boolean expressions. We start with the axioms for terms, where we have two basic E. Conjunction of two rules requirements: Conjunction is defined only between rules of the same kind, 1) None of the operations can change the term type or kind. and the result will derive (any and all) strings that could be 2) In the case of two or more operands, operations are only derived by both the operands. To produce a conjunction of two defined with respect to the same kind of terms. rules, we line up both, and produce a conjunction of terms from each at corresponding positions. That is, given two rules We use hnonterminal >i (short: hnonterminali or > where unambiguous) to represent base nonterminal, and hnonterminal heltEi ‘,’ heltsEi and heltN i ‘,’ heltsN i, the conjunction is helt i ‘,’ helts i. ⊥i or ⊥ for nonterminals with empty definition. Rules with ⊥ E∧N E∧N terms are removed recursively at the end of evaluation of the Proof: C Say the new rule derives a string. Pick an operand. complete expression. Each nonterminal in the deriving rule could be replaced by one nonterminal in operand rule without affecting the derivation. A. Negation of a single term B Say a string is derived by both operands. That string could Negation of a term means that if a term derived a given be split into contiguous substrings such that each substring substring, the negated term will not derive the same substring, is derived by either a terminal or corresponding nonterminals and vice versa. Negation of a terminal is empty. A nonterminal from both operands. If two nonterminals derive the same string, with specialization is negated with respect to the specialization. then their conjunction also derives it.

That is, harray i represents elements in harrayi not derived 2 E We use C and B to indicate the direction of the proof. by harrayEi. Negation of a base nonterminal is ⊥. Negation 3The appendix for this paper contains a partial mechanization for proofs of an empty nonterminal is >. That is, hitem⊥i = hitemi. in HOL4 [26]. F. Disjunction of two rules Proof: C Any string derived by the new ruleset will be derived Disjunction is defined only between rules of the same kind, by a rule in at least one of the operands. B Any string derived and the result will derive (any and all) strings that could be by a rule in a parent operand will be derived by new ruleset derived by one of the operands. To produce a disjunction because the same rule is there in the result. of two rules, merge them or use each as alternatives in any J. Negation of a definition resultant definition. One may merge two rules into one if the two rules differ only by a single specialized nonterminal The result of a negation derives (any and all) strings that which is replaced by a disjunction of operand specializations. will not be derived by the operand. For negation of a definition, any base rule that is not represented by a ruleset in the non- Given two rules heltEi ‘,’ heltsi and heltN i ‘,’ heltsi, the negated definition is added directly to the negated definition. disjunction is heltE∨N i ‘,’ heltsi. On the other hand, given These correspond to the rules with empty terms that we heltEi ‘,’ heltsEi and heltN i ‘,’ heltsN i, the disjunction is exactly the same as original two rules. discarded when the operand was produced. Then, negations from each rulesets are combined together and added to the Proof: C Pick any operand. The merged rule differs from the operand rule in exactly one place, which is a superset of negated definition. The negation of a definition is represented operand specialization, any string derived by the operand could by the negation of its corresponding nonterminal. Proof: Say a string was derived by the operand. A rule that also be derived by the new rule. B Any string that is derived C by the new rule can be split into contiguous substrings such was not present in the original rulesets could not have derived that each term in the new rule derives one substring. The new the string as part of the original non-negated definition. Hence, rule differs from operand rules in exactly one nonterminal, non-represented rules could not derive a string that could be which is a disjunction of operand nonterminals. derived by the operand. Next, all other rulesets are negations of rulesets in the operand none of which can derive a string G. Negation of a ruleset derived by the operand. B If a string was derived by the new The result of negation of a ruleset derives (any and all) definition, this string was either derived by one of the directly strings that will not be derived by the negated ruleset. Given added rules, in which case, there is no corresponding rule in a ruleset with multiple rules, one has to only remember that the operand, or by one of the negated rules, in which case, the they are alternatives. E.g. S = R1|R2|R3. Hence, negation operand could not have derived it anyway. is based on Boolean algebra. That is, R |R |R is same as 1 2 3 K. Conjunction of two definitions R1 ∧R2 ∧R3. Given that each rule negation results in multiple alternates, we apply distributive law. That is the new ruleset is Result of conjunction of two definitions derives (any and all) as follows: {r1 ∧ r2 ∧ r3 : r1 ∈ R1, r2 ∈ R2, r3 ∈ R3} strings derived by both the operands. To produce a conjunction Proof: C Any string derived by any of the new rules will be of two definitions, the rulesets of the same kind from each derived by each of the negations because it is a conjunction operand are paired up, and conjoined together, dropping those of one rule from negation of each rule. Since there cannot be that do not have a pair in either operand. The conjunction a string that is derived from both a rule and its negation, there of two definitions is represented by the conjunction of the does not exist a derivation using the original ruleset. B Any specialization of their corresponding nonterminals. string that is derived by the original ruleset will not be derived Proof: C Any string derived by the result would be derived by by the new because the new is composed of negations from one of the conjoined rulesets, which is a conjunction of rulesets each rule in the original. from operands. B If a string was derived by both operands, a ruleset exists in both that derives the string. A conjunction of H. Conjunction of two rulesets all such ruleset pairs exists in the new definition. Result of conjunction of two rulesets derives (any and L. Disjunction of two definitions all) strings derived by both of the operands. For conjunction between two rulesets, we take one rule from each, and compute The result of disjunction derives (any and all) strings derived the conjunction of both. The new rules will be all such by any of the operands. Disjunction of two definitions is conjunctions: {r1 ∧ r2 : r1 ∈ S1, r2 ∈ S2} again simply a combination of rulesets of the same kind from Proof: C Any string that is derived by one of the rules in the each operand, and is represented by the disjunction of the new ruleset will have been derived by at least one rule from specialization of their corresponding nonterminals. both operand rulesets. B If both operand rulesets derived the Proof: C Any string derived by the new definition was derived same string, then there exist a rule in both that derived it, and by one of the rulesets, which came from one of the operands. the new ruleset contains conjunction of all such rules. B Pick an operand, and say it derived a string. The ruleset from that operand is part of the new definition. I. Disjunction of two rulesets The result of disjunction of two rulesets derives (any and M. Negation of a specialized grammar all) strings derived by any of the operands. Disjunction of two Negation of an arbitrary grammar specialization is pro- rulesets is a new ruleset with rules from both, merging rules duced by first negating its start symbol, and constructing that can be merged. the negations (and other expressions) for any specialized nonterminals that is needed from the definition of the start VI.IMPLEMENTATION AND USAGE SCENARIOS symbol recursively. Given below is the grammar for evocative We have implemented Evogram as a standalone Jupyter4 hjson i hitem i hstringi:null expression N where N is . notebook with detailed steps and examples5. The grammars are hjson i ::= helt i N N accepted and produced in the Fuzzingbook [19] canonical helt i ::=‘ false’ | ‘null’ | ‘true’ N format, and the evocative patterns in the DDSET format. |h numberi|h stringi|h array i|h object i N N Grammars can be converted from and to other popular formats, harray i ::=‘ []’ | ‘[’ helts i ‘]’ N N such as ANTLR. hobject i ::=‘ {}’ | ‘{’ hitems i ‘}’ N N We see the following usage scenarios for Evogram: helts i ::= helt i|h elt i ‘,’ helts i N N N N Precise control during fuzzing. Given a particular pattern or hitems i ::= hitem i|h item i ‘,’ hitems i N N N N a set of patterns that is associated with some behavior hitem i ::= hstringi ‘:’ helt i N N∧N1 of the program, one can use grammar specializations to helt i ::=‘ false’ | ‘true’ N∧N1 specify that inputs generated satisfy arbitrary Boolean |h numberi|h stringi|h array i|h object i N N constraints, inducing certain failures while preventing N. Conjunction of two grammar specializations others. The utility of such a grammar is that, one can The conjunction between two grammar specializations is precisely ask the fuzzer to concentrate on portions of code, produced by first conjoining their start symbol, and hence their or portions of input space one wants to explore further, corresponding definitions recursively. The following is the rest avoiding the behaviors that one does not want to trigger. of the specialized nonterminals for the grammar specialization Testing and validating fixes. The above control over test in- representing Equation (4). puts works particularly well if a concrete failure-inducing input can be generalized towards an abstract failure- hjsonE∧N i ::= heltE∧N i inducing input, as produced by DDSET [25]. When fed heltE∧N i ::= harrayE∧N i|h objectE∧N i with such patterns, the Evogram produces a grammar harrayE∧N i ::=‘ [’ heltsE∧N i ‘]’ whose produced strings all contain instantiated patterns, hobjectE∧N i ::=‘ {’ hitemsE∧N i ‘}’ helts i ::= helt i|h elt i ‘,’ helts i and thus variants of the original failure-inducing input E∧N E∧N E∧N N in all sorts of contexts—which makes them helpful for |h eltN i ‘,’ heltsE∧N i helt i ::=‘ false’ | ‘true’ validating fix correctness. N∧N1 |h numberi|h stringi|h objectN i|h arrayN i Validating inputs. Assuming that a certain vulnerability is

hitemsE∧N i ::= hitemE∧N i|h itemE∧N i ‘,’ hitemsN i known to be induced by a failure inducing pattern under |h itemN i ‘,’ hitemsE∧N i specific contexts, one can encode the specific context hitem i ::= hstring i ‘:’ helt i E∧N E1 N∧N1 under which the failure is induced, and allow only the |h stringi ‘:’ helt i E∧N∧N1 negation of this pattern to proceed to the program, rejecting helt i ::= harray i|h object i E∧N∧N1 E∧N E∧N any string that may induce the failure. This provides O. Disjunction of grammars the developers with a quick fix tool for identifying and quickly rejecting potentially malicious inputs such as SQL Disjunction of two grammars is built by disjunction of their injections without the knowledge of the internals of the start symbol, and corresponding definitions recursively. program. P. Constructing definitions corresponding to nonterminals Algebra of oracles. The abstract behavior-inducing inputs are mined from a given program and a given predicate To convert an evocative expression to grammar, we try to using DDSET. Such abstract inputs correspond to specific construct the start symbol of the new grammar. To do that, program behaviors and are directly representable as we need to apply the given operation to the corresponding evocative patterns. Given such evocative patterns, we definitions of the start symbols of the subject grammars. can build evocative expressions that represent complex These in turn require further specialized nonterminals to be program behaviors and can be used to validate the program computed, which is done recursively until no more specialized behavior for corresponding inputs. nonterminals need to be constructed. Supercharged recognizers. Regular expressions (regexes) are At each step, the nonterminal to be constructed is simplified typically used to match and extract parts of input by into its canonical DNF form. We then reconstruct this term programmers. These inputs may often be encoded in from our smallest specializations (i.e the specializations that a structured format such as JSON, XML, HTML, and correspond to a single pattern) that can be directly reconstructed SEXPR. However, the siren song of regular expressions is from the evocative patterns. hard to ignore6. While the situation has started to change We note that one can also collect all the Boolean expressions with projects such as Combi [27], SemGrep [28], and for the same nonterminal from operand grammars, and solve Coccinelle [29] that rely on the underlying grammar, them to reconstruct the new term in terms of available terms. That is, the algebraic operations can be done on conforming 4https://jupyter.org grammars even if the smallest terms are not known, and the 5https://zenodo.org/record/4456296 nonterminals are not labelled with pattern expressions. 6https://stackoverflow.com/q/1732348/1420407 the constructs they use are limited (disjunction and at TABLE I: Percentage of inputs generated with evocative best conjunction), and the expressions need to be written strategy that induces failure (F ) and non-evocative strategy by hand, which limits their usability. The evocative that did not induce failure (¬F ). expressions can be 1) automatically mined from sample Bug F ¬F expressions and 2) combined to form arbitrarily complex find 07b941b1 100% of 100 96% of 100 find 93623752 100% of 100 97% of 100 matching specifications. find c8491c11 100% of 100 93% of 100 Further, evocative expressions can be extended to support find dbcb10e9 100% of 100 89% of 100 constraints on each term such as length < 10 or grep 3c3bdace 100% of 100 100% of 96 a defined grep 54d55bba 100% of 100 100% of 92 variable is . The relational algebra of such grep 9c45c193 100% of 1 100% of 94 constraints can be easily added to the underlying Boolean Total 100% of 701 99% of 682 algebra. Generating data structures. Algebraic data types (except GADTs) have a 1:1 mapping to context-free grammars, fragments into the grammar. The resulting grammar (we and property checkers such as QuickCheck rely on call this the evocative grammar, and the strategy the generation of data structures to verify properties. The evocative strategy) is then used for input generation. The specialized grammars from evocative expressions could second strategy negates the previous grammar, and the precisely specify how and what properties these structures resulting grammar (we call this the non-evocative grammar should contain. and the strategy the non evocative strategy) is used to Configuring and customizing systems. Most hierarchical produce inputs. The inputs thus generated are fed to the structures can be represented as derivation trees from program, and the program is monitored for the expected context-free grammars. Such structures include URLs, behavior and its absence. file systems, GUI navigation, object hierarchies, and • For the evaluation as recognizers, we require a source more. All these structures have some concept of access of inputs that induces the failure, as well as a source of control defining under what conditions certain elements inputs that is guaranteed not induce the failure. Since can be reached, or certain actions can be performed. Such we are using the subjects from DDSET, we use inputs access control rules could be expressed via evocative generated during the pattern mining for this purpose. patterns, specializing general access rules towards specific That is, we collect the semantically valid strings generated environments. during minimization and abstraction, which were found VII.EVALUATION to have induced a failure or otherwise. We then evaluate We have evaluated Evogram and grammar specializations whether the specialized grammars produced by Evogram both in a testing context (producing inputs that contain specific using the evocative strategy are capable of recognizing evocative patterns) and in a prevention context (checking evocative inputs, and rejecting the non-evocative inputs. whether inputs contain a specific pattern). Specifically, we Similarly, we evaluate whether the specialized grammars pose the following research questions: produced by Evogram using non-evocative strategy are capable of rejecting any failure inducing input. RQ1 How effective are the specialized grammars produced by Evogram in producing expressions that contain (and B. Evaluation Results and Discussion do not contain) any evocative fragments? Our evaluation results are as follows: The result of gener- RQ2 How effective are the generated specialized grammars in ating evocative inputs using the evocative and non-evocative recognizing expressions that contain (or do not contain) strategies, tested against the program for reproducing the failure evocative fragments? is given in TableI. The F column contains the percentage A. Evaluation Setup of inputs using evocative strategy that induced the failure when tested against the program. The ¬F column contains the For evaluation, we use the subjects from DDSET [25]. The percentage of inputs using non-evocative strategy that did not first are programming language interpreters: clojure, closure, induce the failure when tested against the program. rhino and lua. Input languages for these subjects are more From TableI, we find that specialized grammars generated constrained than simple context-free languages, with the ability by Evogram can produce inputs that contain the given evocative to declare and use variables and functions. When using a fragment, and reproduce the expected failure with high accuracy grammar-based fuzzer, the probability of randomly generating (100%) when used as a test generator for UNIX utilities. Further, a semantically valid string is extremely low. Hence, for such when used to produce inputs that do not contain the evocative language interpreters, we rely on evaluating the specialized fragment, Evogram succeeds in avoiding such failure inducing grammars only as recognizers. For the UNIX utilities find and fragments in its inputs with an accuracy of 99.0%. The bug grep, only syntactic validity is required. Hence, we evaluate grep 9c445c193 could not be abstracted. Hence, only a single these utilities as both producers and recognizers. input was produced during fuzzing, and only three (substrings) • For the evaluation as producers, we generate inputs using recognized. two strategies: The first strategy is to insert evocative TABLE II: Recognizing failure inducing inputs with F and as in inducing the requisite failure. Similarly, the same table non-failure inducing inputs with ¬F . shows that we can generate inputs that will not contain the Bug F ¬F problematic patterns, thus freeing the fuzzer to explore rarer find 07b941b1 89.7% of 213 100% of 7 bugs. As our results from TableII show, evocative expressions find 93623752 99.5% of 193 100% of 7 find c8491c11 98.5% of 196 100% of 8 shine in the role of recognizers of specific patterns in input, find dbcb10e9 100% of 374 100% of 6 and can work in the role of a validator for inputs. grep 3c3bdace 94.6% of 203 100% of 28 grep 54d55bba 83.1% of 213 100% of 24 C. Threats to Validity grep 9c45c193 33.3% of 3 100% of 11 clojure 2092 80.2% of 360 99.0% of 100 Our empirical evaluation is subject to the following threats clojure 2345 99.2% of 393 96.1% of 52 to validity: clojure 2450 92.7% of 704 99.6% of 267 clojure 2473 93.7% of 695 98.3% of 58 External Validity. The external validity (generalizability) of clojure 2518 44.4% of 9 98.7% of 76 our results depends on the representativeness of our data clojure 2521 24.3% of 847 99.0% of 111 set. Our subjects were a small set of language interpreters closure 1978 94.8% of 1672 98.0% of 101 closure 2808 100% of 64 88.8% of 27 and two UNIX utilities. Further, only a few bugs were closure 2842 1.5% of 261 98.9% of 176 evaluated, and their patterns chosen for experiments. closure 2937 36.6% of 1293 91.4% of 35 Hence, it is possible that our subjects are not representative closure 3178 95.1% of 719 98.0% of 99 closure 3379 57.6% of 646 80.7% of 26 of the real world. The main mitigation here is that these lua 5.3.5 4 3.1% of 161 98.6% of 142 were actual bugs found in the wild, logged by real people. rhino 385 65.7% of 472 93.3% of 30 Further, the grammars are from popular and complex real rhino 386 97.8% of 363 93.2% of 44 world input and programming languages. Total 73.2% of 10054 97.8% of 1435 Internal Validity. The threat to internal validity (correctness of evaluation) of our results is mitigated by careful verification (including formal proofs) of our algorithms. Evogram grammars induce expected failures with 100% Construct Validity. To mitigate the threat to construct validity accuracy and prevent such failures with 99.0% accuracy. (the degree to which a test measures what it claims to be measure of), we measure how accurate our specialized The result of recognizing evocative inputs using the evocative grammars are as recognizers and generators, which are strategy is given in F column of TableII. The result of the main ways we expect practitioners to use our tool. recognizing non-evocative inputs using the non-evocative strategy is given the ¬F column. VIII.RELATED WORK Looking at Evogram’s performance as a recognizer, from A. Grammar-Based Testing TableII we find that grammars from Evogram are able to Already in 1970, Hanford suggested the use of gram- recognize the failure inducing inputs for most bugs. For a mars to systematically produce valid inputs [7]. Today’s number of bugs, however (lua 5.3.5 4, clojure 2521, closure approaches [19] combine specification-based producers with 2937, clojure 2518, grep 9c45c193), less than half of the symbolic constraint solving [8], reusing fragments from bug failure-inducing inputs were predicted as such. This is due reports [11], search-based strategies [9], [10], grammar cover- to the DDSET pattern specializing to the first failure-inducing age [12], [13], annotations [14], search heuristics [15], power bug, which then leads to an overspecialization in the resulting schedules [16], and local input generators [17]; search-based grammar. However, even with this caveat, the grammars grammar testing is now also applied for security testing [18]. produced by Evogram are able to recognize failure inducing All these tools and approaches can immediately benefit from inputs with 73.2% accuracy and non-failure inducing inputs specialized grammars as produced by Evogram, targeting with 97.8% accuracy. specific input subsets or the locations that process them.

Evogram grammars recognize failure inducing inputs with B. Patterns and Pattern Languages 73.2% and non-failure inducing inputs with 97.8% accuracy. In Software Engineering, patterns have been frequently used to obtain specific search and configuration results. Besides the Evocative expressions represent specialized context-free ubiquitous regular expressions and wildcards, a number of grammars that guarantee production of the requisite pattern approaches allow to express structural properties. somewhere in the input string produced. However, having the Combi [27] shows how patterns based on a base context particular pattern somewhere in the string is no guarantee that free grammar can be used for identifying and transforming a given failure is induced. For example, the null check may structures in code. Semgrep [28] by r2c is another tool that be ignored under specific circumstances. Hence, it is important allows programmers to specify patterns similar to evocative to validate how effective the grammars are, in converting the patterns. Note, though, that both are for recognition, which is a input patterns to actual failures. Our results from TableI using simpler problem than generation, as one can run the matching real world bugs show that the evocative expressions work well rules as as sequence of filters. We note that while the patterns in generating inputs that contain the failure pattern, as well used by Combi are similar to the evocative patterns, they are meant to be specified by hand only. Combi and Semgrep produce inputs that satisfy structural as well as other allow conjunction and disjunction of patterns, but algebras and constraints. negations are not supported. Ambiguous grammars. Our algebra is restricted to non- Micro-grammars [30] is another general approach for partial ambiguous context-free base grammars. However, it may parsing and semantic analysis. be extended to all context-free grammars by taking into Another related work is Coccinelle [29] that defines a account the fact that there could be multiple parse trees pattern language for transformations of C programs such as for an evocative pattern. the Linux kernel. While Coccinelle supports disjunction of We provide a well explained Jupyter notbook7 that details our patterns, conjunction and negation are not supported. technique as well as a partial mechanization of proofs (using Conjunctive and Boolean grammars [31] are a generalization HOL4) for verification8. The complete replication package is over context-free grammars, and specify grammars based on available as a virtual machine at conjunction, (implicit) disjunction and negation of arbitrary DOI:10.5281/zenodo.4456296 context-free grammars. While one can reuse context-free grammar parsers (by applying one grammar after another), it is REFERENCES not clear whether efficient producers exist for such languages. [1] W. M. McKeeman, “Differential testing for software,” Digital Technical Journal, vol. 10, no. 1, pp. 100–107, 1998. [Online]. Available: C. Generalizers https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.83.445 Generalizers are tools that turn a concrete input into a [2] M. Hoschele¨ and A. Zeller, “Mining input grammars from dynamic taints,” in IEEE/ACM Automated Software Engineering, ser. ASE 2016. more general pattern with similar effect. DDSET [25] generates New York, NY, USA: ACM, 2016, pp. 720–725. [Online]. Available: failure patterns from any given failure inducing input and http://doi.acm.org/10.1145/2970276.2970321 the corresponding predicate. SmartCheck from Lee Pike [32] [3] R. Gopinath, B. Mathis, and A. Zeller, “Mining input grammars from dynamic control flow,” in Proceedings of the ACM Joint European generalizes algebraic data structures. Extrapolate by Braquehais Software Engineering Conference and Symposium on the Foundations et al. [33], generalizes counter examples of functional test of Software Engineering (ESEC/FSE), 2020. [Online]. Available: properties. Groce et al. [34] show one can identify canonical https://publications.cispa.saarland/3101/ [4] Y. Chen, A. Groce, C. Zhang, W.-K. Wong, X. Fern, E. Eide, minimal tests with annotations for generalizations. In our and J. Regehr, “Taming compiler fuzzers,” in Proceedings of context, the generalized patterns all serve as starting point the 34th ACM SIGPLAN conference on Programming language for evocative patterns. None of these tools, would produce design and implementation, 2013, pp. 197–208. [Online]. Available: https://doi.org/10.1145/2491956.2462173 Boolean combinations of patterns. [5] A. Groce, C. Zhang, E. Eide, Y. Chen, and J. Regehr, “Swarm testing.” New York, NY, USA: Association for Computing Machinery, 2012. IX.CONCLUSIONAND FUTURE WORK [Online]. Available: https://doi.org/10.1145/2338965.2336763 [6] R. Padhye, C. Lemieux, K. Sen, M. Papadakis, and Y. Le Traon, With Evogram, we have introduced the concept of a “Semantic fuzzing with Zest,” in Proceedings of the 28th ACM SIGSOFT grammar transformer, specializing a grammar towards Boolean International Symposium on Software Testing and Analysis, 2019, pp. 329–340. [Online]. Available: https://doi.org/10.1145/3339069 combinations of pattern occurrences. Such transformations [7] K. V. Hanford, “Automatic generation of test cases,” IBM Systems give testers unprecedented control over the test inputs to be Journal, vol. 9, no. 4, pp. 242–257, Dec. 1970. [Online]. Available: produced. The transformations are proven to be correct; an https://doi.org/10.1147/sj.94.0242 [8] P. Godefroid, A. Kiezun, and M. Y. Levin, “Grammar-based whitebox evaluation shows the usefulness of transformed grammars for fuzzing,” in ACM SIGPLAN Conference on Programming Language producing and checking patterns. Design and Implementation, 2008, pp. 206–215. [Online]. Available: Our future work will focus on the following: https://doi.org/10.1145/1375581.1375607 [9] F. M. Kifetew, R. Tiella, and P. Tonella, “Combining stochastic Applications. The lion’s share of our future work will focus grammars and genetic programming for coverage testing at the on further usage scenarios hinted at in SectionVI, system level,” in International Symposium on Search Based Software Engineering, 2014, pp. 138–152. [Online]. Available: https://doi.org/10. from parsing over pattern matching to configuration and 1007/978-3-319-09940-8 10 adaptation. [10] S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege, “A Richer specifications. We have described how to insert at systematic review of the application and empirical investigation of search-based test case generation,” IEEE Transactions on Software least one failure-inducing fragment into the input (or not). Engineering, vol. 36, no. 6, pp. 742–762, 2010. [Online]. Available: What if one wants a particular number of fragments https://doi.org/10.1109/TSE.2009.52 (say, exactly one)? What if one wants sequences of [11] C. Holler, K. Herzig, and A. Zeller, “Fuzzing with code fragments.” in USENIX Conference on Security Symposium, 2012, pp. 445–458. fragments (one pattern should always follow another [Online]. Available: https://www.usenix.org/conference/usenixsecurity12/ one)? Such constraints can still be represented in a technical-sessions/presentation/holler context-free grammar, and we will be investigating specific [12] R. Lammel,¨ “Grammar testing,” in Fundamental Approaches to Software Engineering (FASE), H. Hussmann, Ed., 2001, pp. 201–216. [Online]. transformations for them. Available: https://doi.org/10.1007/3-540-45314-8 15 Constrained grammars. Some input properties cannot be [13] N. Havrikov and A. Zeller, “Systematically covering input structure,” easily expressed in a context-free grammar; numerical 2019, pp. 189–199. [Online]. Available: https://doi.org/10.1109/ASE. 2019.00027 properties is a typical example. We want to attach (Turing- complete) constraints to individual patterns and then use 7https://nbviewer.jupyter.org/github/vrthra/Ewoks/blob/master/src/FAlgebra.ipynb transformation rules together with constraint solving to 8https://github.com/vrthra/Ewoks/tree/master/mechanization [14] F. M. Kifetew, R. Tiella, and P. Tonella, “Generating valid grammar-based [26] “HOL4,” https://hol-theorem-prover.org/. [Online]. Available: https: test inputs by means of genetic programming and annotated grammars,” //hol-theorem-prover.org/ Empirical Software Engineering, vol. 22, no. 2, pp. 928–961, 2017. [27] R. van Tonder and C. Le Goues, “Lightweight multi-language syntax [Online]. Available: https://doi.org/10.1007/s10664-015-9422-4 transformation with parser parser combinators,” in Proceedings of [15] R. Hodovan,´ A.´ Kiss, and T. Gyimothy,´ “Grammarinator: a the 40th ACM SIGPLAN Conference on Programming Language grammar-based open source fuzzer,” in Proceedings of the 9th ACM Design and Implementation, 2019, pp. 363–378. [Online]. Available: SIGSOFT International Workshop on Automating TEST Case Design, https://doi.org/10.1145/3314221.3314589 Selection, and Evaluation, 2018, pp. 45–48. [Online]. Available: [28] r2c, “Lightweight static analysis for many languages. find and block https://doi.org/10.1145/3278186.3278193 bug variants with rules that look like source code.” https://semgrep.dev/, [16] V. Pham, M. Bohme,¨ A. E. Santosa, A. R. Caciulescu, and A. Roy- 2020, accessed: 2020-08-19. choudhury, “Smart greybox fuzzing,” IEEE Transactions on Software [29] Y. Padioleau, J. L. Lawall, and G. Muller, “Understanding collateral Engineering, to appear; preprint at https://arxiv.org/abs/1811.09447. evolution in Linux device drivers,” in Proceedings of the 1st ACM [17] R. Padhye, C. Lemieux, K. Sen, M. Papadakis, and Y. Le Traon, SIGOPS/EuroSys European Conference on Computer Systems 2006, 2006, “Semantic fuzzing with Zest,” 2019, pp. 329–340. [Online]. Available: pp. 59–71. [Online]. Available: https://doi.org/10.1145/1217935.1217942 https://doi.org/10.1145/3293882.3330576 [30] F. Brown, A. Notzli,¨ and D. Engler, “How to build static checking systems [18] C. Aschermann, T. Frassetto, T. Holz, P. Jauernig, A.-R. Sadeghi, and using orders of magnitude less code,” in Proceedings of the Twenty-First D. Teuchert, “NAUTILUS: Fishing for deep bugs with grammars.” International Conference on Architectural Support for Programming in NDSS, 2019. [Online]. Available: https://www.ndss-symposium.org/ Languages and Operating Systems, 2016, pp. 143–157. ndss-paper/nautilus-fishing-for-deep-bugs-with-grammars/ [31] A. Okhotin, “Conjunctive and Boolean grammars: The true general case [19] A. Zeller, R. Gopinath, M. Bohme,¨ G. Fraser, and C. Holler, of the context-free grammars,” Computer Science Review, vol. 9, pp. 27 “The fuzzing book,” in The Fuzzing Book. Saarland University, – 59, 2013. [Online]. Available: http://www.sciencedirect.com/science/ 2019, retrieved 2019-09-09 16:42:54+02:00. [Online]. Available: article/pii/S157401371300018X https://www.fuzzingbook.org/ [32] L. Pike, “SmartCheck: automatic and efficient counterexample [20] R. Gopinath and A. Zeller, “Building fast fuzzers,” 2019. reduction and generalization,” in Proceedings of the 2014 ACM [21] J. Earley, “An efficient context-free parsing algorithm,” Communications SIGPLAN symposium on Haskell, 2014, pp. 53–64. [Online]. Available: of the ACM, vol. 26, no. 1, pp. 57–61, 1983. https://doi.org/10.1145/2633357.2633365 [22] M. Tomita, Efficient parsing for natural language: a fast algorithm for [33] R. Braquehais and C. Runciman, “Extrapolate: generalizing practical systems. Springer Science & Business Media, 2013, vol. 8. counterexamples of functional test properties,” in Proceedings [23] D. Grune and C. J. Jacobs, “Parsing techniques,” Monographs in of the 29th Symposium on the Implementation and Application of Computer Science. Springer,, p. 13, 2007. Functional Programming Languages, 2017, pp. 1–11. [Online]. Available: [24] E. Scott and A. Johnstone, “Gll parsing,” Electronic Notes in Theoretical https://doi.org/10.1145/3205368.3205371 Computer Science, vol. 253, no. 7, pp. 177–189, 2010. [34] A. Groce, J. Holmes, and K. Kellar, “One test to rule them all,” [25] R. Gopinath, A. Kampmann, N. Havrikov, E. Soremekun, and A. Zeller, in Proceedings of the 26th ACM SIGSOFT International Symposium “Abstracting failure-inducing inputs,” in Proceedings of the 2020 Interna- on Software Testing and Analysis. ACM, 2017, pp. 1–11. [Online]. tional Symposium on Software Testing and Analysis. ACM, 2020. Available: https://doi.org/10.1145/3092703.3092704