arXiv:1810.05661v4 [cs.PL] 13 Mar 2020 09Cprgthl yteonrato() ulcto rights Publication owner/author(s). the by held Copyright 2019 © hoyo computation of Theory aevrfiainadvalidation and verification ware C eeec Format: Reference ACM hsi h uhrsvrino h ok ti otdhr o your for here posted is It work. the of version author’s the is This aacit In JavaScript. ug einadIpeetto PD 1) ue2–6 201 22–26, June USA ’19), (PLDI Implementation and Design guage s.Ntfrrdsrbto.TedfiiieVrino eodwspubl was Record of Version in definitive The redistribution. for Not use. ssudfrdnmcsmoi xcto fthe of execution symbolic dynamic for which sound (ES6), 2015 is reg- ECMAScript complete of the language for expression model ular a present we paper, this missed In or bugs. coverage test poor to impre- leading approximated, or cisely ignored are matching, greedy or backreferences as such engines, expression regular mainstream Com- of insufficient. aspects is mon finding bug and generation test for tools execution-based symbolic in expressions regular for Support Abstract oACM. to USA AZ, Phoenix, 2019, 22–26, June ’19, PLDI h ps://doi.org/10.1145/3314221.3314645 C Concepts CCS coverage. boost therefore and successful queries of expression number regular the increase significantly can and effec- tive is strategy containing the packages that demonstrating Node.js expressions, 1,000 regular over on it evaluated and JavaScript, for engine in execution symbolic model dynamic our a ExpoSE, implemented We greediness. matching and of precedence problem the address to and scheme expressions refinement a regular use classical and constraints string ing exec 22,21,Ponx Z USA. AZ, Phoenix, 2019, 22–26, ’19) (PLDI Implementation and Design Language Programming Executi Symbolic Dynamic S for 2019. Semantics Expression Kinder. Regular Johannes and Mitchell, Duncan Loring, Blake SMT expressions, lar Keywords rceig fte4t C IPA ofrneo Programmi on Conference SIGPLAN ACM 40th the of Proceedings oa olwy nvriyo London of University Holloway, Royal , on eua xrsinSmnisfrDynamic for Semantics Expression Regular Sound h ps://doi.org/10.1145/3314221.3314645 ucin.W oe eua xrsinoeain us- operations expression regular model We functions. [email protected] nomto euiyGroup Security Information yai yblceeuin aacit regu- JavaScript, execution, symbolic Dynamic rceig fte4t C IPA ofrneon Conference SIGPLAN ACM 40th the of Proceedings ntdKingdom United lk Loring Blake • otaeadisengineering its and Software yblcEeuino JavaScript of Execution Symbolic → C,NwYr,N,USA, NY, York, New ACM, eua languages Regular ; . yai analysis Dynamic oa olwy nvriyo London of University Holloway, Royal [email protected] eateto optrScience Computer of Department ,Ponx AZ, Phoenix, 9, . test → 15 ucnMitchell Duncan personal licensed gLan- ng ntdKingdom United pages. June , Soft- ound nof on ished and • ; 1 write hc lo xrsigcntanssc as such constraints expressing allow which eas dp ntermidro hsppraentlim- not paper—are this of remainder the in adopt also we hsi rbe,bcuete r ieyue,a edemon- we as used, widely are they because problem, a is This (the n final a and vlae oanwsrn hr h rtsc curnein occurrence such first the s where string new a to evaluates otewr rbe nrglrlnugsicuecapture include around languages parentheses regular (the in groups regexes problem of word translation the twice). a to appear prevent must (be- that that string language features arbitrary Problematic context-sensitive an a is tag is the cause which tags, XML ing ebrhpi o-eua agae.Tedffiut arises difficulty The languages. include non-regular to in have membership would cover- constraints of where loss bugs to missed lead or age can support of lack this generation, test s rmiglnugs o ntne nJvSrp,oecan one JavaScript, in instance, For pro- many languages. by supported gramming are and strings substituting match- and for ing developers with popular are expressions Regular Introduction 1 expression called JavaScript—often or Perl like languages programming in expressions” “regular but sense, theoretical solvers. SMT modern by solved ficiently strings for theories [ expressions support regular classical often and now solvers theory) ulo taein strate approximated. imprecisely or ignored usually are features these languages. programs, string-manipulating sensitive about reasoning context tools In of intersection encodes tively reynnged acigpeeec fsubexpressions of precedence matching greedy/non-greedy (the backreferences ol lob nlddi lookahead a in included be also could fsrnsi neial [ undecidable is strings of the for fspotfrrglrepesosbcuete r ocom- so are they [ mon because expressions regular for support of tdt ersnigrglrlnugs[ languages regular representing to ited contains srpae ihtestring the with replaced is ntecneto yai yblceeuin(S)for (DSE) execution symbolic dynamic of context the In M ovr upr eua xrsin ntelanguage- the in expressions regular support solvers SMT eea etn n eicto ol nld oedegree some include tools verification and testing Several .*? 24 /goo+d/.test(s) test , §7.1 27 snnged) nadto,aysc expression such any addition, In non-greedy). is "go" "d" /<(\w+)>.*?<\/\1>/ , 29 . xml bv.Atog n eea theory general any Although above. example , Similarly, . olwdb n rmr curne of occurrences more or one by followed , 34 , 37 \1 .I atclr M stsaiiymod- (satisfiability SMT particular, In ]. udserUiest Munich University Bundeswehr eern otecpuegop,and group), capture the to referring [email protected] ots hte h tigvleof value string the whether test to eerhIsiueCODE Institute Research 7 s.replace(/goo+d/,"better") ,mn tigcntansaeef- are constraints string many ], oansKinder Johannes "better" \w+ Germany 1 assaypi fmatch- of pair any parses , 2 , nteeapeabove), example the in 6 . , 3 15 (?=) .Frisac,the instance, For ]. , 25 s , hc effec- which , 26 regex L( ∈ , 34 term a , , goo+d 38 – "o" 40 ) ], PLDI’19,June22–26,2019,Phoenix,AZ,USA BlakeLoring,Duncan Mitchell, and Johannes Kinder from the typical mixing of constraints in path conditions— offer the match, split, search and replace methods that simply generating a matching word for a standalone regex is expect a RegExp argument. easy (without lookaheads). To date, there has been only lim- A regex accepts a string if any portion of the string matches ited progress on this problem, mostly addressing immediate the expression, i.e., it is implicitly surrounded by wildcards. needs of implementations with approximate solutions, e.g., The relative position in the string can be controlled with an- for capture groups [29] and backreferences [27, 30]. How- chors, with ^ and $ matching the start and end, respectively. ever, neither matching precedence nor lookaheads have been Flags in regexes can modify the behavior of matching op- addressed before. erations. The ignore case flag i ignores character cases when In this paper, we propose a novel scheme for support- matching. The multiline flag m redefines anchor characters ing ECMAScript regex in dynamic symbolic execution and to match either the start and end of input or newline char- show that it is effective in practice. We rely on the specifica- acters. The unicode flag u changes how unicode literals are tion of regexes and their associated methods in ECMAScript escaped within an expression. The sticky flag y forces match- 2015 (ES6). However, our methods and findings should be ing to start at RegExp.lastIndex, which is updated with easily transferable to most other existing implementations. the index of the previous match. Therefore, RegExp objects In particular, we make the following contributions: become stateful as seen in the following example: • We fully model ES6 regex in terms of classical regular r = /goo+d/y; languages and string constraints (§4) and cover sev- r.test("goood"); // true; r.lastIndex = 6 eral aspects missing from previous work [27, 29, 30]. r.test("goood"); // false; r.lastIndex = 0 We introduce the notion of a capturing language to The meaning of the global flag g varies. It extends the effects make the problem of matching and capture group as- of match and replace to include all matches on the string signment self-contained. and it is equivalent to the sticky flag for the test and exec • We introduce a counterexample-guided abstraction re- methods of RegExp. finement (CEGAR) scheme to address the effect of greed- iness on capture groups (§5), which allows us to de- 2.2 Capture Groups ploy our model in DSE without sacrificing soundness Parentheses in regexes not only change operator precedence for under-approximation. (e.g., (ab)* matches any number of repetitions of the string • We present the first systematic study of JavaScript "ab" while ab* matches the character "a" followed by any regexes, examining feature usage across 415,487 pack- number of repetitions of the character "b") but also cre- ages from the NPM software repository. We show that ate capture groups. Capture groups are implicitly numbered non-regular features are widely used (§7.1). from left to right by order of the opening parenthesis. For 1 2 In the remainder of the paper we review ES6 regexes (§2) example, /a|((b)*c)*d/ is numbered as /a|( ( b)*c)*d/. and present an overview of our approach by example (§3). Where only bracketing is required, a non-capturing group We then detail our regex model using a novel formulation (§4), can be created by using the syntax (?: ... ). and we propose a CEGAR scheme to address matching prece- For regexes, capture groups are important because the dence (§5). We discuss an implementation of the model as regex engine will record the most recent substring matched part of the ExpoSE symbolic execution engine for JavaScript against each capture group. Capture groups can be referred (§6) and evaluate its practical impact on DSE (§7). Finally, we to from within the expression using backreferences (see §2.3). review related work (§8) and conclude (§9). The last matched substring for each capture group is also returned by some of the API methods. In JavaScript, the re- turn values of match and exec are arrays, with the whole 2 ECMAScript Regex match at index 0 (the implicit capture group 0), and the last We review the ES6 regex specification, focusing on differ- matched instance of the ith capture group at index i. In the ences to classical regular expressions. We begin with the example above, "bbbbcbcd".match(/a|((b)*c)*d/) will regex API and its matching behavior (§2.1) and then explain evaluate to the array ["bbbbcbcd", "bc", "b"]. capture groups (§2.2), backreferences (§2.3), and operator precedence (§2.4). ES6 regexes are comparable to those of 2.3 Backreferences other languages but lack Perl’s recursion and lookbehind A backreference in a regex refers to a numbered capture group and do not require POSIX-like longest matches. and will match whatever the engine last matched the cap- ture group against. In general, the addition of backreferences 2.1 Methods, Anchors, Flags to regexes makes the accepted languages non-regular [3]. ES6 regexes are RegExp objects, created from literals or the Inside quantifiers (Kleene star, Kleene plus, and other rep- RegExp constructor. RegExp objects have two methods, test etition operators), the string matched by the backreference and exec, which expect a string argument; String objects can change across multiple matches. For example, the regex 2 SoundRegularExpressionSemanticsforDSEofJavaScript PLDI ’19, June 22–26, 2019, Phoenix, AZ, USA
Table 1. Regular expression operators, separated by classes 1 let timeout = '500'; of precedence. 2 for (let i = 0; i < args.length; i++) { 3 let arg = args[i]; Operator Name Rewriting 4 let parts = /<(\w+)>([0-9]*)<\/\1>/.exec(arg); if (parts) { (r) Capturing parentheses 5 if \n Backreference 6 (parts[1] === "timeout") { (?:r) Non-capturing parentheses 7 timeout = parts[2]; (?=r) Positive lookahead 8 } (?!r) Negative lookahead 9 ... \b Word boundary 10 } \B Non-word boundary 11 } assert r * Kleene star 12 (/^[0-9]+$/.test(timeout) == true); r *? Lazy Kleene star Listing 1. Code example using regex. r + Kleene plus r *r r +? Lazy Kleene plus r *?r r{m,n} Repetition rn| ... |rm introduce our model for regex by example (§3.3) and explain m n r{m,n}? Lazy repetition r | ... |r how to eliminate spurious solutions by refinement (§3.4). r? Optional r|ϵ r?? Lazy optional ϵ|r 3.1 The Word Problem and Capturing Languages For any given classical regular expression r, we write w ∈ r1r2 Concatenation L(r) whenever w is a word within the (regular) language r1|r2 Alternation generated by r. For a regex R, we also need to record values of capture groups within the regex. To this end, we intro- duce the following notion: /((a|b)\2)+/ can match the string "aabb", with the back- reference \2 being matched twice: the first time, the cap- Definition1 (Capturing Language). The capturing language ture group contains "a", the second time it contains "b". of a regex R, denoted Lc (R), is the set of tuples (w, C0,..., Cn) This logic applies recursively, and it is possible for backref- such thatw isawordofthelanguageof R and each C0,..., Cn erences to in turn be part of outer capture groups. is the substring of w matched by the corresponding num- bered capture group in R. 2.4 Operator Evaluation A word w is therefore matched by a regex R if and only if We explain the operators of interest for this paper in Table 1; ∃C0,..., Cn : (w, C0,..., Cn) ∈ Lc (R). It is not matched if the implementation described in §6 supports the full ES6 and only if ∀C0,..., Cn : (w, C0,..., Cn) < Lc (R). For read- syntax [14]. Some operators can be rewritten into semanti- ability, we will usually omit quantifiers for capture variables cally equivalent expressions to reduce the number of cases where they are clear from the context. to handle (shown in the Rewriting column). Regexes distinguish between greedy and lazy evaluation. 3.2 Regex In Dynamic Symbolic Execution Greedy operators consume as many characters as possible The code in Listing 1 parses numeric arguments between such that the entire regular expression still matches; lazy op- XML tags from its input variable args, an array of strings. erators consume as few characters as possible. This distinction— The regex in line 4 breaks each argument into two capture called matching precedence—is unnecessary for classical reg- groups, the tag and the numeric value (parts[0] is the en- ular languages, but does affect the assignment of capture tire match). When the tag is “timeout”, it sets the timeout groups and therefore backreferences. value accordingly (lines 6–7). On line 12, a runtime asser- Zero-length assertions or lookarounds do not consume any tion checks that the timeout value is truly numeric after the characters but still restrict the accepted word, enforcing a arguments have been processed. The assertion can fail be- language intersection. Positive or negative lookaheads can cause the program contains a bug: the regex in line 4 uses contain any regex, including capture groups and backrefer- a Kleene star and therefore also admits the empty string as ences. In ES6, lookbehind is only available through \b (word the number to set, and JavaScript’s dynamic type system boundary), and \B (non-word boundary), which are com- will allow setting timeout to the empty string. monly used to only (or never) match whole words in a string. DSE finds such bugs by systematically enumerating paths, including the failure branches of assertions [17]. Starting 3 Overview from a concrete run with input, say, args[0] = "foo", the In an overview of our approach, we now define the word DSE engine will attempt to build a path condition that en- problem for regex (§3.1) and how it arises in DSE (§3.2). We codes the branching decisions in terms of the input values. It 3 PLDI’19,June22–26,2019,Phoenix,AZ,USA BlakeLoring,Duncan Mitchell, and Johannes Kinder then attempts to systematically flip clauses in the path con- capturing language constraint dition and query an SMT solver to obtain input assignments = covering different paths. This process repeats forever or un- (w2,C1) ∈ Lc ((?:a|(b))\1) ⇒ = ′ ′ ′ ′ til all paths are covered (this program has an unbounded w2 w1++ w2 ∧(w1, C1) ∈ Lc (a|(b))∧(w2) ∈ Lc (\1). number of paths as it is looping over an input string). Without support for regex, the DSE engine will concretize When treating the alternation, either the left is satisfied and arg on the call to exec, assigning the concrete result to the capture group becomes undefined (which we denote as parts. With all subsequent decisions therefore concrete, the ), or the right is satisfied and the capture is locked to the path condition becomes pc = true and the engine will be match, which we model as unable to cover more paths and find the bug. ′ = ′ = ′ (w1 ∈ L(a)∧C1 )∨(w1 ∈ L(b)∧C1 w1). Implementing regex support ensures that parts is sym- bolic, i.e., its elements are represented as formulas during Finally we model the backreference, which is case depen- symbolic execution. The path condition for the initial path dent on whether the capture group it refers to is defined or not: thus becomes pc = (args[0], C0, C1, C2) < Lc (R) where R = <(\w+)>([0-9]*)<\/\1>. Negating the only clause = = ′ = = ′ = (C1 ⇒ w2 ϵ)∧(C1 , ⇒ w2 C1). and solving yields, e.g., args[0] = "0". DSE then uses this input assignment to cover a second path with pc = Putting this together, we obtain a model for R: (args[0], C0, C1, C2) ∈ Lc (R)∧C1 , "timeout". Negating ′ ′ (w, C0, C1) ∈ Lc (R) =⇒ w = w1 ++ w ++ w ++ w3 the last clause yields, e.g., “
4 Modeling ES6 Regex this direct correspondence can be expressed as
We now detail the process of modeling capturing languages. (w,C0, C1,..., CK ) ∈ Lc (S{m, n}) ⇐⇒ ′ After preprocessing a given ES6 regex R to R (§4.1), we n m (w, C0, C1,0,0,..., CK,n−m,n) ∈ Lc (S | ... | S ) model constraints (w, C ,..., C ) ∈ L (R′) by recursively 0 n c ∃ translating terms in the abstract syntax tree (AST) of R′ ∧ x ∈ {0,..., n − m} : m+x to classical regular language membership and string con- (w, C0, C1,x,0,..., CK,x,m+x−1) ∈ Lc (S ) straints (§4.2–4.3). Finally, we show how to model negated ∀ ′ m+x ′ ′ ∧ x > x, (w, C0, C1,x ′,0,..., CK,x ′,m+x ′−1) < Lc (S ) constraints (w, C0,..., Cn) < Lc (R ) (§4.4). ∧ ∀i ∈ {1,..., K}, Ci = Ci,x,m+x−1 . 4.1 Preprocessing
For illustrative purposes, we make the concatenation R1R2 4.2 Operators and Capture Groups of terms R1, R2 explicit as the binary operator R1 · R2. Any Let t be the next term to process in the AST of R′. If t is regex can then be split into combinations of atomic elements, capture-free and purely regular, there is nothing to do in capture groups and backreferences (referred to collectively this step. If t is non-regular, it contains k + 1 capture groups as terms, in line with the ES6 specification [14]), joined by (with k ≥ −1) numbered i through i + k. At each recur- explicit operators. Using the rules in Table 1, we rewrite sive step, we express membership of the capturing language ′ any R to an equivalent regex R containing only alternation, (w, Ci,..., Ci+k ) ∈ Lc (t) through a model consisting of string concatenation, Kleene star, capture groups, non-capturing and regular language membership constraints, and a set of parentheses, lookarounds, and backreferences. We rewrite remaining capturing language membership constraints for any remaining lazy quantifiers to their greedy equivalents, subterms of t. Note that we record the locations of capture as the models are agnostic to matching precedence (this is groups within the regex in the preprocessing step. When dealt with in refinement). splitting t into subterms t1 andt2, capture groups Ci ,..., Ci+j Note that the rules for Kleene plus and repetition dupli- are contained in t1 and Ci+j+1,..., Ci+k are contained in t2 cate capture groups, e.g., rewriting/(a){1,2}/ to/(a)(a)|(a)/ for some j. The models for individual operations are given adds two capture groups. We therefore explicitly relate cap- in Table 2; we discuss specifics of the rules below. ture groups between the original and rewritten expressions. When matching an alternation |, capture groups on the + When rewriting a Kleene plus expression S containing K non-matching side will be undefined, denoted by , which capture groups, S*S has 2K capture groups. For a constraint is distinct from the empty string ϵ. of the form (C ,..., C ) ∈ L (S+), the rewriting yields 1 K c When modeling quantification t = t1∗, we assume t1 does not contain backreferences (we address this case in §4.3). In ˆ (C , C ,..., C , C ,..., C ) ∈ L (S*S). this instance, we model t via the expression t1*t1|ϵ, where 0 1,1 K,1 1,2 K,2 c ˆ t1 is a regular expression corresponding to t1, except each set of capturing parentheses is rewritten as a set of non- Since S*S contains two copies of S, Ci, j corresponds to the ˆ capturing parentheses. In this way, t1 is regular (it is backreference- ith capture in the jth copy of S in S*S. We express the direct ˆ free by assumption). However, t1*t1|ϵ is not semantically correspondence between captures as equivalent to t: if possible, capturing groups must be sat- ˆ isfied, so t1* cannot consume all matches of the expression. ˆ (w, C0, C1,..., CK ) ∈ Lc (S+) ⇐⇒ We encode this constraint with the implication that t1* must match the empty string whenever t |ϵ does. (w, C0, C1,1,..., CK,1, C1,2,..., CK,2) ∈ Lc (S*S) 1 Lookahead constrains the word to be a member of the ∧∀i ∈ {1,..., K}, Ci = Ci,2. languages of both the assertion expression and t2. The word boundary \b is effectively a single-character lookaround for For repetition, if S{m, n} has K capture groups, then S ′ = word and non-word characters. Because the boundary can n ... m K + + ′ S | | S has 2 (n m)(n − m 1) captures. In S , sup- occur both ways, the model uses disjunction for the end of pose we index our captures as Ci, j,k where i ∈ {1,..., K} w1 and the start of w2 being word and non-word, or non- is the index of the capture group in S, j ∈ {0,..., n − m} word and word characters, respectively. The non-word bound- denotes which alternate the capture group is in (0 being the ary \B is defined as the dual of \b. rightmost), and k ∈ {0,...,m + j − 1} indexes the copies For capture groups, we bind the next capture variable Ci th of S within each alternate. Intuitively, we pick a single x ∈ to the string matched by t1. The i capture group must be {0,..., n − m} that corresponds to the first satisfied alter- the outer capture and the remaining captures Ci+1,..., Ci+k nate. Comparing the assignment of captures in S{m, n} to must therefore be contained within t1. There is nothing to S ′, we know that the value of the capture is the last possible be done for non-capturing groups and recursion continues match, so Ci = Ci,x,m+x−1 for all i ∈ {1,..., K}. Formally, on the contained subexpression. 5 PLDI’19,June22–26,2019,Phoenix,AZ,USA BlakeLoring,Duncan Mitchell, and Johannes Kinder
Table 2. Models for regex operators.
Operation t Overapproximate Model for (w, Ci ,..., Ci+k )∈Lc (t)
(w, Ci ,..., Ci+j )∈Lc (t1) ∧ Ci+j+1 = ... = Ci+k = Alternation t1|t2 ∨ (w, Ci+j+1, ..., Ci+k )∈Lc (t2) ∧ Ci = ... = Ci+j = Concatenation t1 · t2 w = w1 ++ w2 ∧(w1, Ci , ..., Ci+j )∈Lc (t1)∧(w2, Ci+j+1, ..., Ci+k )∈Lc (t2) = ˆ Backreference-free w w1 ++ w2 ∧ w1 ∈ L(t1*)∧(w2, Ci ,..., Ci+k )∈Lc (t1|ϵ) t1* Quantification ∧ w2 = ϵ =⇒ (w1 = ϵ ∧ Ci = ... = Ci+k = ) Positive Lookahead (?=t1)t2 (w, Ci ,..., Ci+j )∈Lc (t1.*)∧(w, Ci+j+1,..., Ci+k )∈Lc (t2)
Negative Lookahead (!=t1)t2 (w, Ci ,..., Ci+j ) < Lc (t1.*)∧(w, Ci+j+1,..., Ci+k )∈Lc (t2)
Input Start t1^ (w, Ci ,..., Ci+k )∈Lc (t1)∧(w, Ci , ..., Ci+k )∈ L(. ∗ h)
Input Start (Multiline) t1^ (w, Ci ,..., Ci+k )∈Lc (t1)∧(w, Ci , ..., Ci+k )∈ L(. ∗ h|\n)
Input End $t1 (w, Ci ,..., Ci+k )∈Lc (t1)∧(w, Ci , ..., Ci+k ) ∈ L(i.∗)
Input End (Multiline) $t1 (w, Ci ,..., Ci+k )∈Lc (t1)∧(w, Ci , ..., Ci+k ) ∈ L(i|\n.∗)
w = w1 ++ w2 ∧(w1, Ci , ..., Ci+j )∈Lc (t1)∧(w2, Ci+j+1, ..., Ci+k )∈Lc (t2) Word Boundary t1\b t2 ∧ (w1 ∈ L(.*\W)∨ w1 = ϵ)∧ w2 ∈ L(\w.*) ∨ w1 ∈ L(.*\w)∧(w2 ∈ L(\W.*)∨ w2 = ϵ) w = w1 ++ w2 ∧(w1, Ci , ..., Ci+j )∈Lc (t1)∧(w2, Ci+j+1, ..., Ci+k )∈Lc (t2) Non-Word Boundary t1\B t2 ∧ (w1 < L(.*\W)∧ w1 , ϵ)∨ w2 < L(\w.*) ∧ w1 < L(.*\w)∨(w2 < L(\W.*)∧ w2 , ϵ) = Capture Group (t1) (w, Ci+1, ..., Ci+k )∈Lc (t1) ∧ Ci w Non-Capturing (?:t ) (w, Ci ,..., C + )∈Lc (t1) Group 1 i k Base Case t regular w ∈ L(t)
Anchors assert the start (^) and end ($) of input; we repre- the backreference \1 and the second instance of \2 are im- sent the beginning and end of a word via the meta-characters mutable. However, the first instance of \2 is mutable: each h and i, respectively. In most instances when handling these repetition of the outer capture group under the Kleene plus operations, t1 will be ϵ; this is because it is rare to have regex can change the value of the second (inner) capture group, operators prior to those marking the start of input (or after in turn changing the value of the backreference inside this marking the end of input, respectively). In both these cases, quantification. For example, the string "aabbbbb" satisfies we assert that the language defines the start or end of input— this regex, but "aababaa" does not. To fully characterize and that as a result of this, the language of t1 must be an these distinctions, we introduce the following definition: empty word, though the capture groups may be defined (say through t1 containing assertions with nested captures). We Definition 2 (Backreference Type). Let t be the kth capture give separate rules for matching a regular expression with group of a regex R. Then the multiline flag set, which modify the behavior of anchors to accept either our meta-characters or a line break. 1. \k is empty if either k is greater than the number of capture groups in R, or \k is encountered before t in 4.3 Backreferences a post-order traversal of the AST of R; Table 3 lists our models for different cases of backreferences 2. \k is mutable if \k is not empty, and both t and \k are in the AST of regex R; \k is a backreference to the kth capture subterms of some quantified term Q in R; group of R. Intuitively, each instance of a backreference is 3. otherwise, \k is immutable. a variable that refers to a capture group and has a type that depends on the structure of R. When a backreference is empty, it is defined as ϵ, because We call a backreference immutable if it can only eval- it refers to a capture group that either is a superterm, e.g., uate to a single value when matching; it is mutable if it /(a\1)*/, or appears later in the term, e.g., /\1(a)/. can take on multiple values, which is a rare but particularly There are two cases for immutable backreferences. In the tricky case. For example, consider /((a|b)\2)+\1\2/. Here, first case, the backreference is not quantified. In our model 6 SoundRegularExpressionSemanticsforDSEofJavaScript PLDI ’19, June 22–26, 2019, Phoenix, AZ, USA
Table 3. Modeling backreferences.
Type of \k Capturing Language Approximation Model
Empty (w)∈Lc (\k) Exact w = ϵ
Immutable (w)∈Lc (\k) Overapproximate (Ck = =⇒ w = ϵ)∧(Ck , =⇒ w = Ck ) = = = , = ∃ 0 : = m Immutable (w)∈Lc (\k*) Overapproximate (Ck ⇒ w ϵ)∧(Ck ⇒ m ≥ w ++ i=0Ck ) = = ∃ 1: = m (w,Ck )∈Lc ((?:(t1)\k)*) w ϵ ∧ Ck ∨ m ≥ w ++ i=1(σi,1 ++ σi,2) Mutable Overapproximate t1is capture group-free ∧ ∀i > 1, (σi,1, Ck i )∈Lc (t1)∧ σi,2 = Ck i ∧ Ck = Ck m , , , = = ∃ 1: = m (w,Ck )∈Lc ((?:(t1)\k)*) w ϵ ∧ Ck ∨ m ≥ w ++ i=1(σi,1 ++ σi,2) Mutable Unsound t1is capture group-free ∧(σi,1, Ck )∈Lc (t1)∧ ∀i ≥ 1, (σi,1 = σ1,1 ∧ σi,2 = σ1,1) for R, Ck has already been modeled with an equality con- 4.4 Modeling Non-Membership straint, so we can bind the backreference to it. In the sec- The model described so far overapproximates membership ond case, the backreference occurs within a quantification; of a capturing language. We define an analogous model for here, the matched word is a finite concatenation of identical non-membershipof the form ∀C0,..., Cn : (w, C0,..., Cn) < copies of the referenced capture group. Both models also in- Lc (R). Intuitively, non-membership models assert that for corporate the corner case where the capture group is due all capture group assignments there exists some partition of to alternation or an empty Kleene star. Following the ES6 the word such that one of the individual constraints is vio- standard, the backreference evaluates to ϵ in this case. lated. Most models are simply negated. In concatenation and Mutable backreferences appear in the form (...t1...\k...)* quantification, only language and emptiness constraints are th where t1 is the k capture group; ES6 does not support for- negated, so the models take the form ward referencing of backreferences, so in (...\k...t ...)*, \k 1 = is empty. For illustration purposes, the fourth entry of Ta- w w1 ++ w2 ble 3 describes the simplest case for mutable backreferences, ∧ ... < Lc (...) ∨ ... < Lc (...) other patterns are straightforward generalizations. In this = = th ∨(w2 ϵ ∧ ¬(w1 ϵ ...)) . case, we assume t1 is the k capture group but is otherwise capture group-free. We can treat the entirety of this term In the same manner, the model for capture groups is at once: as such, any word in the language is either ϵ, or (w, C + ,..., C + ) < L (t )∧C = w. for some number of iterations, we have the concatenation i 1 i k c 1 i of a word in the language of t1 followed by a copy of it. We Returning to the example of §3.3, the negated model for introduce new variables Ck,i referring to the values of the ∀C0, C1 : (w, C0, C1) < Lc ((?:a|(b))\1) becomes capture group in each iteration, which encodes the repeated ∀C , C : w = w ++ w ′ ++ w ′ ++ w matching on the string until settling on the final value for 0 1 1 1 2 3 = ′ ++ ′ Ck . In this instance, we need not deal with the possibility ∧C0 w1 w2 that any C is , since the quantification ends as soon as ′ = ′ = ′ k,i ∧ ¬ (w1 ∈ L(a)∧C1 )∨(w1 ∈ L(b)∧C1 w1) t does not match. 1 ∨ ¬(C = =⇒ w ′ = ϵ)∨¬(C , =⇒ w ′ = C ) Unfortunately, constraints generated from this model are 1 2 1 2 1 hard to solve and not feasible for current SMT solvers, be- ∨ w1 < L((:?.|\n)*?) ∨ w3 < L((:?.|\n)*?) . cause they require “guessing” a partition of the matched string variable into individual and varying components. To 5 Matching Precedence Refinement make solving such queries practical, we introduce an alter- We now explain the issue of matching precedence (§5.1) and native to the previous rule where we treat quantified back- introduce a counterexample-guided abstraction refinement references as immutable. The resulting model is shown in scheme (§5.2) to address it. We discuss termination (§5.3) the last row of Table 3. E.g., returning to /((a|b)\2)+\1\2/, and the overall soundness of our approach (§5.4). we accept ("aaaaaaaaa", "aaaaaaaaa", "aaaa", "a"), but not ("aabbaabbb", "aabbaabbb", "aabb", "b"). We discuss the 5.1 Matching Precedence soundness implications in §5.4. Quantified backreferences The model in Tables 2 and 3 does not account for match- are rare (see §7.1), so the effect is limited in practice. ing precedence (see §3.4). A standards-compliant ES6 regex matcher will derive a unique set of capture group assign- ments when matching a string w, because matching prece- dence dictates that greedy (non-greedy) expressions match 7 PLDI’19,June22–26,2019,Phoenix,AZ,USA BlakeLoring,Duncan Mitchell, and Johannes Kinder
Algorithm 1: Counterexample-guided abstraction re- constraints. We number the original capturing language con- 0 finement scheme for matching precedence. straints ≤ j < m so that we can refer to them as (wj , C0, j , ..., ⊡ ⊡ , < Input : Constraint problem P including models for m Cnj, j ) j Lc (Rj ), where ∈ {∈ }. The algorithm re- ⊡ turns null if P is unsatisfiable, or a satisfying assignment constraints (wj , C0, j ,..., Cnj , j ) j Lc (Rj ). Output:null if P is unsatisfiable, or a satisfying assignment with correct matching precedence. for P otherwise In a loop, we first pass the problem P to an external SMT 1 M := null; solver. The solver returns a satisfying assignment M or null 2 Failed := false; if the problem is unsatisfiable, in which case we are done 3 do (lines 4–6). If M is not null, the algorithm uses a concrete 4 M := Solve(P); regular expression matcher (e.g., Node.js’s built-in matcher) 5 if M = null then ♮ to populate concrete capture variables Ci, j corresponding to 6 return null; the words wj in M. 7 Failed := false; Lines 8–22 describe how the assignments of capture groups 0 1 8 for j := to m − do are checked for each regular expression Rj in the original ♮ ♮ 9 (C ,...,C ) := ConcreteMatch(M[w ], R ) ; 0, j nj, j j j problem P. We first check whether the concrete matcher re- ♮ ♮ 10 if (C ,...,C ) then 0, j nj , j turned a list of valid capture group assignments, i.e., whether 11 if ⊡j = ∈ then the word M[wj ] from the satisfying assignment matches con- 12 for i := 0 to nj do cretely. If it did, then wj is a member of the language gen- ♮ , 13 if Ci, j M[Ci, j ] then erated by Rj . If ⊡j = ∈, i.e., the membership constraint was 14 Failed := true; positive, then we must check if the capture group assign- 15 P := P ∧(wj = M[wj ] =⇒ ments are consistent with those from M (line 13). If they = ♮ 0≤i ≤n Ci, j C ) ; are, we move on to the next regex, otherwise we refine the Ó j i, j constraint problem by fixing capture group assignments to 16 else // Non-membership query their concrete values for the matched word (line 15). Dually, 17 Failed := true; if a modeled non-membership constraint was satisfiable but := , 18 P P ∧(wj M[wj ]); the word from the current satisfying assignment M[wj ] did match concretely, we refine the problem by asserting that w 19 else // No concrete match [ ] 20 if ⊡j = ∈ then must not equal that word (line 18). We do the same if M wj 21 Failed := true; did not match concretely but came from a satisfied positive 22 P := P ∧(wj , M[wj ]); membership constraint (line 22). If no refinement was necessary we have confirmed the
23 while Failed; overall assignment satisfies P and return M (line 24). Other- 24 return M; wise, the loop continues with solving the refined problem.
5.3 Termination Unsurprisingly, CEGAR may require arbitrarily many refine- as many (as few) characters as possible before moving on to ments on pathological formulas and never terminate. This is the next [14]. These requirements are not part of our model, unavoidable due to undecidability [7]. In practice, we there- as encoding them directly into SMT would require nesting fore impose a limit on the number of refinements, leading to of quantifiers for each operator, making them impractical unknown as a possible third result. SMT solvers already may for automated solving. timeout or report unknown for complex string formulas, so this does not lead to additional problems in practice. 5.2 CEGAR for ES6 Regular Expression Models We eliminate infeasible elements of the capturing language 5.4 Soundness admitted by our model through counter example-guided ab- When constructing the rules in Tables 2 and 3, we followed straction refinement (CEGAR). the semantics of regular expressions as laid out in the ES6 Algorithm 1 is a CEGAR-based satisfiability checker for standards document [14]. The ES6 standard is written in a constraints modeled from ES6 regexes, which relies on an semi-formal fashion, so we are confident that our transla- external SMT solver with classical regular expression and tion into logic is accurate, but cannot have formal proof. Ex- string support and an ES6-compliant regex matcher. The isting attempts to encode ECMAScript semantics into logic algorithm takes an SMT problem P (derived from the DSE such as JSIL [8]orKJS[28] do not include regexes. path condition) as a conjunction of constraints, some of which With the exception of the optimized rule for mutable back- model the m ≥ 0 original capturing language membership references, our models are overapproximate, because they 8 SoundRegularExpressionSemanticsforDSEofJavaScript PLDI ’19, June 22–26, 2019, Phoenix, AZ, USA
Algorithm 2: RegExp.exec(input) Algorithm 2 first processes flags to begin from the end of ′ the previous match for sticky or global flags, and it rewrites 1 input := ‘h’ + input + ‘i’; the regex to accept lower and upper case variants of charac- 2 if sticky or global then ters for the ignore case flag. 3 offset := lastIndex > 0 ? lastIndex +1:0; ′ ′ We introduce the h and i meta-characters to input which 4 input := input .substring(offset); ′ actas markersfor the start and end ofa string during match- 5 source := ‘(:?.|\n)*?(’ + source + ‘)(:?.|\n)*?’; ing. Next, if the sticky or global flags are set we slice in- 6 if caseIgnore then ′ ′ put at lastIndex so that the new match begins from the 7 source := rewriteForIgnoreCase(source ); end of the previous. Due to the introduction of our meta- ′ ′ 8 if (input , C0, ..., Cn )∈Lc (source ) then lastIndex ′ characters needs to be offset by 1 if it is greater 9 Remove h and i from (input , C0,..., Cn ); than zero. We then rewrite the regex source to allow for 10 lastIndex := lastIndex + C0.startIndex + C0.length; characters to precede and succeed the match. Note that we , ..., 11 result := [C0 Cn ]; use (?:.|\n)*? rather than .*? because the wildcard . con- 12 result.input := input; sumes all characters except line breaks in ECMAScript regexes. 13 result.index := C0.startIndex; To avoid adding these characters to the final match we place 14 return result; the original regex source inside a capture group. This forms 15 else C0, which is defined to be the whole matched string [14]. 16 lastIndex := 0; Once preprocessing is complete we test whether the input 17 return undefined; string and fresh string for each capture group are within the capturing language for the expression. If they are then a re- sults object is created which returns the correctly mapped capture groups, the input string, and the start of the match ignore matching precedence. When the CEGAR loop termi- in the string with the meta-characters removed. Otherwise nates, any spurious solutions from overapproximation are lastIndex is reset and undefined is returned. eliminated. As a result, we have an exact procedure to decide (non)-membership for capturing languages of ES6 regexes 6.2 ExpoSE without quantified backreferences. In the presence of quantified backreferences, the model ExpoSE is a DSE engine which uses the Jalangi2 [19] frame- after CEGAR termination becomes underapproximate. Since work to instrument a piece of JavaScript software in order to DSE itself is an underapproximate program analysis (due to create a program trace. As the program terminates, ExpoSE concretization, solver timeouts, and partial exploration), our calls the SMT solver Z3 [13] to identify all feasible alter- model and refinement strategy are sound for DSE. nate test-cases from the trace. These new test cases are then queued and the next test case is selected for execution, in the 6 Implementation manner of generational search [18]. The ExpoSE framework allows for the parallel execution of individual test cases, ag- We now describe an implementation of our approach in the gregating coverage and alternative path information as each DSE engine ExpoSE1 [27]. We explain how to model the test case terminates. This parallelization is achieved by exe- regex API with capturing language membership (§6.1) and cuting each test case as a unique process allocated to a ded- give a brief overview of ExpoSE (§6.2). icated single core; as such the analysis is highly scalable. Our strategy for test case selection is similar to the CUPA 6.1 Modeling the Regex API strategy proposed by Bucur et al. [9]. We use program fork The ES6 standard specifies several methods that evaluate points to prioritize unexplored code: each expression is given regexes [14]. We follow its specified pseudocode for RegExp a unique identifier and scheduled test cases are sorted into .exec(s) to implement matching and capture group assign- buckets based upon which expression was being executed ment in terms of capturing language membership in Algorithm 2. when they were created. We select the next test case by Notably, our algorithm implements support for all flags and choosing a random test case from the bucket that has been operators specified for ES6. accessed least during the analysis; this prioritizes test cases RegExp.test(s) is precisely equivalent to the expression triggered by less common expressions. RegExp.exec(s) !== undefined. In the same manner, one can construct models for other regex functions defined for 7 Evaluation ES6. Our implementation includes partial models for the re- maining functions that allow effective test generation in prac- We now empirically answer the following research ques- tice but are not semantically complete. tions: (RQ1) Are non-classical regexes an important problem in 1ExpoSE is available at h ps://github.com/ExpoSEJS/ExpoSE. JavaScript? 9 PLDI’19,June22–26,2019,Phoenix,AZ,USA BlakeLoring,Duncan Mitchell, and Johannes Kinder
Table 4. Regex usage by NPM package. Table 5. Feature usage by unique regex.
Feature Count % Feature Total % Unique % Packages on NPM 415,487 100.0% Total Regex 9,552,546 100% 305,691 100% ... with source files 381,730 91.9% Capture Groups 2,360,178 24.71% 119,051 38.94% ... with regular expressions 145,100 34.9% Global Flag 2,620,755 27.44% 90,356 29.56% ... with capture groups 84,972 20.5% Character Class 2,671,565 27.97% 71,040 23.24% ... with backreferences 15,968 3.8% Kleene+ 1,541,336 16.14% 67,508 22.08% ... with quantified backreferences 503 0.1% Kleene* 1,713,713 17.94% 66,526 21.76% Ignore Case Flag 1,364,526 14.28% 58,831 19.25% Ranges 1,273,726 13.33% 52,155 17.06% (RQ2) Does accurate modeling of ES6 regexes make DSE- Non-capturing 1,236,533 12.94% 25,946 8.49% based test generation more effective? Repetition 360,578 3.7% 17,068 5.58% (RQ3) Does the performance of the model and the refine- Kleene* (Lazy) 230,060 2.41% 13,250 4.33% ment strategy enable practical analysis? Multiline Flag 137,366 1.44% 10,604 3.47% Word Boundary 336,821 3.53% 9,677 3.17% We answer the first question with a survey of regex usage Kleene+ (Lazy) 148,604 1.56% 6,072 1.99% in the wild (§7.1). We address RQ2 by comparing our ap- Lookaheads 176,786 1.85% 3,123 1.02% proach against an existing partial implementation of regex Backreferences 64,408 0.67% 2,437 0.80% support in ExpoSE [27] on a set of widely used libraries Repetition 2,412 0.03% 221 0.07% (§7.2). We then measure the contribution of each aspect of (Lazy) our approach on over 1,000 JavaScript packages (§7.3). We Quantified 1,346 0.01% 109 0.04% answer RQ3 by analyzing solver and refinement statistics BRefs per query (§7.4). Sticky Flag 98 <0.01% 60 0.02% Unicode Flag 73 <0.01% 48 0.02% 7.1 Surveying Regex Usage We focus on code written for Node.js, a popular framework for standalone JavaScript. Node.js is used for both server and desktop applications, including popular tools Slack and Skype. We analyzed 415,487 packages from the NPM reposi- common problems, and the inclusion of dependencies (fore- tory, the primary software repository for open source Node.js going proper dependency management). To adjust for this, code. Nearly 35% of NPM packagescontain a regex, 20% con- we provide data for both all expressions encountered and tain a capture group and 4% contain a backreference. for just unique expressions. In both cases, there are signifi- cant numbers of capture groups, backreferences, and other Methodology We developed a lightweight static analysis non-classical features. As the occurrence rate of quantified that parses all source files in a package and identifies regex backreferences is low, we do not differentiate between mu- literals and function calls. We do not detect expressions of table and immutable backreferences. the form new RegExp(...), as they would generally require a more expensive static analysis. Our numbers therefore pro- Conclusions Our findings confirm that regexes are widely vide a lower bound for regex usage. used and often contain complex features. Of particular im- portance is a faithful treatment of capture groups, which Results We found regex usage in JavaScript to be wide- appear in 20.45% of the packages examined. On the flip side, spread, with 145,100 packages containing at least one regex since quantified backreferences make up just 0.01% of regexes, out of a total 415,487 scanned packages. Table 4 lists the the optimization introduced in §4.3 will rarely lead to addi- number of NPM packages containing regexes, capture groups, tional underapproximation during DSE. backreferences, and backreferences appearing within quan- tification. Note that a significant number of packages make use of capture groups and backreferences, confirming the 7.2 Improvement Over State of the Art importance of supporting them. We compare our approach against the original ExpoSE [27], Table 5 reports statistics for all 9M regexes collected, giv- which is, to our knowledge, the only available and func- ing for each feature the fraction of expressions including it. tional implementation of regex support in JavaScript. Many regexes in NPM packages are not unique; this appears to be due to repeated inclusion of the same literal (instead Methodology We evaluated statement coverage achieved of introduction of a constant), the use of online solutions to by both versions of ExpoSE on a set of libraries, which we 10 SoundRegularExpressionSemanticsforDSEofJavaScript PLDI ’19, June 22–26, 2019, Phoenix, AZ, USA
Table 6. Statement coverage with our approach (New) vs. regex queries presented in this paper has a substantial im- [27] (Old) and the relative increase (+) on popular NPM pact on practical problems in DSE. We also see that other im- packages (Weekly downloads). LOC are lines loaded and provements to ExpoSE, such as ES6 support, have affected RegEx are regular expression functions symbolically exe- coverage. Therefore, we continue with an evaluation of the cuted. individual aspects of our model.
Library Weekly LOC RegEx Old(%) New(%) +(%) 7.3 Breakdown of Contributions babel-eslint 2,500k 23,047 902 21.0 26.8 27.6 We now drill down into how the individual improvements fast-xml-parser 20k 706 562 3.1 44.6 1,338.7 in regex support are contributing to increases in coverage. js-yaml 8,000k 6,768 78 4.4 23.7 438.6 Methodology From the packages with regexes from our minimist 20,000k 229 72,530 65.9 66.4 0.8 survey §7.1, we developed a test suite of 1,131 NPM libraries moment 4,500k 2,572 21 0.0 52.6 ∞ query-string 3,000k 303 50 0.0 42.6 ∞ for which ExpoSE is able to automatically generate a mean- semver 1,800k 757 616 51.7 46.2 −10.6 ingful test harness. In each of the libraries selected, ExpoSE url-parse 1,400k 322 448 60.9 71.8 17.9 executed at least one regex operation on a symbolic string, validator 1,400k 2,155 94 67.5 72.2 7.0 which ensures that the library contains some behavior rele- xml 500k 276 1,022 60.2 77.5 28.7 vant to the scope of this paper. The test suite constructed in yn 700k 157 260 0.0 54.0 ∞ this manner contains numerous libraries that are dependen- cies of packages widely used in industry, including Express and Lodash.2 Automatic test generation typically requires a bespoke chose for their popularity (with up to 20M weekly down- test harness or set of parameterized unit tests [33] to achieve loads) and use of regex. This includes the three libraries min- high coverage in codethat does not have a simple command imist, semver, and validator, which the first version of Ex- line interface, including libraries. ExpoSE’s harness explores poSE was evaluated on [27]. To fairly compare original Ex- libraries fully automatically by executing all exported meth- string poSE against our extension, we use the original automated ods with symbolic arguments for the supported types library harness for both. Therefore we do not take advan- , boolean, number, null and undefined. Returned objects tage of other improvements for test generation, such as sym- or functions are also subsequently explored in the same man- bolic array support, which we have added in the course of ner. our work. We re-executed each package six times for one We executed each package for one hour, which typically hour each on both versions, using 32-core machines with allowed to reach a (potentially initial) coverage plateau, at 256GB of RAM, and averaged the results. We limited the which additional test cases do not increase coverage further. refinement scheme to 20 iterations, which we identified as We break down our regex support into four levels and mea- effective in preliminary testing (see §7.4). sure the contribution and cost of each one to line coverage and test execution rate (Table 7). As baseline, we first ex- Results Table 6 contains the results of our comparison. To ecute all regex methods concretely, concretizing the argu- provide an indication of program size, we use the number of ments and results. In the second configuration, we add the lines of code loaded at runtime (JavaScript’s dynamic method model for ES6 regex and their methods, including support of loading dependencies makes it hard to determine a mean- for word boundaries and lookaheads, but remove capture ingful LOC count statically). groups and concretize any accesses to them, including back- The results demonstrate that ExpoSE extended with our references. Third, we also enable full support for capture model and refinement strategy can improve coverage more groups and backreferences. Fourth, we finally also add the than tenfold on our sample of widely-used libraries. In the refinement scheme to address overapproximation. cases of moment, query-string, and yn, the lack of ES6 sup- Results Table 7 shows, for each level of support, the num- port in the original ExpoSE prohibited meaningful analysis, ber and percentage of target packages where coverage im- leading to 0% coverage. In the case of semver, we see a de- proved; the geometric mean of the relative increase in cover- crease in coverage if stopped after one hour. This is due to age; and the mean test execution rate. The final row shows the modeling of regex increasing solving time (see also §7.4). the effect of enabling full support compared to the baseline. The coverage deficit disappears when executing both ver- Note that the number of packages improved is less than the sions of ExpoSE with a timeout of two hours. sum of the rows above, since the coverage of a package can be improved by multiple features. Conclusions We find that our modifications to ExpoSE make test generation more effective in widely used libraries 2Raw data for the experiments, including all package names, is available at using regex. This suggests that the new method of solving h ps://github.com/ExpoSEJS/PLDI19-Raw-Data. 11 PLDI’19,June22–26,2019,Phoenix,AZ,USA BlakeLoring,Duncan Mitchell, and Johannes Kinder
Table 7. Breakdown of how different components con- Table 8. Solver times per package and query. tribute to testing 1, 131 NPM packages, showing number (#) and fraction (%) of packages with coverage improvements, Constraint Solver Time the geometric mean of the relative coverage increase from Packages/Queries Minimum Maximum Mean the feature (Cov), and test execution rate. Allpackages 0.04s 12h15m 2h34m Withcapturegroups 0.20s 12h15m 2h40m Improved Cov Tests With refinement 0.46s 12h 15m 2h 48m Regex Support Level #% +(%) min Where refinement limit is hit 3.49s 11h 07m 3h 17m Concrete Regular Expres- - - - 11.46 Allqueries 0.001s 22m26s 0.15s sions Withcapturegroups 0.001s 22m26s 5.53s + Modeling RegEx 528 46.68% +6.16% 10.14 With refinement 0.005s 18m 51s 22.69s + Captures & Backreferences 194 17.15% +4.18% 9.42 Where refinement limit is hit 0.120s 18m 51s 58.85s + Refinement 63 5.57% +4.17% 8.70 All Features vs. Concrete 617 54.55% +6.74%
Methodology We collected data on queries during the NPM In a dataset of this size that includes many libraries that experiments (§7.3) to provide details on SMT query success make only little use of regex, average coverage increases are rates and execution times, as well as on the usage of the expected to be small. Nevertheless, we see that dedicated refinement scheme. support improves the coverage of more than half of pack- ages that symbolically executed at least one regex function. Results We found that 753 (66%) of the 1,131 packages As expected, the biggest improvement comes from support- tested executed at least one query containing a capture group ing basic symbolic execution of regular expressions, even or backreference. Of these packages, 653 (58% overall) con- without capture groups or regard for matching precedence. tained at least one query to the SMT solver requiring re- However, we see further improvements when adding cap- finement, and 134 (12%) contained a query that reached the ture groups, which shows that they indeed affect program refinement limit. semantics. Refinement affects fewer packages, although it In total, our experiments executed 58,390,184 SMT queries significantly contributes to coverage where it is required. to generate test cases. As expected, the majority do not in- This is because a lucky solver may generate correct inputs volve regexes, but they form a significant part: 4,489,581 on the first attempt, even in ambiguous settings. (7.6%) queries modeled a regex, 645,295 (1.1%) modeled a On some libraries in the dataset, the approach is highly capture group or backreference, 74,076 (0.1%) required use effective. For example, in the manifest parser n4mf-parser, of the refinement scheme and 2,079 (0.003%) hit the refine- full support improves coverage by 29% over concrete; in the ment limit. The refinement scheme was overwhelmingly ef- 2 8% format conversion library sbxml2json, by 14%; and in the fective: only . of queries with at least one refinement 0 003% browser detection library mario, by 16%. In each of these also reached the refinement limit ( . of all queries where packages the refinement scheme contributed to the improve- a capture group was modeled). Of the refined SMT queries, ment in coverage. In general, the largest increases are seen the mean number of refinements required to produce a valid in packages that include regular expression-based parsers. satisfying assignment was 2.9; the majority of queries re- Each additional feature causes a small decrease in aver- quired only a single refinement. age test execution rate. Although a small fraction (∼1%) of Table 8 details time spent processing SMT problems per- queries can take longer than 300s to solve, concurrent test package and per-query. We provide the data over the four execution prevents DSE from stalling on a single query. key aspects of the problem: we report the time spent in the constraint solver both per package and per query in total, Conclusions Full support for ES6 regex improves perfor- as well as the time in the constraint solver for the particu- mance of DSE of JavaScript in practice at a cost of a 16% in- larly challenging parts of our strategy. We found that the crease in execution time (RQ2). An increase in coverage at use of refinements increased the average per-query solving lower execution rate in a fixed time window suggests that time by a factor of four; however, this is dominated by SMT full regular expression support increases the quality of indi- queries that hit the refinement limit, which took ten times vidual test cases. longer to run on average. The low minimum time spent in the solver in some packages can be attributed to packages 7.4 Effectiveness on Real-World Queries where a regular expression was encountered early in execu- We now investigate the performance of the model and re- tion but limitations in the test harness or function models finement scheme to answer RQ3. Finally, we also discuss the (unrelated to regular expressions) prevented further explo- refinement limit and how it affects analysis. ration. 12 SoundRegularExpressionSemanticsforDSEofJavaScript PLDI ’19, June 22–26, 2019, Phoenix, AZ, USA
Conclusions We find the refinement scheme is highly ef- code on NPM is very modular. For instance, among the top fective, as it is able to solve 97.2% of encountered constraint 25 most depended-upon NPM libraries, the largest is 30 KLOC problems containing regexes. It is also necessary, as 10% of (but contains no regex). Several packages selected for our queries containing a capture group had led to a spurious sat- evaluation, such as babel-eslint, had between 20-30 KLOC isfying assignment and required refinement. and were meaningfully explored with the generic harness. Usually, only a small number of refinements are required to produce a correct satisfying assignment. Therefore, even refinement limits of five or fewer are feasible and may im- 8 Related Work prove performance with low impact on coverage. In prior work, we introduced ExpoSE and partial support for encoding JavaScript regex in terms of classical regular lan- 7.5 Threats to Validity guage membership and string constraints [27]. This initial We now look at potential issues affecting the validity of our take on the problem was lacking support for several prob- results, in particular soundness, package selection, and scal- lematic features such as lookaheads, word boundaries, and ability. anchors. Matching precedence was presented as an open problem, which we have now addressed through our refine- Soundness In addition to soundness of the model (see §5.4), ment scheme. one must consider the soundness of the implementation. In In theory, regex engines can be symbolically executed the absence of a mechanized specification for ES6 regex, our themselves through the interpreter [9]. While this removes code cannot be proven correct, so we use an extensive test the need for modeling, in practice the symbolic execution suite for validation. However, assuming the concrete matcher of the entire interpreter and regex engine quickly becomes is specification-compliant, Algorithm 1 will, if it terminates, infeasible due to path explosion. return a specification-compliant model of the constraint for- There have been several other approaches for symbolic mula even if the implementation of §4 contains bugs. In the execution of JavaScript; most include some limited support worst case, the algorithm would not terminate, leading to for classical regular expressions. Li et al. [24] presented an timeouts and loss of coverage. Bugs could therefore only automated test generation scheme for programs with regu- have lowered the reported coverage improvements. lar expressions by on-line generation of a matching function Package Selection and Harness In §7.3, we chose pack- for each regular expression encountered, exacerbating path ages identified in our survey (§7.1) where our generic har- explosion. Saxena et al. [29] proposed the first scheme to ness encountered a regular expression within one hour of encode capture groups through string constraints. Sen et al. DSE. This allowed us to focus the evaluation on regex sup- [31] presented Jalangi, a tool based on program instrumen- port as opposed to evaluating the quality of the harness (and tation and concolic values. Li and Ghosh [23] and Li et al. having to deal with unreachable code in packages). Use of [22] describe a custom browser and symbolic execution en- this harness may have limited package selection to simpler, gine for JavaScript and the browser DOM, and a string con- unrepresentative libraries. However, we found that simple straint solver PASS with support for most JavaScript string APIs do not imply simple code: the final dataset contains operations. Although all of these approaches feature some several complex packages, such as language parsers, and the support for ECMAScript regex (such as limited support for types of regexes encountered were in line with the survey capture groups), they ignore matching precedence and do results. On simple code we found that ExpoSE would often not support backreferences or lookaheads. reach 100% coverage; failure to do so was either due to the Thoméetal. [32] propose a heuristic approach for solving complexity of the code or the lack of support for language constraints involving unsupported string operations. We choose features unrelated to regex and APIs that would require ad- to model operations unsupported by the solver and employ ditional modeling (e.g., the file system). a CEGAR scheme to ensure correctness. Abdulla et al. [2] propose the use of a refinement scheme to solve complex Scalability Scalability is a challenge for DSE in general, constraint problems, including support for context-free lan- and is not specific to our model for regex. Empirically, execu- guages. The language of regular expressions with backref- tion time for a single test (instrumentation, execution, and erences is not context-free [10] and, as such, their scheme constraint generation) grows linearly with program size, as does not suffice for encoding all regexes; however, their ap- does the average size of solver queries. The impact of query proach could serve as richer base theory than classic regular length on solving time varies, but does not appear to be expressions. Scott et al. [30] suggest backreferences can be exacerbated by our regex model. In principle, our model is eliminated via concatenation constraints, however they do compatible with compositional approaches [4, 16] and state not present a method for doing so. merging [5, 21], which can help DSE scale to large programs. Further innovations from the string solving community, The scalability of our approach suffices for Node.js, how- such as work on the decidability of string constraints in- ever: JavaScript has smaller LOC counts than, e.g., C++, and volving complex functions [12, 20] or support for recursive 13 PLDI’19,June22–26,2019,Phoenix,AZ,USA BlakeLoring,Duncan Mitchell, and Johannes Kinder string operations [35, 36], are likely to improve the perfor- [6] Nikolaj Bjørner, Vijay Ganesh, Raphaël Michel, and Margus Veanes. mance of our approach in future. We incorporate our tech- 2012. SMT-LIB Sequences and Regular Expressions. In Int. Workshop niques at the level of the DSE engine rather than the con- on Satisfiability Modulo Theories (SMT). [7] Nikolaj Bjørner, Nikolai Tillmann, and Andrei Voronkov. 2009. Path straint solver, which allows our tool to leverage advances Feasibility Analysis for String-Manipulating Programs. In Tools and in string solving techniques; at the same time, we can take Algorithms for the Construction and Analysis of Systems (TACAS). advantage of the native regular expression matcher and can [8] Martin Bodin, Arthur Chargueraud, Daniele Filaretti, Philippa Gard- avoid having to integrate implementation language-specific ner, Sergio Maffeis, Daiva Naudziuniene, Alan Schmitt, and Gareth details for regular expressions into the solver. Smith. 2014. A Trusted Mechanised JavaScript Specification. In ACM SIGPLAN-SIGACT Symp. Principles of Programming Languages A previous survey of regex usage across 4,000 Python ap- (POPL). plications [11] also provides a strong motivation for model- [9] Stefan Bucur, Johannes Kinder, and George Candea. 2014. Prototyp- ing regex. Our survey extends this work to JavaScript on a ing symbolic execution engines for interpreted languages. In Archi- significantly larger sample size. tectural Support for Programming Languages and Operating Systems (ASPLOS). [10] Cezar Câmpeanu, Kai Salomaa, and Sheng Yu. 2003. A Formal Study of 9 Conclusion Practical Regular Expressions. Int. J. Foundations of Computer Science In this paper we presented a model for the complete regex 14, 06 (2003). language of ES6, which is sound for the dynamic symbolic [11] Carl Chapman and Kathryn T. Stolee. 2016. Exploring Regular Expres- sion Usage and Context in Python. In Int. Symp. on Software Testing execution of the test and exec functions. We model regex and Analysis (ISSTA). membership constraints in terms of string constraints and [12] Taolue Chen, Yan Chen, Matthew Hague, Anthony W. Lin, and Zhilin classical regular language membership. We introduced a novel Wu. 2018. What is decidable about string constraints with the Re- CEGAR scheme to address the challenge of matching prece- placeAll function. PACMPL 2, POPL (2018), 3:1–3:29. dence, whichso farhadbeenlargely ignoredin relatedwork. [13] Leonardo Mendonça de Moura and Nikolaj Bjørner. 2008. Z3: An Ef- ficient SMT Solver. In Tools and Algorithms for the Construction and To the best of our knowledge, ours is the first comprehen- Analysis of Systems (TACAS). sive solution for ES6. We demonstrated that regexes—and [14] ECMA International. 2015. ECMAScript 2015 Language Specification. specifically their non-regular features—are extensively used [15] Xiang Fu, Michael C. Powell, Michael Bantegui, and Chung-Chih Li. in JavaScript and that existing DSE-based analyses would 2013. Simple linear string constraints. Formal Asp. Comput. 25, 6 therefore suffer coverage loss from concretization. In a large (2013). [16] Patrice Godefroid. 2007. Compositional Dynamic Test Generation. In scale evaluation of over 1,000 Node.js programs, our novel ACM SIGPLAN-SIGACT Symp. Principles of Programming Languages solution outperforms existing partial approaches to the prob- (POPL). lem and demonstrates the viability of our model for improv- [17] Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: di- ing the analysis of string-manipulating JavaScript programs. rected automated random testing. In ACM SIGPLAN Conf. Program- ming Language Design and Implementation (PLDI). [18] Patrice Godefroid, Michael Levin, and David Molnar. 2008. Automated Acknowledgments Whitebox Fuzz Testing. In Network and Distributed System Security Blake Loring was supported by the EPSRC Centre for Doc- Symp. (NDSS). toral Training in Cyber Security at Royal Holloway, Univer- [19] Liang Gong, Michael Pradel, Manu Sridharan, and Koushik Sen. 2015. DLint: Dynamically Checking Bad Coding Practices in JavaScript. In sity of London (EP/K035584/1). Int. Symp. on Software Testing and Analysis (ISSTA). [20] Lukás Holík, Petr Janku, Anthony W. Lin, Philipp Rümmer, and Tomás References Vojnar. 2018. String constraints with concatenation and transducers solved efficiently. PACMPL 2, POPL (2018), 4:1–4:32. [1] Parosh Aziz Abdulla, Mohamed Faouzi Atig, Yu-Fang Chen, Lukás [21] Volodymyr Kuznetsov, Johannes Kinder, Stefan Bucur, and George Holík, Ahmed Rezine, Philipp Rümmer, and Jari Stenman. 2015. Norn: Candea. 2012. Efficient state merging in symbolic execution. In ACM An SMT Solver for String Constraints. In Computer Aided Verification SIGPLAN Conf. Programming Language Design and Implementation (CAV). (PLDI). [2] Parosh Aziz Abdulla, Mohamed Faouzi Atig, Yu-Fang Chen, Bui Phi [22] Guodong Li, Esben Andreasen, and Indradeep Ghosh. 2014. SymJS: Diep, Lukáš Holík, Ahmed Rezine, and Philipp Rümmer. 2017. Flat- automatic symbolic testing of JavaScript web applications. In Founda- ten and Conquer: A Framework for Efficient Analysis of String Con- tions of Software Engineering (FSE). straints. In ACM SIGPLAN Conf. Programming Language Design and [23] Guodong Li and Indradeep Ghosh. 2013. PASS: String solving with pa- Implementation (PLDI). rameterized array and interval automaton. In Haifa Verification Conf. [3] Alfred V. Aho. 1990. Algorithms for Finding Patterns in Strings. In (HVC). Handbook of Theoretical Computer Science (Vol. A), Jan van Leeuwen [24] Nuo Li, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram (Ed.). MIT Press, 255–300. Schulte. 2009. Reggae: Automated test generation for programs us- [4] Saswat Anand, Patrice Godefroid, and Nikolai Tillmann. 2008. ing complex regular expressions. In Automated Software Engineering Demand-driven Compositional Symbolic Execution. In 14th Int. Conf. (ASE). Tools and Algorithms for the Construction and Analysis of Systems [25] Tianyi Liang, Andrew Reynolds, Cesare Tinelli, Clark Barrett, and (TACAS). Morgan Deters. 2014. A DPLL(T) Theory Solver for a Theory [5] Thanassis Avgerinos, Alexandre Rebert, Sang Kil Cha, and David of Strings and Regular Expressions. In Computer Aided Verification Brumley. 2014. Enhancing symbolic execution with Veritesting. In (CAV). 36th Int. Conf. Software Engineering (ICSE). 1083–1094. 14 SoundRegularExpressionSemanticsforDSEofJavaScript PLDI ’19, June 22–26, 2019, Phoenix, AZ, USA
[26] Tianyi Liang, Nestan Tsiskaridze, Andrew Reynolds, Cesare Tinelli, [34] Minh-Thai Trinh, Duc-Hiep Chu, and Joxan Jaffar. 2014. S3: A Sym- and Clark Barrett. 2015. A Decision Procedure for Regular Member- bolic String Solver for Vulnerability Detection in Web Applications. ship and Length Constraints over Unbounded Strings. In Int. Symp. In Conf. Computer and Commun. Sec. (CCS). on Frontiers of Combining Systems (FroCoS). [35] Minh-Thai Trinh, Duc-Hiep Chu, and Joxan Jaffar. 2016. Progressive [27] Blake Loring, Duncan Mitchell, and Johannes Kinder. 2017. ExpoSE: Reasoning over Recursively-Defined Strings. In Computer Aided Veri- Practical Symbolic Execution of Standalone JavaScript. In Int. SPIN fication (CAV). Symp. on Model Checking Software (SPIN). [36] Minh-Thai Trinh, Duc-Hiep Chu, and Joxan Jaffar. 2017. Model Count- [28] Daejun Park, Andrei Stefănescu, and Grigore Roşu. 2015. KJS: A Com- ing for Recursively-Defined Strings. In Computer Aided Verification plete Formal Semantics of JavaScript. In ACM SIGPLAN Conf. Pro- (CAV). gramming Language Design and Implementation (PLDI). [37] Margus Veanes, Peli de Halleux, and Nikolai Tillmann. 2010. Rex: [29] Prateek Saxena, Devdatta Akhawe, Steve Hanna, Feng Mao, Stephen Symbolic regular expression explorer. In Software Testing, Verification McCamant, and Dawn Song. 2010. A Symbolic Execution Framework and Validation (ICST). for JavaScript. In IEEE Symp. Sec. and Privacy (S&P). [38] Yunhui Zheng, Vijay Ganesh, Sanu Subramanian, Omer Tripp, Mur- [30] Joseph D. Scott, Pierre Flener, and Justin Pearson. 2015. Constraint phy Berzish, Julian Dolby, and Xiangyu Zhang. 2017. Z3str2: an ef- Solving on Bounded String Variables. In Integration of AI and OR Tech. ficient solver for strings, regular expressions, and length constraints. in Constraint Prog. (CPAIOR). Formal Methods in System Design 50, 2-3 (2017). [31] Koushik Sen, Swaroop Kalasapur, Tasneem Brutch, and Simon Gibbs. [39] Yunhui Zheng, Vijay Ganesh, Sanu Subramanian, Omer Tripp, Julian 2013. Jalangi: a selective record-replay and dynamic analysis frame- Dolby, and Xiangyu Zhang. 2015. Effective Search-Space Pruning work for JavaScript. In Foundations of Software Engineering (FSE). for Solvers of String Equations, Regular Expressions and Length Con- [32] Julian Thomé, Lwin Khin Shar, Domenico Bianculli, and Lionel C. straints. In Computer Aided Verification (CAV). Briand. 2017. Search-driven string constraint solving for vulnerability [40] Yunhui Zheng, Xiangyu Zhang, and Vijay Ganesh. 2013. Z3-str: A detection. In Int. Conf. Software Engineering (ICSE). Z3-based String Solver for Web Application Analysis. In Foundations [33] Nikolai Tillmann and Wolfram Schulte. 2005. Parameterized unit tests. of Software Engineering (FSE). In Foundations of Software Engineering (FSE).
15