Adaptive LL(*) Parsing: The Power of Dynamic Analysis Terence Parr Sam Harwell Kathleen Fisher University of San Francisco University of Texas at Austin Tufts University [email protected] [email protected] kfi[email protected] Abstract PEGs are unambiguous by definition but have a quirk where Despite the advances made by modern parsing strategies such rule A ! a j ab (meaning “A matches either a or ab”) can never as PEG, LL(*), GLR, and GLL, parsing is not a solved prob- match ab since PEGs choose the first alternative that matches lem. Existing approaches suffer from a number of weaknesses, a prefix of the remaining input. Nested backtracking makes de- including difficulties supporting side-effecting embedded ac- bugging PEGs difficult. tions, slow and/or unpredictable performance, and counter- Second, side-effecting programmer-supplied actions (muta- intuitive matching strategies. This paper introduces the ALL(*) tors) like print statements should be avoided in any strategy that parsing strategy that combines the simplicity, efficiency, and continuously speculates (PEG) or supports multiple interpreta- predictability of conventional top-down LL(k) parsers with the tions of the input (GLL and GLR) because such actions may power of a GLR-like mechanism to make parsing decisions. never really take place [17]. (Though DParser [24] supports The critical innovation is to move grammar analysis to parse- “final” actions when the programmer is certain a reduction is time, which lets ALL(*) handle any non-left-recursive context- part of an unambiguous final parse.) Without side effects, ac- free grammar. ALL(*) is O(n4) in theory but consistently per- tions must buffer data for all interpretations in immutable data forms linearly on grammars used in practice, outperforming structures or provide undo actions. The former mechanism is general strategies such as GLL and GLR by orders of magni- limited by memory size and the latter is not always easy or pos- tude. ANTLR 4 generates ALL(*) parsers and supports direct sible. The typical approach to avoiding mutators is to construct left-recursion through grammar rewriting. Widespread ANTLR a parse tree for post-parse processing, but such artifacts funda- 4 use (5000 downloads/month in 2013) provides evidence that mentally limit parsing to input files whose trees fit in memory. ALL(*) is effective for a wide variety of applications. Parsers that build parse trees cannot analyze large data files or infinite streams, such as network traffic, unless they can be pro- 1. Introduction cessed in logical chunks. Computer language parsing is still not a solved problem in Third, our experiments (Section 7) show that GLL and GLR practice, despite the sophistication of modern parsing strategies can be slow and unpredictable in time and space. Their com- and long history of academic study. When machine resources plexities are, respectively, O(n3) and O(np+1) where p is the were scarce, it made sense to force programmers to contort length of the longest production in the grammar [14]. (GLR is their grammars to fit the constraints of deterministic LALR(k) typically quoted as O(n3) because Kipps [15] gave such an al- or LL(k) parser generators.1 As machine resources grew, re- gorithm, albeit with a constant so high as to be impractical.) In searchers developed more powerful, but more costly, nondeter- theory, general parsers should handle deterministic grammars ministic parsing strategies following both “bottom-up” (LR- in near-linear time. In practice, we found GLL and GLR to be style) and “top-down” (LL-style) approaches. Strategies in- ˜135x slower than ALL(*) on a corpus of 12,920 Java 6 library clude GLR [26], Parser Expression Grammar (PEG) [9], LL(*) source files (123M) and 6 orders of magnitude slower on a sin- [20] from ANTLR 3, and recently, GLL [25], a fully general gle 3.2M Java file, respectively. top-down strategy. LL(*) addresses these weaknesses by providing a mostly de- Although these newer strategies are much easier to use than terministic parsing strategy that uses regular expressions, rep- LALR(k) and LL(k) parser generators, they suffer from a resented as deterministic finite automata (DFA), to potentially variety of weaknesses. First, nondeterministic parsers some- examine the entire remaining input rather than the fixed k- times have unanticipated behavior. GLL and GLR return multi- sequences of LL(k). Using DFA for lookahead limits LL(*) ple parse trees (forests) for ambiguous grammars because they decisions to distinguishing alternatives with regular lookahead were designed to handle natural language grammars, which are languages, even though lookahead languages (set of all pos- often intentionally ambiguous. For computer languages, am- sible remaining input phrases) are often context-free. But the biguity is almost always an error. One can certainly walk the main problem is that the LL(*) grammar condition is statically constructed parse forest to disambiguate it, but that approach undecidable and grammar analysis sometimes fails to find reg- costs extra time, space, and machinery for the uncommon case. ular expressions that distinguish between alternative produc- tions. ANTLR 3’s static analysis detects and avoids potentially- 1 We use the term deterministic in the way that deterministic finite automata undecidable situations, failing over to backtracking parsing de- (DFA) differ from nondeterministic finite automata (NFA): The next symbol(s) uniquely determine action. cisions instead. This gives LL(*) the same a j ab quirk as PEGs 1 2014/3/24 for such decisions. Backtracking decisions that choose the first ALL(*) parsers handle the task of matching terminals and matching alternative also cannot detect obvious ambiguities expanding nonterminals with the simplicity of LL but have such A ! α j α where α is some sequence of grammar sym- O(n4) theoretical time complexity (Theorem 6.3) because in bols that makes α j α non-LL(*). the worst-case, the parser must make a prediction at each input symbol and each prediction must examine the entire remaining 1.1 Dynamic grammar analysis input; examining an input symbol can cost O(n2). O(n4) is in line with the complexity of GLR. In Section 7, we show In this paper, we introduce Adaptive LL(*) , or ALL(*) , parsers empirically that ALL(*) parsers for common languages are that combine the simplicity of deterministic top-down parsers efficient and exhibit linear behavior in practice. with the power of a GLR-like mechanism to make parsing de- The advantages of ALL(*) stem from moving grammar anal- cisions. Specifically, LL parsing suspends at each prediction ysis to parse time, but this choice places an additional bur- decision point (nonterminal) and then resumes once the pre- den on grammar functional testing. As with all dynamic ap- diction mechanism has chosen the appropriate production to proaches, programmers must cover as many grammar position expand. The critical innovation is to move grammar analysis to and input sequence combinations as possible to find grammar parse-time; no static grammar analysis is needed. This choice ambiguities. Standard source code coverage tools can help pro- lets us avoid the undecidability of static LL(*) grammar anal- grammers measure grammar coverage for ALL(*) parsers. High ysis and lets us generate correct parsers (Theorem 6.1) for any coverage in the generated code corresponds to high grammar non-left-recursive context-free grammar (CFG). While static coverage. analysis must consider all possible input sequences, dynamic The ALL(*) algorithm is the foundation of the ANTLR 4 analysis need only consider the finite collection of input se- parser generator (ANTLR 3 is based upon LL(*)). ANTLR 4 quences actually seen. was released in January 2013 and gets about 5000 download- The idea behind the ALL(*) prediction mechanism is to s/month (source, binary, or ANTLRworks2 development envi- launch subparsers at a decision point, one per alternative pro- ronment, counting non-robot entries in web logs with unique duction. The subparsers operate in pseudo-parallel to explore IP addresses to get a lower bound.) Such activity provides evi- all possible paths. Subparsers die off as their paths fail to match dence that ALL(*) is useful and usable. the remaining input. The subparsers advance through the input The remainder of this paper is organized as follows. We be- in lockstep so analysis can identify a sole survivor at the min- gin by introducing the ANTLR 4 parser generator (Section 2) imum lookahead depth that uniquely predicts a production. If and discussing the ALL(*) parsing strategy (Section 3). Next, multiple subparsers coalesce together or reach the end of file, we define predicated grammars, their augmented transition the predictor announces an ambiguity and resolves it in favor network representation, and lookahead DFA (Section 4). Then, of the lowest production number associated with a surviving we describe ALL(*) grammar analysis and present the pars- subparser. (Productions are numbered to express precedence as ing algorithm itself (Section 5). Finally, we support our claims an automatic means of resolving ambiguities like PEGs; Bi- regarding ALL(*) correctness (Section 6) and efficiency (Sec- son also resolves conflicts by choosing the production specified tion 7) and examine related work (Section 8). Appendix A has first.) Programmers can also embed semantic predicates [22] to proofs for key ALL(*) theorems, Appendix B discusses algo- choose between ambiguous interpretations. rithm pragmatics, Appendix C has left-recursion elimination ALL(*) parsers memoize analysis results, incrementally and details. dynamically building up a cache of DFA that map lookahead phrases to predicted productions. (We use the term analysis in the sense that ALL(*) analysis yields lookahead DFA like static 2. ANTLR 4 LL(*) analysis.) The parser can make future predictions at the ANTLR 4 accepts as input any context-free grammar that does same parser decision and lookahead phrase quickly by consult- not contain indirect or hidden left-recursion.2 From the gram- ing the cache.