CSCI 742 - Compiler Construction Lecture 18 Cocke-Younger-Kasami (CYK) Algorithm Instructor: Hossein Hojjat February 26, 2018 Recap: Chomsky Normal Form (CNF) A CFG is in Chomsky Normal Form if each rule is of the form A BC ! A a ! where a is any terminal • A,B,C are non-terminals • B, C cannot be start variable • We allow the rule S if L ! 2 1 Recap: Conversion to CNF 1. remove unproductive non-terminals (optional) 2. remove unreachable non-terminals (optional) 3. make terminals occur alone on right-hand side 4. reduce arity of every production to 2 ≤ 5. remove -production rules X ! 6. remove unit productions X Y ! 7. unproductive non-terminals 8. unreachable non-terminals 2 CYK (Cocke-Younger-Kasami) One of the earliest recognition and parsing algorithms • Bottom-up parsing (starts with words) • Independently developed by J.Cocke, D.Younger, T.Kasami (CYK) • Build solutions compositionally from sub-solutions • Based on a “dynamic programming” approach • 3 CYK algorithm: Basic idea For a grammar G and a word w: For every substring v of length 1, find all non-terminals A such that • 1 A ∗ v ) 1 For every substring v of length 2, find all non-terminals A such that • 2 A ∗ v2 . ) . For the unique substring w of length w , find all non-terminals A such that • j j A ∗ w ) Check whether S belongs to the last set 4 Substring Recognition s1,4 s2,4 word: w = a a a • 1 2 ··· n s1,3 Xi;j set of all non-terminals deriving the s3,4 • substring s s2,3 i;j s1,2 Start non-terminal S derives w (= s ) • 1;n s1,1 s2,2 s3,3 s4,4 if and only if S is a member of X1;n a1 a2 a3 a4 5 Substring Recognition How to check if the RHS of rule A A A A derives s ? • ! 1 2 ··· n i;j A1 A2 An (i < k < j) ai ak 1 ak ak+1 aj ··· − We need to check all the possible matching of substrings with A • i non-terminals Results of checking subproblem solutions are reusable • Subproblem results are computed once and then memoized for later • A1 A2 Matching in CNF: ai ak 1 ak aj ··· − ··· 6 if B belongs to X1;1 and C to X2;5 and if A BC is a grammar rule ! then add A to X1;5 Recognition Table X1,5 X1,4 X2,5 X1,3 X2,4 X3,5 X1,2 X2,3 X3,4 X4,5 X1,1 X2,2 X3,3 X4,4 X5,5 a1 a2 a3 a4 a5 Each row corresponds to one length of substrings Bottom Row: words of length 1 • Second from Bottom Row: words of length 2 • . • Top Row: word w • 7 Recognition Table if B belongs to X1;1 and C to X2;5 X1,5 and if A BC is a grammar rule ! then add A to X X1,4 X2,5 1;5 X1,3 X2,4 X3,5 X1,2 X2,3 X3,4 X4,5 X1,1 X2,2 X3,3 X4,4 X5,5 a1 a2 a3 a4 a5 Each row corresponds to one length of substrings Bottom Row: words of length 1 • Second from Bottom Row: words of length 2 • . • Top Row: word w • 7 if B belongs to X1;1 and C to X2;5 and if A BC is a grammar rule ! then add A to X1;5 Recognition Table X1,5 X1,4 X2,5 X1,3 X2,4 X3,5 X1,2 X2,3 X3,4 X4,5 X1,1 X2,2 X3,3 X4,4 X5,5 a1 a2 a3 a4 a5 Each row corresponds to one length of substrings Bottom Row: words of length 1 • Second from Bottom Row: words of length 2 • . • Top Row: word w • 7 if B belongs to X1;1 and C to X2;5 and if A BC is a grammar rule ! then add A to X1;5 Recognition Table X1,5 X1,4 X2,5 X1,3 X2,4 X3,5 X1,2 X2,3 X3,4 X4,5 X1,1 X2,2 X3,3 X4,4 X5,5 a1 a2 a3 a4 a5 Each row corresponds to one length of substrings Bottom Row: words of length 1 • Second from Bottom Row: words of length 2 • . • Top Row: word w • 7 if B belongs to X1;1 and C to X2;5 and if A BC is a grammar rule ! then add A to X1;5 Recognition Table X1,5 X1,4 X2,5 X1,3 X2,4 X3,5 X1,2 X2,3 X3,4 X4,5 X1,1 X2,2 X3,3 X4,4 X5,5 a1 a2 a3 a4 a5 Each row corresponds to one length of substrings Bottom Row: words of length 1 • Second from Bottom Row: words of length 2 • . • Top Row: word w • 7 Example S AB CB ! j A BA a grammar: ! j intput: babab B BC b ! j C AC a ! j B A, C B A, C B b a b a b 8 Example S AB CB ! j A BA a grammar: ! j intput: babab B BC b ! j C AC a ! j A, B S A, B S B A, C B A, C B b a b a b 8 Example S AB CB ! j A BA a grammar: ! j intput: babab B BC b ! j C AC a ! j S S S A, B S A, B S B A, C B A, C B b a b a b 8 Example S AB CB ! j A BA a grammar: ! j intput: babab B BC b ! j C AC a ! j S, A S S S A, B S A, B S B A, C B A, C B b a b a b 8 Example S AB CB ! j A BA a grammar: ! j intput: babab B BC b ! j C AC a ! j S S, A S S S A, B S A, B S B A, C B A, C B b a b a b 8 CYK algorithm Standard CYK operates only on context-free grammars in • Chomsky Normal Form (CNF) To illustrate some of the benefits of CNF we use a non-CNF • grammar 9 Example word: 12.3e+4 Number ! Integer j Real Integer ! Digit j Integer Digit Real ! Integer Fraction Scale Fraction ! : Integer Scale ! e Sign Integer j Digit ! 0 j 1 j · · · j 8 j 9 Sign ! + j − 1 2 . 3 e + 4 10 Grammar has unit-rules: after finding a non-terminal, • we need to repetitively search for other non-terminals that derive the same word Chain of rules: Number Integer Digit • ) ) Example word: 12.3e+4 Number ! Integer j Real Integer ! Digit j Integer Digit Real ! Integer Fraction Scale Fraction ! : Integer Scale ! e Sign Integer j Sign Digit ! 0 j 1 j · · · j 8 j 9 Digit Digit Digit Digit Sign ! + j − 1 2 . 3 e + 4 We start with substrings of length 1 • 10 Example word: 12.3e+4 Number ! Integer j Real Integer ! Digit j Integer Digit Real ! Integer Fraction Scale Fraction ! : Integer Scale ! e Sign Integer j Number Number Number Number Integer Integer Integer Sign Integer Digit ! 0 j 1 j · · · j 8 j 9 Digit Digit Digit Digit Sign ! + j − 1 2 . 3 e + 4 We start with substrings of length 1 • Grammar has unit-rules: after finding a non-terminal, • we need to repetitively search for other non-terminals that derive the same word Chain of rules: Number Integer Digit 10 • ) ) Example word: 12.3e+4 Number ! Integer j Real Integer ! Digit j Integer Digit Real ! Integer Fraction Scale Fraction ! : Integer Number, Integer Fraction Scale ! e Sign Integer j Number Number Number Number Integer Integer Integer Sign Integer Digit ! 0 j 1 j · · · j 8 j 9 Digit Digit Digit Digit Sign ! + j − 1 2 . 3 e + 4 10 Example word: 12.3e+4 Number ! Integer j Real Integer ! Digit j Integer Digit Real ! Integer Fraction Scale Scale Fraction ! : Integer Number, Integer Fraction Scale ! e Sign Integer j Number Number Number Number Integer Integer Integer Sign Integer Digit ! 0 j 1 j · · · j 8 j 9 Digit Digit Digit Digit Sign ! + j − 1 2 . 3 e + 4 10 Example word: 12.3e+4 Number ! Integer j Real Integer ! Digit j Integer Digit Number, Real Real ! Integer Fraction Scale Scale Fraction ! : Integer Number, Integer Fraction Scale ! e Sign Integer j Number Number Number Number Integer Integer Integer Sign Integer Digit ! 0 j 1 j · · · j 8 j 9 Digit Digit Digit Digit Sign ! + j − 1 2 . 3 e + 4 10 Example word: 12.3e+4 Number ! Integer j Real Number, Real Integer ! Digit j Integer Digit Number, Real Real ! Integer Fraction Scale Scale Fraction ! : Integer Number, Integer Fraction Scale ! e Sign Integer j Number Number Number Number Integer Integer Integer Sign Integer Digit ! 0 j 1 j · · · j 8 j 9 Digit Digit Digit Digit Sign ! + j − 1 2 . 3 e + 4 10 Example word: 12.3 Number ! Integer j Real Integer ! Digit j Integer Digit Real ! Integer Fraction Scale Number, Integer Fraction Fraction ! : Integer Number Number Number Scale ! e Sign Integer j Integer Integer Integer Digit ! 0 j 1 j · · · j 8 j 9 Digit Digit Digit Sign ! + j − 1 2 . 3 11 Example word: 12.3 Number ! Integer j Real Integer ! Digit j Integer Digit Number, Real Real ! Integer Fraction Scale Number, Real Number, Integer Fraction Scale Fraction ! : Integer Number Number Number Scale ! e Sign Integer j Integer Integer Integer Digit ! 0 j 1 j · · · j 8 j 9 Digit Digit Digit Sign ! + j − 1 2 .