The complexity of enumeration and counting for acyclic conjunctive queries Arnaud Durand Cambridge, march 2012 Arnaud Durand enumeration and counting for acyclic conjunctive queries Counting and enumeration Counting Enumeration I Output the number of solutions I Generate all solutions one by one I Example : ]sat, ]Perfect Matching, Permanent, I Output can be large problems on graph, knots I Natural algorithmic task polynomials This talk: look at these two tasks for fragments of conjunctive (i.e. ; ) queries (mainly ACQ) f9 ^g Arnaud Durand enumeration and counting for acyclic conjunctive queries Outline I Enumeration I A tour on complexity measure for enumeration I Results for query problems I Complexity characterization for ACQ I Weighted Counting I Polynomial representation of query I Counting through polynomial evaluation I Characterization of the tractability frontier for ACQ Common point : role of free variables in formulas. Arnaud Durand enumeration and counting for acyclic conjunctive queries Enumeration: measure of complexity Intractability I NP-completeness of the existence of one solution (hard to start) I hard after generation of a part of the solution set (hard to continue) Tractability I Polynomial total time (succeed but don't know how) I Incremental Polynomial Time (harder as the number of generated solutions increases) I Polynomial delay (regular process) Arnaud Durand enumeration and counting for acyclic conjunctive queries Example I Generate all models of a propositional formula (hum... intractable) I Generate all independant sets of maximum size of a graph (intractable) I Generate all maximal (for inclusion) independant sets I polynomial delay for lexicographic ordering I intractable for reverse lexicographic ordering I Generate all models of a 2-CNF propositional formula (polynomial delay) Arnaud Durand enumeration and counting for acyclic conjunctive queries Complexity of enumeration of satisfiability problems I Let ' be a formula with variables x1,..., xn. I If ' x SAT : enumerate all solutions of ' x ^ : 1 2 ^ : 1 I If ' x SAT : enumerate all solutions of ' x ^ 1 2 ^ 1 Theorem (Creignou, Hebrard'97) In the Boolean case, there is no other efficient algorithm than the one described above. Consequently, easy (i.e. polynomial delay) cases are Horn, Anti-Horn, 2-CNF and affine. Schnoor & Schnoor'07 The situation is more complex for CSP on arbitrary finite domain See also Bulatov, Dalmau, Grohe & Marx'10 Arnaud Durand enumeration and counting for acyclic conjunctive queries Enumeration algorithm: some precision Precomputation step Algorithms in two steps: + Enumeration step(s) I Computation models (in this talk) : RAM with uniform cost (but with bounded values in registers) I Let be an enumeration algorithm for Enum(R) and x I . A 2 I timei (x) denotes the time when the algorithm finishes to write the ith solution if it exists. I delayi (x) = timei (x) timei (x). +1 − Delay may be : sub-exponential, polynomial, linear, constant (?) Warning: space is an important factor! Arnaud Durand enumeration and counting for acyclic conjunctive queries Small classes of enumeration problems I An enumeration algorithm is constant delay if there is a A constant c such that for any x I and i, delayi (x) c. 2 ≤ I Enum(R) is computable with constant delay (Enum(R) Constant-Delay) if there is a constant delay 2 algorithm which computes Enum(R). A I Enum(R) Constant-Delaylin if it is reducible in linear 2 time (in the input size) to a problem in Constant-Delay precomputation reduction ≡ Arnaud Durand enumeration and counting for acyclic conjunctive queries Query problems Query I = set of formulas L I = set of (finite) structures S Enumeration problem Enum( ; ) L S input: φ and 2 L A 2 S output: enumerate φ( ) = ¯a Dk ( ; ¯a) = φ(¯x) A f 2 j A j g Counting problem Count( ; ) L S input: φ and 2 L A 2 S output: φ( ) = ¯a Dk ( ; ¯a) = φ(¯x) j A j jf 2 j A j gj Arnaud Durand enumeration and counting for acyclic conjunctive queries Complexity Measures Framework for complexity analysis of multi-parameterized problems Parameters , ' (inputs) A '( ) A But also: number of free variables, of quantified variables, alternation, arity... I , ' and '( ) : Combined Complexity. jAj j j j A j I and '( ) : Data Complexity jAj j A j I ' and '( ) : Expression Complexity j j j A j Parameterized complexity: express the complexity in terms of , ' and '( ) but consider ' as a parameter. jAj j j j A j j j Arnaud Durand enumeration and counting for acyclic conjunctive queries Some tractable cases for enumeration Data complexity τ I degd : the class of τ-structures of maximal degree d. τ ≤ Enum(fo; deg ) constant-delaylin. (D. Grandjean'07, Lindell'08) d 2 Delay : triply exponential in the size of formula (Kazana, Segoufin’10) τ I twk : the class of τ-structures of treewidth at most k. τ Enum(mso; tw ) linear-delaylin. (Bagan'06, Courcelle'07, Frick, ..) k 2 Arnaud Durand enumeration and counting for acyclic conjunctive queries Conjunctive Queries Conjunctive Queries (CQ) I : ; -fragment of first-order logic i.e. formulas L f9 ^g '(¯x) y¯ (¯x; y¯) where is a conjunction of atoms. ≡ 9 I Combined complexity of model checking: NP-complete. No hope for enumeration. I Tractable fragments for decision by restricting the formula (acyclicity, hypergraph decomposition methods). Arnaud Durand enumeration and counting for acyclic conjunctive queries Hypergraph associated to formulas I The hypergraph of a formula ' is the hypergraph H = (V ; E) such that I V is the set of variables of ' I for each atom R(¯v) of ', we associate a hyperedge var(R(¯v)) t u x v y z Arnaud Durand enumeration and counting for acyclic conjunctive queries Acyclic Hypergraph Most general notion: α-acyclicity. An hypergraph is α-acyclic if the application of the following rules as long as possible outputs the empty hypergraph: 1. Remove hyperedges contained in other hyperedges; 2. Remove vertices that appear in at most one hyperedge. A conjunctive query is acyclic if its associated hypergraph is acyclic. Arnaud Durand enumeration and counting for acyclic conjunctive queries Acyclic hypergraph/query: examples Acyclic Cyclic Acyclic Large class of hypergraph/queries. Base case for more evolved decomposition methods. Remark: Adding an edge to a cyclic hypergraph may result in an acyclic hypergraph. Arnaud Durand enumeration and counting for acyclic conjunctive queries Acyclic Hypergraph: alternative definition A join tree of a hypergraph = (V ; E) is a pair ( ; λ) where H T = (VT ; T ) is a tree and λ is a function from VT to E such that: T I For each e E, there is a t VT such that λ(t) = e (each 2 2 vertex of is a bag containing one edge) T I For each v V , the set t VT : v λ(t) is a connected 2 f 2 2 g subtree of T . Equivalent characterization : is acyclic iff it has a join tree. H Arnaud Durand enumeration and counting for acyclic conjunctive queries Acyclic Hypergraph: alternative definition Acyclic hypergraphs I organize the edges of a graph into a “join tree” e a, c, e, g f d g a, b, c a, e, f c, d, e a b c I with a connectivity condition: edges that contain vertex v are connected in the tree I hypergraph is acyclic if it has a join tree Arnaud Durand enumeration and counting for acyclic conjunctive queries 8/20 Complexity of acyclic conjunctive queries ACQ Acyclic Conjunctive Queries. ACQ=6 Acyclic Conjunctive Queries with inequalities Example: '(x; y) z t u(Rxyz Ryzt Rztu Syt x = u) ≡ 9 9 9 ^ ^ ^ ^ 6 Known results ACQ solvable in O( ' '( ) ) times (Yannakakis'81) j j × jAj × j A j ACQ=6 solvable in O(2O(j'j) '( ) log2( )) times × jAj × j A j × jAj (Papadimitriou & Yannakakis-99) or in O(( ' !) '( ) ) (Bagan-D.-Grandjean-07) j j × jAj × j A j Arnaud Durand enumeration and counting for acyclic conjunctive queries Yannakakis algorithm Input:( ;'), a tree decomposition T A - Take a leaf t VT , let R(¯x; z¯) 2 the associated atom. - Take its father associated to S(¯x; y¯)( y¯ z¯ = ) f g \ f g ; Filter relation S in by relation A R : S(x,y) S := (¯a; b¯) S and c¯(¯a; c¯) R f 2 9 2 g R(x,z) Arnaud Durand enumeration and counting for acyclic conjunctive queries Yannakakis algorithm Input:( ;'), a tree decomposition T A - Take a leaf t VT , let R(¯x; z¯) 2 the associated atom. - Take its father associated to S(¯x; y¯)( y¯ z¯ = ) f g \ f g ; Filter relation S in by relation A R : S(x,y) S := (¯a; b¯) S and c¯(¯a; c¯) R f 2 9 2 g Continue bottom up with new data and drop R(¯x; z¯) from the list of atoms Arnaud Durand enumeration and counting for acyclic conjunctive queries Enumeration of ACQ Enumeration Can we enumerate the results efficiently ? Easy but not that fast! 0 I let ' (x1) x2 ::: xk y¯'(¯x; y¯) ≡ 9 0 9 9 I For a ENUM(' ; ) 2 A I let 'a '(a; x ;:::; xk ; y¯) ≡ 2 I For b¯ ENUM('a; ) : output (a; b¯) 2 A Linear O( ' ) delay. j j × jAj Similarities with the SAT enumeration in easy cases. Arnaud Durand enumeration and counting for acyclic conjunctive queries Enumeration of ACQ Easy and fast if all variables are free (i.e. participate to the result). I Compute a tree decomposition I Filter each parent with its children so that only tuples that can be extended to a solution remain ("local" consistency) I run through the tree and output the solutions Linear O( ' ) precomputation and constant O( ' ) delay. j j × jAj j j Works also with disequalities (but delay exponential in ' ) j j Arnaud Durand enumeration and counting for acyclic conjunctive queries What's the point with enumeration Tree decomposition T for ' - Blue boxes : quantified variables only - Yellow boxes : free variables only - In real life : green boxes also..