PATHS IN INFINITE TREES: LOGICS AND AUTOMATA Von der Fakultät für Mathematik, Informatik und Naturwissenschaften der RWTH Aachen University zur Erlangung des akademischen Grades einer Doktorin der Naturwissenschaften genehmigte Dissertation vorgelegt von Diplom-Informatikerin ALEXANDRA SPELTEN aus Koblenz, Rheinland-Pfalz Berichter: Universitätsprofessor Dr. Dr.h.c. Wolfgang Thomas Universitätsprofessor Dr. Erich Grädel Tag der mündlichen Prüfung: 28. Januar 2013 Diese Dissertation ist auf den Internetseiten der Hochschulbibliothek online verfügbar. Abstract In this thesis, several logical systems over infinite trees and infinite words are studied in their relation to finite automata. The first part addresses (over the infinite binary tree) “path logic” and “chain logic” as fragments of monadic second-order logic that allow quantification over paths, respectively subsets of paths of the binary tree. Many systems of bran- ching-time logic are subsumed by chain logic. We introduce ranked alternating tree automata as a computation model that characterizes chain logic. The main idea is to associate ranks to states such that in an automaton run, starting from the root, the ranks have to decrease and are allowed to remain unchanged only in one direction (either in existential or universal branching). The second part of the thesis is motivated by chain logic over infinite-bran- ching trees (where the successors of a node are indexed by natural numbers). A path through the N-branching tree is given by an ω-word over the infinite alpha- bet N. As a preparation for the study of path logics over such trees, we develop a theory of logics and automata over infinite alphabets, more precisely over alpha- bet frames (M, L), given by a relational structure M, supplying the alphabet, and a logic L that is used in specifying letter properties and automaton transitions. Two types of automata (and logics) for the specification of word properties are presented, depending whether or not relations between successive letters are in- cluded. We obtain results that clarify under which circumstances the nonempti- ness problem is solvable, and we apply these results to show (un-)decidability results on path logics over infinitely-branching trees that result from given struc- tures by weak and strong “tree iteration”. Zusammenfassung In der vorliegenden Arbeit werden verschiedene Logiksysteme über unendlichen Bäumen und unendlichen Wörtern in ihrer Beziehung zu endlichen Automaten untersucht. Dazu analysieren wir im ersten Teil als Fragmente von monadischer Logik zweiter Stufe die Logiken “Pfadlogik” und “Kettenlogik” (über dem unendlichen Binärbaum), die Quantifizierung über Pfaden bzw. Teilmengen von Pfaden des Binärbaums erlauben. Viele Systeme der “branching time logic” werden von Ket- tenlogik erfaßt. Als automatentheoretische Charakterisierung von Kettenlogik führen wir alternierende Rang-Baumautomaten ein, deren Zuständen jeweils ein Rang zugeordnet werden. Die Ketten werden dann simuliert, indem sicherge- stellt wird, dass in einem Lauf des Automaten, beginnend in der Wurzel, die Ränge stetig abfallen müssen und nur in höchstens einer Richtung des Baumes erhalten bleiben dürfen (in existentieller oder universeller Verzweigung). Der zweite Teil dieser Dissertation wird von Kettenlogik über unendlich ver- zweigten Bäumen (wobei die Nachfolger eines Knotens durch natürliche Zahlen indiziert werden) motiviert. Ein Pfad durch einen solchen N-verzweigten Baum wird durch ein ω-Wort über dem unendlichen Alphabet N beschrieben. Als Vorbereitung für die Studie von Pfadlogiken über solchen Bäumen entwickeln wir eine Theorie von Logiken und Automaten über unendlichen Alphabeten; genauer über Alphabetrahmen (M, L), welche durch eine Struktur M, die das Alphabet stellt, und eine Logik L, mit der Eigenschaften von Buchstaben und Automatentransitionen spezifiziert werden, gegeben sind. Wir präsentieren zwei Typen von Automaten (und Logiken) für die Spezifikation von Worteigenschaf- ten, abhängig davon, ob Relationen aufeinanderfolgender Buchstaben enthalten sind oder nicht. Wir erhalten Resultate, die klarstellen, unter welchen Bedingun- gen das Leerheitsproblem lösbar ist, und wenden diese daraufhin an, um (Un-) Entscheidbarkeitsergebnisse für Pfadlogiken über unendlich verzweigten Bäu- men zu zeigen, welche aus gegebenen Strukturen durch schwache und starke “Baumiteration” entstehen. Contents 1. Introduction................................................... .. 1 2. Background on MSO and Automata................................ 7 3. Path Logic and Automata over Binary Trees......................... 19 3.1 PathLogicandChainLogic .......................... ......... 20 3.2 Ranked Alternating Tree Automata. .......... 21 3.3 From Chain Logic to Automata and Back. ....... 29 4. Infinite-Branching Trees and Words over Infinite Alphabets .......... 41 4.1 Motivation...................................... ............ 41 4.2 Languages over Structured Alphabets . .......... 43 4.2.1 Word Models: Definitions. ........ 43 4.2.2 (M, L)-Automata ....................................... 48 4.2.2.1 Definitions and Nonemptiness Problem . .... 48 4.2.2.2 ClosureProperties........................... ........ 50 4.2.2.3 Equivalence between MSO and Automata over (M, L) ... 56 4.2.3 StrongAutomata ................................ ........ 62 4.2.3.1 Definitions.................................. ........ 63 4.2.3.2 An Undecidability Result . ....... 64 4.2.3.3 Closure properties and Nonemptiness Problem. ..... 66 4.2.4 Comparing Distant Letters . ......... 69 4.3 TreeModels...................................... ........... 74 4.3.1 Iterating Relational Structures . ........... 75 4.3.2 WeakTreeIterations............................. ......... 78 4.3.3 StrongTreeIterations........................... .......... 80 5. Conclusion................................................... ... 83 Bibliography ................................................... .... 85 i ii Contents Chapter 1 Introduction Many fields of theoretical computer science that deal with verification or synthe- sis of state-based systems have their foundation in the theory of finite automata, more precisely in results that connect automata with logical systems. The first results of this kind were shown in the 1960s, first by Büchi, Elgot, and Trakh- tenbrot connecting finite automata and monadic second-order logic over finite words as formalisms that are expressively equivalent (cf. [Büc60, Elg61, Tra61]). Further steps were analogous results over infinite words (Büchi [Büc62]), finite trees (Doner [Don70], Thatcher, Wright [TW68]), and finally the celebrated “tree theorem” of Rabin [Rab69] that showed the expressive equivalence between “Ra- bin tree automata” and monadic second-order logic (MSO-logic) over the binary tree. As a main application, the MSO-theory of the infinite binary tree was shown to be decidable. The use of MSO-logic can be motivated by the aim to describe runs of finite automata on the structure under consideration. A run defines a “coloring” (by states) on this input structure, and thus the existence of a run (or a finite coloring, say with k colors) can be expressed by a statement on the existence of k subsets of the structure’s domain, one for each color, collecting those elements that carry this color. Thus monadic logic is a natural choice to describe existence of runs, explaining partly the “match made in heaven” [Var03] between MSO-logic and finite automata. However, in the applications of MSO-logic (over trees) in the areas of verifi- cation and synthesis, certain fragments of MSO-logic are taken rather than full ∗ MSO-logic. This is visible, for example, in the systems CTL and CTL of “compu- tation tree logic”. Here only quantifiers over paths are needed (besides first-order ∗ quantifiers), i.e., very special subsets of the tree. For example the CTL -formula p1 ∧ E(Gp2 ∧ FEFp3) says “p1 is true in s0, and starting from s0 there is an infinite path π through the model such that all states on π satisfy p2, and in some state of π a path ′ π starts with some state satisfying p3”. The situation does not change when adding “automaton operators” on the ∗ ∗ paths, leading to the system ECTL of “extended CTL ”. Here a further kind of quantifier enters, namely quantifiers over subsets of paths, also called chains. 1 2 Chapter 1: Introduction In papers of the 1980s, Thomas [Tho84, Tho87] has introduced and studied these subsystems of MSO-logic, called path logic and chain logic. It was shown that they are strictly less expressive than MSO-logic (and that path logic is strictly weaker than chain logic). This weakness was then exploited to show the decid- ability of the chain logic theory of an expanded tree structure, namely the binary tree expanded by the “equal-level predicate” (connecting two tree nodes if they occur on the same level of the tree, i.e., have the same distance to the root). Also a kind of “dual approach” to subsystems of MSO-logic was pursued in the consideration of antichain logic (where second-order quantifiers range over sets whose elements are pairwise incomparable with respect to partial tree or- dering, cf. [Tho84]). In these studies, the “composition method” adapted to path logic and chain logic was applied. This method is built on the notion of “types” that capture the equivalence between structures with respect to formulas of a given quantifier-rank, and results on the computation of types of structures that are composed from substructures (e.g. words composed by the concatenation of segments). A missing piece in