Regular Languages of Nested Words: Fixed Points, Automata, and Synchronization ∗ Marcelo Arenas Departamento de Ciencia de la Computaci´on Pontificia Universidad Cat´olica de Chile [email protected] Pablo Barcel´o Leonid Libkin Departamento de Ciencia de la Computaci´on School of Informatics Universidad de Chile University of Edinburgh [email protected] [email protected] Abstract Nested words provide a natural model of runs of programs with recursive procedure calls. The usual connection between monadic second-order logic (MSO) and automata extends from words to nested words and gives us a natural notion of regular languages of nested words. In this paper we look at some well-known aspects of regular languages – their characterization via fixed points, deterministic and alternating automata for them, and synchronization for defining regular relations – and extend them to nested words. We show that mu-calculus is as expressive as MSO over finite and infinite nested words, and the equivalence holds, more generally, for mu-calculus with past modalities evaluated in arbitrary positions in a word, not only in the first position. We introduce the notion of alternating automata for nested words, show that they are as expressive as the usual automata, and also prove that Muller automata can be determinized (unlike in the case of visibly pushdown languages). Finally we look at synchronization over nested words. We show that the usual letter-to-letter synchronization is completely incompatible with nested words (in the sense that even the weakest form of it leads to an undecidable formalism) and present an alternative form of synchronization that gives us decidable notions of regular relations. 1 Introduction Nested words, introduced by Alur and Madhusudan [6], extend finite or infinite words with a hierar- chical nesting structure. The intuitive idea is that a nested word represents a model of execution of a program with recursive procedure calls; the nesting relation then connects matching calls and returns, while other elements correspond to internal operations. An example of a finite nested word is shown in the picture below, where the ri’s are returns matching calls ci’s. c1 c2 c3 r3 c4 r4 r2 c5 c6 r6 r5 c7 r7 r1 ∗This is the full version of the extended abstract presented at the 34th Intl. Colloquium on Languages, Automata, and Programming, ICALP’07. 1 Such structures naturally appear in XML documents that are string representations of trees using opening and closing tags [32, 8], or in software verification of programs with stack-based control flow [4, 2]. A nested word automaton [6] runs from left to right, similarly to a finite state automaton, but each time it encounters a “return” position, the next state depends not only on the current state but also on the state of the matching “call”. A nice property of nested words and their automata is that they share logical characterizations with the usual (unnested) words: the finite-automaton model has the same expressiveness as monadic second-order logic (MSO) [5, 6]. This gives us a natural and robust notion of regular languages of nested words, with the expected closure properties, decision procedures, and logical characterizations. For finite or infinite unnested words, an alternative way of describing regularity logically is via the modal µ-calculus (cf. [7]). That is, µ-calculus formulae evaluated in the first position of a word define precisely the regular languages. Moreover, µ-calculus formulae with past modalities evaluated in an arbitrary position of a word have precisely the power of MSO formulae with one free first-order variable. As our first result, we extend these equivalences to the case of finite and infinite nested words. We then look at automata characterizations of regular languages of nested words. Nondeterministic and deterministic automata have previously been considered [5, 6, 25], and [5] showed that automata can be determinized in the finite case, but in the infinite case this is impossible even for automata with a Muller acceptance condition (unlike in the case of the usual ω-words). Then [25] introduced a different automaton model and showed that it admits a determinization procedure over nested words. We expand this in two ways. First we introduce alternation in the case of nested word automata, and prove that alternating automata can still be translated into nondeterministic ones. Second, we refine the determinization procedure for automata from [25] to show that over infinite nested words, every regular language is definable by a deterministic Muller automaton. This also gives us some corollaries about the structure of regular languages of nested ω-words. We finally turn our attention to the notion of regular relations. Over words or trees, one moves from sets to relations by using letter-to-letter synchronization. For example, over words, an automaton runs over a tuple of words viewing the tuple of ith letters of the words as a single letter of an expanded alphabet [18]. The same approach works for trees, ranked and unranked [11]. The notion of regular relations also leads to a notion of automatic structures [13, 15, 10], i.e. decidable first-order structures over words in which all definable relations are regular. In the case of nested words, there are two ways of synchronizing them: either by considering their linear structure, as for words, or by considering the tree structure imposed by the nesting relation. While in the latter case we essentially deal with the known case of trees, we show that, in contrast, the notion of letter-to-letter synchronization that uses the linear structure is incompatible with nested words: the simplest extension of nested word automata with such synchronization is undecidable. We also show how the tree synchronization can be interpreted over the linear structure, by presenting an alternative call-return notion of synchronization. Related work Regular languages of nested words are a special case of visibly pushdown languages (VPL) [5], which are a restriction of the class of context-free languages that subsumes all regular properties and some non-regular properties relevant in program analysis (e.g. stack-inspection prop- erties and pre-post conditions). VPLs in many ways resemble regular languages: they have the same closure properties, and most natural problems related to them are decidable. The idea of VPLs is that the input alphabet Σ is partitioned into three parts, Σc, Σr, Σi, of symbols viewed as procedure 2 calls, returns, and internal operations. A machine model for VPLs is a special pushdown automaton that pushes a symbol onto the stack in a call, pops one symbol in a return, and does not touch the stack when reading an internal symbol. VPLs were introduced in [5] and regular languages of nested words in [6]. Nested words can be viewed as special classes of trees (and we shall use this often in the paper); such tree representations were introduced in [5, 6] as well. Applications in program analysis are discussed, e.g., in [2, 4], and applications in processing tree-structured data in [32, 8]. Alternating automata for nested words were introduced independently, and at about the same time, in [14]. In this paper, we compare our automata model with the automata model introduced in [14] and, in particular, we use a result from [14] to prove that alternation can be eliminated in our case. There are several related results on µ-calculus and MSO, e.g. their equality over infinite binary trees [29] or finite unranked trees [9] or expressive-completeness of µ-calculus [21]. We explain in Section 3 why we cannot derive our result from those. Another fixed-point logic VPµ is defined in [2] to specify properties of executions of programs. It differs from the standard versions of µ-calculus we look at as its fixed points are evaluated not over sets of nodes but over sets of subtrees of the program; further, its expressiveness is known to be different from MSO [3]. Nondeterministic automata for VPLs and regular languages of nested words were defined in [5, 6], and [5] observed that Muller automata for VPLs (over infinite words) are not determinizable. Then [25] noticed that this is due to VPLs having potentially arbitrarily many unmatched calls/returns, and introduced a different automaton model (stair automata) that can be determinized. We use them to show how to determinize finite-state Muller automata over nested ω-words. None of these papers addresses alternating automata over nested words. Letter-to-letter synchronization for defining regular relations is an old notion [18], and the concept of universal automatic structures [15, 13] is based on it. Although such automatic structures exist for both words and trees [10, 11], we show here that letter-to-letter synchronization is incompatible with nesting structure. A very different notion of synchronization for pushdown automata (that generalizes VPLs) was studied in [16]. Organization Basic definitions are given in Section 2. We describe MSO unary queries via µ-calculus in Section 3. In Section 4 we study automata for nested words, define alternating automata, and describe determinization for Muller automata. In Section 5 we look at synchronization and regular relations for nested words. 2 Preliminaries ∗ Words, ω-words, and automata Let Σ be a finite alphabet. A finite word w = a1 ...an in Σ is represented as a logical structure h {1,...,n} , (Pa)a∈Σ ,< i, where < is the usual linear order on {1,...,n}, and Pa is the set of i’s such that ai = a. We shall use w to refer to both the word and its logical representation. Infinite, or ω-words, are sequences a1a2 · · · of symbols in Σ indexed by positive + natural numbers, and are represented as structures hN , (Pa)a∈Σ,<i.