Qualitative Analysis of Synchronizing Probabilistic Systems Mahsa Shirmohammadi

Qualitative analysis of synchronizing probabilistic systems Mahsa Shirmohammadi

To cite this version:

Mahsa Shirmohammadi. Qualitative analysis of synchronizing probabilistic systems. Numerical Anal- ysis [cs.NA]. École normale supérieure de Cachan - ENS Cachan; Université libre de Bruxelles (1970-..), 2014. English. NNT : 2014DENS0054. tel-01153942

HAL Id: tel-01153942 https://tel.archives-ouvertes.fr/tel-01153942 Submitted on 20 May 2015

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. UNIVERSITÉ LIBRE DE BRUXELLES ÉCOLE NORMALE SUPÉRIEURE Service des méthodes formelles DE CACHAN et vérification Laboratoire spécification et vérification

Qualitative Analysis of Probabilistic Synchronizing Systems

Mahsa Shirmohammadi

Dissertation submitted for the joint degree of Doctor of Philosophy in computer science from Université Libre de Bruxelles, Belgium and Ecole Normale Supérieure de Cachan, France

Committee in charge: Laurent Doyen: Supervisor Thierry Massart: Supervisor Stefan Kiefer: Reviewer Jeremy Sproston: Reviewer Joost-Pieter Katoen: Examiner Jean François Raskin: Examiner Béatrice Bérard: Examiner Emmanuel Filiot: Examiner

Defense date: 10th December 2014

To Maryam, my mother.

Acknowledgment

I gratefully acknowledge all who supported me to accomplish my wish. Being so lucky to have supports from several countries and in several languages; I try to mention all, however I am surely bound to forget a few. Laurent, I admire the way you turn science into art. Your seriousness, patience, insights, enthusiasm and not to forget your expectations motivated me to work hard, to be thirsty to learn how to carry out deep research, write clear proofs and give good talks. I am so grateful that you made me rewrite proofs several times, for some more than there are fingers in a hand, to reformulate and simplify my ideas and to turn a problem up and down, and look from one angle to another. Wishing that I hadfilled my backpack with more of your knowledge and advices, I am at the same time grateful for the freedom I had while writing my thesis; I found my academic personality those days, thanks Laurent. Thierry, we have heard “doubt your beliefs and believe in your doubts” more than once from Laurent; however the one who taught it to me is you. I will always remain amazed by your capability offinding mistakes in proofs and counter examples for conjectures. You taught me to question and challenge myself, to not be easily convinced, thanks for all of those, and much more for the social support you never forgot to offer me. Thanks for welcoming me atfirst in Brussels, for helping me integrate in my new life, for simply being ’there’ when the sensitivity of a girl overlapped with her professionalism. Thanks Thierry. I am so pleased that Stefan Kiefer, Jeremy Sproston, Jean François Raskin, Joost-Pieter Katoen, Béatrice Bérard and Emmanuel Filiot accepted to be the other jury members of my thesis. A further thank for all asked questions during my private defense, some of which spark new directions of research. I am grateful to the Belgian national fund for scientific research (F.N.R.S.) and Inter- national relation office of ENS de Cachan which provided me with two scholarships during my study: one allowed me to fully concentrate on my thesis, and the other one facilitated my dual life between France and Belgium over three years. I should not forget any member of LSV (ENS de Cachan) and the Verification group at ULB, with a special mention to Benedikt, Frédéric, Gilles, Dietmar, Luc, Cristina, Thida, Thomas and Julien who smiled at me and made refreshing small talks every time I met them during coffee breaks. I also appreciate all the helps I received from the secretaries of both labs: Pascaline, Maryka, Véronique and Virgine several times took care of administrative papers instead of the lost-in-French Mahsa. I wish to say thanks to Nicolas Markey (a big thanks), Stéphanie Delaune and Alain Finkel for constructive advice; to Kim Larsen and Line for welcoming me in Aalborg university while collaborating; to Alexander for teaching me how to teach; to Sylvain for saving my introduction; to Nathann for knocking the door of my office in a gloomy winter day; to all adorable girls who surrounded both me and my surviving feminine trait: Sinem jan janam, dear Aina, delicate Marie, small mommy Sandie, strong Raffaella, happiest ever Barbara, lively Aiswarya, organized Laure, fun Juliette and the other fun Juliette, kind Eli,

i sportive Line, creative Carla, hard-working Atefeh, backgammon-champion Öykü, eastern- soul Xiajou, sweet Agathe, the only-musician-friend Nona and far-away Eleni (girls, I lived moments with you, your weddings, your 25th, 30th birthdays, your pregnancy worried days, your graduations, your love or hate days, simply normal days, thank you, thank you for all); to Gabriel and Tristan to be myfirst and so good friends in Europe; to Guillermo, Romain and the other Romain for being such unique charming office-mates, and for enjoying summer BBQs where I have to mention Ocan and Mathieu, the delicious- sauce maker; to Chris, Hernan, Vince and César for never rejecting to try what I cook and even complimenting it, and more than that for the friendship and respect between us; to Ali and Hani my best ever friends that keep our relation fresh and deep from such a long distance; to my soul-mates Mina, Mona and Somayeh for not letting me disconnect from the other world of mine by long and late phone conversations; to Mani for lifting spirit words; and to dear Hossein for encouraging me to pursue my post-graduate study and be more independent and more honest. Vincent and Christoph, I am indebted to you guys. Ask me why, and I need to think minutes before taking out one of the hundreds cards from the deck of friendly supports, scientific advices, stressful critical times, relaxed laughs and smiles, sunny and rainy but all memorable days. Vince, you have been so generous to me, so more than what others can guess, so perfectly careful with the fragile glass of my soul, thanks. Chris, you make me look into myself and hunt clues of peace and braveness whose existences were lost over years, thanks. Special thanks to Vince for going through all my thesis, and to Chris for commenting my introduction and my post-doctorate application to Oxford. Thanks Vince, thanks Chris. Madaram Maryam, my mother Maryam, one tip of afinger from your kindness and devotion was sufficient to make me who I am, though you lived your life in each of ours, your children. My eyes have always felt shy to look up at the height of the shadow of your support and to say thanks; I can only write it down as down as my short mount of achievements. Thanks Maryam, Madaram.

Mahsa so thankful, after herﬁrst Thanksgiving dinner at Tony’s place Paris, FMSH library December 2014.

ii Abstract

Markov decision processes (MDPs) arefinite-state probabilistic systems with both strategic and random choices, hence well-established to model the interactions between a controller and its randomly responding environment. An MDP can be mathematically 1 viewed as a1 2 -player stochastic game played in rounds when the controller chooses an action, and the environment chooses a successor according to afixed probability distribution. There are two incomparable views on the behavior of an MDP, when the strategic choices arefixed. In the traditional view, an MDP is a generator of sequence of states, called the state-outcome; the winning condition of the player is thus expressed as a set of desired sequences of states that are visited during the game, e.g. Borel condition such as reachability. The computational complexity of related decision problems and memory requirement of winning strategies for the state-outcome conditions are well-studied. Recently, MDPs have been viewed as generators of sequences of probability distributions over states, called the distribution-outcome. We introduce synchronizing conditions defined on distribution-outcomes, which intuitively requires that the probability mass accumulates in some (group of) state(s), possibly in limit. A probability distribution isp-synchronizing if the probability mass is at leastp in some state, and a sequence of probability distributions is always, eventually, weakly, or stronglyp-synchronizing if respectively all, some, infinitely many, or all butfinitely many distributions in the sequence arep-synchronizing. For each synchronizing mode, an MDP can be(i) sure winning if there is a strategy that produces a1-synchronizing sequence;(ii) almost-sure winning if there is a strategy that produces a sequence that is, for allϵ>0, a (1-ϵ)-synchronizing sequence;(iii) limit-sure winning if for allϵ>0, there is a strategy that produces a (1-ϵ)-synchronizing sequence. We consider the problem of deciding whether an MDP is winning, for each synchronizing and winning mode: we establish matching upper and lower complexity bounds of the problems, as well as the memory requirement for optimal winning strategies. As a further contribution, we study synchronization in probabilistic automata (PAs), that are kind of MDPs where controllers are restricted to use only word-strategies; i.e. no ability to observe the history of the system execution, but the number of choices made so far. The synchronizing languages of a PA is then the set of all synchronizing word- strategies: we establish the computational complexity of the emptiness problems for all synchronizing languages in all winning modes. We carry over results for synchronizing problems from MDPs and PAs to two-player turn-based games and non-deterministicfinite state automata. Along with the main results, we establish new complexity results for alternatingfinite automata over a one-letter alphabet. In addition, we study different variants of synchronization for timed and weighted automata, as two instances of infinite-state systems.

iii iv Résumé Les Markov Decision Process (MDP) sont des systèmesfinis probabilistes avec à la fois des choix aléatoires et des stratégiques, et sont ainsi reconnus comme de puissants outils pour modéliser les interactions entre un contrôleur et les réponses aléatoires de l’environment. Mathématiquement, un MDP peut être vu comme un jeu stochastique à un joueur et demi où le contrôleur choisit à chaque tour une action et l’environment répond en choisissant un successeur selon une distribution de probabilitésfixée. Il existe deux représentations incomparables du comportement d’un MDP une fois la stratégiefixée. Dans la représentation classique, un MDP est un générateur de séquences d’états, appelées state-outcome ; les conditions gagnantes du joueur sont ainsi exprimées comme des ensembles de séquences désirables d’états qui sont visités pendant le jeu, e.g. les conditions de Borel. La complexité des problèmes de décision et de mémoire requise des stratégies gagnantes pour les conditions dites state-outcome ont été langement étudiées. Depuis peu, les MDPs sont aussi considérés comme des générateurs de séquences de distributions de probabilités sur les états, appelées distribution-outcome. Nous introdui- sons des conditions de synchronisation sur les distributions-outcome, qui intuitivement demandent à ce que la masse de probabilité s’accumule dans un (ensemble d’) état, po- tentiellement de façon asymptotique. Une distribution de probabilités est p-synchrone si la masse de probabilité est d’au moins p dans un état ; et la séquence de distributions de probabilités est toujours, éventuellement, faiblement, ou fortement p-synchrone si, respec- tivement toutes, au moins une, une infinité ou toutes sauf un nombrefini de distributions dans la séquence sont p-synchrones. Pour chaque type de synchronisation, un MDP peut être (i) sûrement gagnant si il existe une stratégie qui génère une séquence1-synchrone ; (ii) presque-sûrement gagnant si il existe une stratégie qui génère une séquence (1−ϵ)- synchrone et cela pour toutϵ>0 ; (iii) asymptotiquement gagnant si pour toutϵ>0, il existe une stratégie produisant une séquence (1−ϵ)-synchrone. Nous considérons le pro- blème consistant à décider si un MDP est gagnant, pour chaque type de synchronisation et chaque mode gagnant : nous établissons les bornes supérieures et inférieures de la com- plexité de ces problèmes et de la mémoire requise pour une stratégie gagnante optimale. En outre, nous étudions les problèmes de synchronisation pour les automates probabilistes (PAs) qui sont en fait des instances de MDP où les contrôleurs sont restreints à utiliser uniquement des stratégies-mots ; c.à.d. qu’ils ne peuvent pas observer l’historique de l’exécution du système et ne peuvent connaître que le nombre de choix effectués. Les langages synchrones d’un PA sont donc l’ensemble des stratégies-mots synchrones : nous établissons la complexité des problèmes des langages synchrones vides pour chaque mode gagnant. Nous répercutons nos résultats de synchronisation obtenus sur les MDPs et PAs aux jeux à tours à deux joueurs ainsi qu’aux automatesfinis non-déterministes. Nous éta- blissons de nouveaux résultats de complexité sur les automatesfinis alternants avec des alphabets à une lettre. Enfin, nous étudions plusieurs variations de synchronisation sur deux instances de systèmes infinis que sont les automates temporisés et pondérés.

v vi Contents

1 Introduction 1 1.1 Formal veriﬁcation of reactive systems ...... 4 1.2 Probabilistic systems ...... 5 1.2.1 State-outcome and distribution-outcome views ...... 6 1.2.2 Analysis of state-outcomes in Markov chains ...... 7 1.2.3 Analysis of distribution-outcomes in Markov chains ...... 8 1.2.4 Analysis of state-outcomes in MDPs ...... 8 1.2.5 Analysis of distribution-outcomes in MDPs ...... 9 1.3 Synchronizing problems in MDPs ...... 9 1.3.1 Relations to games and to alternatingﬁnite automata ...... 12 1.4 Synchronizing problems in probabilistic automata ...... 12 1.5 Synchronizing problems in timed and weighted automata ...... 14 1.6 Structure of this manuscript ...... 14

2 Preliminaries 17 2.1 Finite automata and words ...... 18 2.2 Markov decision processes and strategies ...... 20 2.2.1 Markov decision processes ...... 21 2.2.2 Markov chains ...... 22 2.2.3 Strategies ...... 23 2.3 Probabilistic Automata and randomized word ...... 25 2.4 Winning and acceptance conditions ...... 26 2.4.1 Languages of NFAs ...... 27 2.4.2 Path-outcomes of MDPs ...... 28 2.4.3 Languages of PAs ...... 29 2.5 Decision Problems ...... 29 2.5.1 Problems for NFAs ...... 29 2.5.2 Problems for MDPs ...... 30 2.5.3 Problems for PAs ...... 34

3 Synchronizing Problems 37 3.1 Synchronization in NFAs ...... 38 3.1.1 Finite synchronization in NFAs ...... 38 3.1.2 Inﬁnite synchronization in NFAs ...... 42

vii 3.2 Synchronization in MDPs ...... 45 3.3 Synchronization in PAs ...... 51 3.4 Relation to one-letter alternatingﬁnite automata ...... 52 3.4.1 Problems for 1L-AFA ...... 54 3.4.2 Connection with MDPs...... 56 3.5 Relation to two-player turn-based games...... 58 3.6 Discussions ...... 59

4 Always and Eventually Synchronizing Condition 63 4.1 Always synchronizing condition in MDPs and PAs ...... 64 4.2 Eventually synchronizing condition in MDPs ...... 66 4.2.1 Sure eventually synchronization ...... 67 4.2.2 Almost-sure eventually synchronization ...... 69 4.2.3 Limit-sure eventually synchronization ...... 73 4.3 Eventually synchronization in PAs ...... 80 4.3.1 Sure eventually synchronization ...... 81 4.3.2 Almost-sure eventually synchronization ...... 83 4.3.3 Limit-sure eventually synchronization ...... 85 4.4 Discussion ...... 85

5 Weakly Synchronizing Condition 89 5.1 Weak synchronization in MDPs ...... 90 5.1.1 Sure weak synchronization ...... 90 5.1.2 Almost-sure weak synchronization ...... 93 5.1.3 Limit-sure weak synchronization ...... 96 5.2 Weak synchronization in PAs ...... 103 5.2.1 Sure weak synchronization ...... 103 5.2.2 Almost-sure weak synchronization ...... 105 5.2.3 Limit-sure weak synchronization ...... 105 5.3 Discussion ...... 106

6 Strongly Synchronizing Condition 115 6.1 Strong synchronization in MDPs ...... 116 6.1.1 Strong synchronization with function max ...... 116 6.1.2 Strong synchronization with function sum ...... 122 6.2 Strong synchronization in PAs ...... 124 6.2.1 Sure strong synchronization with function max ...... 124 6.2.2 Almost-sure strong synchronization with function max ...... 126 6.2.3 Sure strong synchronization with function sum ...... 130 6.2.4 Almost-sure strong synchronization with function sum ...... 131 6.2.5 Limit-sure strong synchronization ...... 134 6.3 Discussion ...... 138

viii 7 Synchronization in Timed Automata 141 7.1 Preliminaries ...... 142 7.1.1 Timed automata and timed words ...... 142 7.1.2 Problems in TAs ...... 144 7.2 Synchronization in TA ...... 145 7.3 Synchronization in deterministic TAs ...... 146 7.4 Synchronization in non-deterministic TAs ...... 153

8 Synchronization in Weighted Automata 157 8.1 Preliminaries ...... 158 8.1.1 Weighted automata ...... 158 8.1.2 Minsky machine and vector addition systems with states ...... 159 8.2 Synchronization in WA ...... 160 8.3 Location-synchronization in deterministic WAs ...... 162 8.3.1 Location-synchronization under lower-bounded safety condition . . 162 8.3.2 Location-synchronization under general safety condition ...... 169

9 Conclusion 181

Bibliography 185

ix x Introduction1

“We are like dice thrown on the plains of destiny.”

- Rita Mae Brown

The last slice of a pizza is usually too tempting for a child to think of sharing with a sibling, this more delicious than all other slices. The parents’ solution to avoid possible arguments could be as easy as tossing a coin and letting fate decide. The two siblings learn what is the chance of winning that last slice: one half. If not this exact story, a familiar story is a player in the backgammon game 1 wishing for a double when getting mars; no matter how much faith he has, he knows within himself that there is only one-sixth of a chance for that wish to come true. Why is that belief vividly carved in each of our minds? Simply, we know that there are thirty six possible pairs of faces for two dice, the desired event of a double is a set of only six pairs, andﬁnally we assume that dice are fair, meaning that when rolling a dice, there is an equally likely chance for each face to come up. In other words, if one repeats the experiment of rolling a dice for a suﬃciently large number of rounds, each face will statistically show up in one-sixth of the rounds. Gambling casino chips on a random combination of outcomes in a game is an entertaining activity for players and a business for the house. The simplest popular gaming

1. One of the oldest board game; excavations in Shahr-e Sukhteh ("The Burnt City") in Iran have shown that the game existed there around 3000 BC. The game is played on a board with black and white checkers for two players. In each round, players in turn throw two dice and move the checkers on the board as many cells as the numbers shown on the dice. A player wins after taking out all her checkers from board. When the loser has no checkers out, he is mars and misses two points. A double is when both dice have the same face up. A player who rolls doubles plays the numbers on the faces twice.

1 a. General rules of transitions b. The Markov chain with few missing transitions ⟨i+1,0⟩ p p p forj<3 pi ⟨0,0⟩ 0 ⟨1,0⟩ 1 ⟨2,0⟩ 2 ··· ⟨i, j⟩ p0 1−p 0 p1 1−p 1 p2 1−p i ⟨i+1, j+1⟩ ⟨0,1⟩ ⟨1,1⟩ ⟨2,1⟩ ···

1−p 0 1−p 1 1−p 2 ⟨i+1,0⟩ ⟨0,2⟩ ··· ⟨1,2⟩ ··· ⟨2,2⟩ ··· pi ⟨i,3⟩ ⟨0,3⟩ ··· ⟨1,3⟩ ··· ⟨2,3⟩ ··· 1−p i ⟨i+1,1⟩

Figure 1.1: A Markov chain that models a simplified Futurity slot machine where a state⟨i, j⟩ indicates the internal mode with thei-component: the modes0 and2 are mapped to tight and mode1 is mapped to loose. From a state with modei, the chain moves to a state with modei+1 (mod 3). Thej-component of states indicates the number of consecutive losses. The probabilityp i is the probability for the player to win in modei. machines are slot machines that take coins and randomly bring up on their reels three symbols out of a set of symbols, and if the outcome consists of three identical symbols, the player wins the bet. The slot machines are usually designed to be slightly favorable to the casino while still letting players trust the fairness of the game. The Futurity slot machine designed by the Mills Novelty Company of Chicago in 1936 [EL10], for instance, has a special feature that is returning back all coins bet if the player loses consecutively for a certain number of times; this feature gives the player the impression of having more chances to win. However, there is another feature invisible to the player: the machine has a cyclic sequence of internal modes consisting of some loose and some tight modes. The machine is programmed to have an expected payoff in favor of the player in the loose mode, and in favor of the casino in the tight mode. The sequence of invisible internal modes for the machine has more tight modes, and so is to the advantage of the casino. To delicately hide this trick and at the same time ensure the profit of the casino in the long run, casino owners wish to analyze the functionality of slot machines before investment. To this aim, a slot machine can be modeled by a (discrete-time) Markov chain, that is a mathematical object to model systems following a sequence of linked events. In the model of a slot machine, the linked events are the consecutive rounds of plays with no win. A simplified version of a Futurity slot machine is graphically described in Figure 1.1. The Markov chain models a machine with the cyclic sequence of internal modes “tight, loose, tight” and the misleading feature of returning all coins after three consecutive losses. The states of the machine are drawn as boxes and labeled with two components⟨i, j⟩,

2 see Figure 1.1.b. Thei-component encodes the “internal mode” (i.e., the index of internal modes in the cyclic sequence) andj-component stores “the number of consecutive losses”. Transitions, drawn as arrows between states, encode the feeding of the machine with a coin and the resulting win or loss that changes the internal mode and may also change the number of consecutive losses. Thus, transitions move the chain from one state to another one. Figure 1.1.a depicts the general rules of transitions. Transitions are augmented with probabilities that indicate how likely the player is to win or lose at states of the chain. A transition starting in some state⟨i, j⟩ is augmented with probabilityp i if it encodes a win; it is thus redirected to⟨i+1,0⟩ where the number of consecutive losses is reset. Otherwise, such transition is augmented with probability1−p i and is redirected to⟨i+1, j+1⟩ to store one more loss. The machine must meet a set of requirements and conditions called a specification. For instance, abiding the law, slot machines must be programmed to pay out as winnings at least 75% of the money wagered by players, a safety condition. Most gamblersfind gambling in a casino entertaining as long as the loss is roughly comparable to the gain. Stock brokers though might not feel entertained when losing even a small amount of money in the equity markets. Since markets are usually unstable and subject to dramatic and sudden changes, optimal decisions must be made instantaneously in order to exchange, buy and sell, securities that increase their profits. To predict the sudden changes or predict the effect of a placed exchange in the market, traders must carefully monitor and analyze the history of all trades andfluctuations, as statistical experiments. Extracting information from such big amounts of data in a short time is error-prone and tiring for humans. On the other hand, enthusiastic human traders commonly rely for their decisions unconsciously on their instincts, in particular after getting exhausted. Automated traders or trading robots are pieces of software designed to perform various calculations and analyze large amounts of data quickly in order to make optimal decisions almost instantly and more precisely than human traders. To design a trading robot, the market is usually modeled as a mathematical object such as a Markov decision process, which is a generalization of Markov chains. In addition to the stochastic aspects of systems that Markov chains can model, Markov decision processes model a kind of non-determinism to show the variety of the robot’s choices. A Markov decision process consists of several states and transitions. States encode thefinancial status of the market such as the price of companies’ shares on the stock market. Transitions encode placing orders that may cause changes in the market, thus the current state of the market will be changed to another state. Transitions are augmented with labels 2 and probabilities: labels vary between different possible orders, and the probabilities are computed by statistical analysis. The probability of a transition from one state to a successor state is showing how likely the placed order changes the prices in the market into what is indicated in the successor. A robot trader is programmed to place electronic orders in each state of the model. The robot must meet some specification, e.g., the dealings of a robot of a trading company must never cause dropping the value of

2. It is the main diﬀerence between Markov chains and Markov decision processes: transitions in Markov chains are only augmented with probabilities, though they are augmented with both labels and probabilities in Markov decision process.

3 the shares of the company in the market (safety condition). Otherwise, the wealth of the company is automatically decreased and not to forget the bad reputation. The shareholders of a trading company would not gamble in replacing the human element with automatic traders without guarantees that the trading robots perform as well as advertised. Such guarantees ought to promise that the robots make optimal decisions to fulfill the specification. A way to ensure the correctness of the designed robot is testing, which is broadly used in commerce and industry. Testing usually consists of running the robot on artificial markets (or simulations of the market) and observing the outcomes of the robot’s decisions. Having a set of correct behaviors, testing thus checks whether the outcome of a simulation contains those correct behaviors. The drawback of the testing approach is its non-completeness, which may cause missing some unexpected situations where the decisions of the robot breach a safety condition and could cause the shareholders to lose some money. Another approach is formal verification, a technique that promises the absolute guarantee for the correctness of a system by providing a mathematical proof (if such a proof exists 3). System verification also comes to prominence outside thefinancial sphere. It becomes even more crucial when any failure even a small deviation from the required behavior for the system is life threatening. In December 2013, it was reported that thefirst implanted artificial heart stopped beating. This misfortune caused the death of a seventy-six years old man who had received the heart, and the disappointment of French surgeons as well as many patients in need of a heart transplant. The artificial heart was designed to beat for at leastfive years, though an unexpected circuit shortcut happened after 74 days and caused the dysfunction. A mathematical proof to verify the correctness of the artificial heart would have ideally prevented the previous failure, and hopefully all dysfunctions of the second implanted heart.

1.1 Formal veriﬁcation of reactive systems

Reactive systems, such as artificial organs and stock markets, are non-terminating systems that maintain an ongoing interaction with the environment rather than returning some value upon termination. In thefield of formal methods and verification of reactive systems, we abstract reactive systems with mathematical models such as transition systems, finite state automata, Markov chains, Markov decision processes, etc., in order to take out some complexities in the system while retaining enough information to verify the correctness of the design. The behavior of a system is usually viewed as a sequence of interactions between the system and the environment; roughly speaking, the set of all behaviors is the language of the model. Computer-aided verification develops systematic approaches that verify whether all behaviors of the model correctly meet some required specification, that are expressed by mathematical objects such as some formula in some logic (e.g. a temporal logic such as LTL, CTL) or some regular (orω-regular) language or automata. The cor-

3. The veriﬁcation problem for some cases is undecidable.

4 rectness relation is usually chosen based on the model of the system and the mathematical object expressing the specification. For the case where both the system and the specification are modeled by automata, language inclusion ensures that all sequences of reactions in the system are desired behaviors in the specification language. Considering language inclusion as the correctness relation, the system behaviors are distinctly partitioned into “correct” and “incorrect“ ones, and the verification seeks to guarantee that all behaviors are correct without any nuance. This Boolean and qualitative view on the correctness is classical and is well-suited for modelling systems where no fault can be tolerated, such as artificial organs. The system model and the specification can be augmented with some quantitative aspects to be suitable for real-life applications. Real-time systems as an example are effec- tively modeled by timed automata where the model is enriched with some clocks and the transitions arefired with respect to time constraints. Specifications, thus, in this setting may express time operational deadlines or constrain the longest response time [AD94]. The battery consumption of mobile systems is described by weighted automata where transitions carry weights to reflect the amount of power recharged or discharged on taking each transition, and the evolution of system is subject to constraints on the accumulated weight (that is the power level). Specifications may then describe the power-life resistance in different scenarios [FJLS11]. As we have seen in systems such as stock markets, probabilities are assigned to possible responses or reactions in an abstracted model of the system in order to statistically predict the behavior of such complex systems; probabilistic automata and Markov models are well-established syntaxes for describing such systems [dA97, BK08]. Despite quantitative aspects in the system models and specifications, the correctness may follow the Boolean view, and impose a yes/no answer to the verification problem; the quantitative verification though seeks a graduated correctness for systems.

1.2 Probabilistic systems

Markov chains and Markov decision processes (MDPs) are widely used in the quantitative and qualitative verification of probabilistic systems [Var85, CY95, FV97, dA97, BK08]. Discrete-time Markov chains (or simply Markov chains) are arguably most popular model for verifying the evolution of a self-sufficient random process that has no interaction with its environment. As an extension, MDPs are a well-established syntax for open systems in which interactions between a controller and its stochastic environment happen. In MDPs, both non-deterministic and probabilistic choices thus coexist. We have seen how these Markov models can be used to model casino machines and stock markets. There are also applications in planning, randomized algorithms, communication protocols and biology [AH90, FP06, BBS06, KNP08]. The model-checking problem seeks the verification of a given property which can be expressed by qualitative or quantitative properties. The controller synthesis problem asks for computing a control strategy that ensures a specification, and thus is relevant only for MDPs (and not for Markov chains).

5 1.2.1 State-outcome and distribution-outcome views There are two incomparable views on Markov models: state-based and distribution-based views.

Markov chains. In the state-based view, Markov chains are treated as transition systems with states to which aﬁxed probability distribution, that determines the successor states, is assigned. The transition thus happens from one state to another, and a Markov chain is then a generator of sequence of states, called the behavior or the state-outcome of the Markov chain. In the distribution-based view, Markov chains are treated as transformers of probabilistic distributions over the nodes of the chains. A Markov chain can thus be encoded in a deterministic transition system whose states are probability distributions from which there is a unique outgoing transition to the successor state, i.e., the successor probability distribution. The transition thus happens from one probability distribution to another, and a Markov chain is then a generator of probability distributions, called the distribution- outcome or symbolic-outcome of the Markov chain. The distribution-outcome is an inﬁ- nite sequence of probability distributions obtained by iteratively applying the probabilistic transition matrix of the chain to the unique initial distribution (over nodes of the chain).

MDPs. The controller in an MDP can be instructed by a strategy (sometimes called scheduler [BK08]). When a strategy isﬁxed in an MDP, the system induced under the strategy is a Markov chain, and as expected the two views can carry over. The MDP under aﬁxed strategy thus can be viewed as a state-outcome or distribution-outcome generator.

Comparison of state-outcomes and distribution-outcomes. The state-outcome of stochastic systems is classical and well-studied, for example see [Var85, CY95, FV97, BK98a, BK98b, dAH00, KNP05, BKHW05, BCH+07, BK08, EKVY08, CD11, BHK+12, BGB12, CJS12, CHJS13]. Typical state-based properties for Markov models are safety, reachability, Büchi, and coBüchi, which require the system to visit a target state always, once, infinitely often, and ultimately always, respectively. The quantitative and qualitative analysis of such state-based properties and model checking of probabilistic temporal logics such as PCTL, PCTL∗ has been well-studied for Markov models. The distribution-outcomes of stochastic systems has recently been used in several works on design and verification of reactive systems [KVAK10, CKV+11, AAGT12]. This semantics is adequate in many applications, such as systems biology, sensor networks, robot planning, etc. [BBMR08, DMS14a, HMW09]. Such system consist of several copies of the same process (molecules, sensors, robots, etc.), and the relevant information along the execution of the system is the number of processes in each state, or the relative frequency (i.e., the probability) of each state. The logics defined on the two semantics, state-outcome and distribution-outcome, are incomparable, i.e., there are properties that are expressible in one but not the other. In particular, the approaches to decide model-checking problem in one cannot be reduced to

6 the other [KVAK10]. In one hand, intuitively, logics like PCTL and PCTL∗ do not allow for reasoning about the probability of being at diﬀerent (groups of) states at the same time, whereas we will see an example of such properties expressed on the distribution- outcome. On the other hand, properties expressible with the distribution-based semantics do not take into account the branching structure of the system. In addition, for probabilistic systems a natural bisimulation relation on the distribution-outcome has recently been introduced [HKK14], and has been proved to be incomparable with the standard bisimulation relation (deﬁned on the state-outcome).

1.2.2 Analysis of state-outcomes in Markov chains In the state-based view, a Markov chain is viewed as a generator of sequences of states. Aσ-algebra can be deﬁned on the probability space produced by the sequence of states, and the probability of events that are sets of sequences of states, can be measured. Prob- abilistic linear time properties and probabilistic temporal logics such as PCTL, PCTL∗ assert quantitative or qualitative properties as the speciﬁcation. Verifying a qualitative property that usually asks whether a certain event will happen almost-surely (i.e., with probability1) or with non-zero probability, is called the qualitative analysis of Markov chains. An instance of a qualitative property is the PCTL formula

Pr=1(!♦⟨0,3⟩)∧ Pr =1(!♦⟨1,3⟩)∧ Pr =1(!♦⟨2,3⟩)

where Pr=1 requires that the events!♦⟨i,3⟩ happens with probability1,!♦⟨i,3⟩ requires that the state⟨i,3⟩ is inﬁnitely often visited along the system execution. This formula asserts the following almost-surely property for the model of Futurity slot machine drawn in Figure 1.1 on page 2:

“all states encoding the three consecutive losses in the game, are almost-surely visited inﬁnitely often.”

Quantitative analysis of Markov chains is similar to model-checking problem of qualitative properties, but constrain the probability or expectation of certain events with regards to any value, not only the extreme bounds 4. An instance of a quantitative property is ! " Pr<0.05 (⟨0,3⟩ ∨ ⟨1,3⟩ ∨ ⟨2,3⟩)→⃝⃝⃝(⟨0,3⟩ ∨ ⟨1,3⟩ ∨ ⟨2,3⟩) where⃝ is the temporal operator to assert a property on the next state visited along the system execution. This formula asserts the following quantitative property for the model of Futurity slot machine drawn in Figure 1.1 on page 2:

“the scenario where a player loses three times consecutively and gets all his coins returned, must only happen two times in a row with the probability less than0.05.”

4. In qualitative analysis, the only allowed probability bounds are the extreme bounds, one and zero: Pr=1 (almost-surely) and Pr>0 (non-zero).

7 In order to model-check qualitative state-based properties, graph-based techniques are employed [BK08]. However, the quantitative analysis usually is reduced to solving a system of linear equations.

1.2.3 Analysis of distribution-outcomes in Markov chains In the distribution-based view, a Markov chain is viewed as a generator of sequence of probability distributions. The speciﬁcation can thus assert properties on the current probability of a set of nodes that is not expressible in the state-based logics. An instance of such a property is

∃t such thatX t(q)≥1 whereX is, roughly speaking, the chance for the Markov chain to be inq at timet. The above formula asserts that

“there is a timet when the Markov chain is almost surely in the stateq.”

Qualitative and quantitative analysis diﬀer in the distribution-based view like they do in the state-based semantics. Model checking of qualitative limit distribution-based properties in Markov chains usually relies on the computation of stationary distributions and period- icity of the chains, which has been well-studied, e.g. see [KS83, Nor98, Ser13]. In recent works, the veriﬁcation of quantitative properties of the distribution-based semantics was shown undecidable [KVAK10]. Decidability is obtained for special subclasses [CKV+11], or through approximations in which distributions are discretized using intervals [AAGT12].

1.2.4 Analysis of state-outcomes in MDPs In the state-based semantics, when resolving non-determinism by strategies, there can be different probability spaces in an MDP, and thus the probability of a certain event might be measured unequally [Var85]. The quantitative analysis of an MDP against a specification, expressed for example by PCTL∗ orω-regular properties, amounts to calcu- lating the minimal and maximal probabilities that can be guaranteed when ranging over all strategies. The qualitative analysis simply asks whether there exists a strategy ensuring that the specification holds almost-surely (with probability1) or limit-surely (with probability arbitrary close to1); and the synthesis problem seeks the computation of such winning strategies. The calculation of minimal and maximal probabilities is well-studied and is usually achieved by graph algorithms operating on the graph underlying the MDP accompanied with some linear constraints. The obtained linear programs can be solved by means of an iterative approximation algorithm (called value iteration) [BK08]. For tra- ditionalω-regular properties, in particular safety and reachability conditions, memoryless strategies are sufficient as optimal strategies [CH12]. To solve the synthesis problem, such optimal strategies can be constructed by some complementary analysis while computing the probability bounds for model checking.

8 1.2.5 Analysis of distribution-outcomes in MDPs Compared to the state-outcome in MDPs, there has been little work on the analysis of the distribution-outcome view. Since the Skolem problem, which has been open for over eighty years, can be reduced 5 to the quantitative analysis, studying the qualitative analysis of the distribution-based semantics is more promising. There is only one logic defined on the distribution-outcomes in MDPs. The logic is defined by propositions using linear inequalities over probability distributions, and using modal operators to reason about temporal behavior [KVAK10]. The verification of the defined logic is in general undecidable even for a restricted class of strategies that determine their choice of actions based on the current state of the system and the number of choices made so far, called counting strategies (or Markovian schedulers). The decidability results are only obtained by imposing some restrictions either on the class of counting strategies, or on the propositions (where the inequality in the proposition is non-zero). As we will see, the main results of this thesis lie in the direction of the qualitative analysis of the distribution-outcomes in MDPs. We introduce different variants of synchronizing properties on distribution-outcomes of MDPs. In contrast with previous work, we consider the qualitative analysis with the extreme bound1, and we prove that synchronizing properties are decidable for a general class of strategies that select actions depending on the full history of the system execution, called perfect-information strategies.

1.3 Synchronizing problems in MDPs

The main contribution of this thesis is a novel approach to the qualitative analysis and the synthesis problem of the distribution-outcomes in MDPs. We introduce synchronizing properties defined on the distribution-outcomes, which require that the winning strategy brings the MDP in some (group of) state(s) with a large probability, possibly in limit. In other words, the probability mass (say at least probabilityp>0) in the distribution, that indicates the likelihood of the system to be in different states at specific states, is accumulated in some target state. We define variants ofp-synchronizing conditions on the symbolic-outcomes of an MDP, and study the qualitative analysis of such properties where p=1 orp tends to1. We consider the symbolic-outcome of the behaviors of MDPs as sequencesX 0X1X2 ··· of probability distributionsX i :Q→ [0, 1] over thefinite state spaceQ of the system, whereX i(q) is the probability that the MDP is in stateq∈Q afteri steps. For0≤ p≤1, a distribution isp-synchronized if the probability mass accumulated in a single state (or a group of states) is at leastp. An MDP is always (resp., eventually, weakly or strongly)p-synchronizing if the probabilityp is always (resp., ultimately once, infinitely often, or ultimately at every step) synchronized along the execution. More precisely, a sequenceX 0X1 ··· of probability distributions is

5. The results also holds for Markov chains, where the quantitative veriﬁcation of the distribution- outcome semantics is in general already undecidable.

9 (a) alwaysp-synchronizing ifX i isp-synchronized for alli;

(b) eventuallyp-synchronizing ifX i isp-synchronized for somei;

(d) stronglyp-synchronizing ifX i isp-synchronized for all butfinitely manyi’s. An MDP is thus eventually, always, weakly or stronglyp-synchronizing if there exists some strategy whose symbolic-outcome is accordingly eventually, always, weakly or stronglyp-synchronizing. It is easy to see that alwaysp-synchronizing implies strongly p-synchronizing, which implies weaklyp-synchronizing. Moreover, weaklyp-synchronizing implies eventuallyp-synchronizing. We show that, in general, this hierarchy is strict. We study the qualitative analysis of synchronizing conditions in such quantitative models. The qualitative synchronizing properties, corresponding to the case where eitherp=1 orp tends to1 are analogous to the traditional safety, reachability, Büchi, and coBüchi conditions (in the state-based semantics). We consider the following qualitative (winning) modes: (i) sure winning, if there is a strategy that generates an {always, eventually, weakly, strongly}1-synchronizing sequence; (ii) almost-sure winning, if there is a strategy that generates a sequence that is, for all ϵ>0, {always, eventually, weakly, strongly} (1−ϵ)-synchronizing; (iii) limit-sure winning, if for allϵ>0, there is a strategy that generates a {always, eventually, weakly, strongly} (1−ϵ)-synchronizing sequence. The definitions are summarized in Table 3.1 on page 47. These three winning modes 1 are classically considered in stochastic games; MDPs can be viewed as1 2 -player stochastic 1 games. Some games in casinos can be analyzed with such1 2 -player stochastic games. Until recently, slot machines were not elaborate enough to have any human interaction other than feeding the machine with the betting coins. However, with microprocessors nowadays ubiquitous, a more interactive gambling game has become widely popular, that is video slot machines. This game can in fact be viewed as a computer game where players interact only via a touch screen and where there is no mechanical constraints. It allows the designers to display more virtual reels and also add some interaction with the player through bonus features. For instance, when the player hits some specific combination on the reels, he may choose to either obtain free spins or hold some reels while the others are re-spun, or bet for another prize, etc. Due to this human interaction in the bonus features, a video slot machine cannot be modeled anymore by a Markov chain but requires an MDP where the random responses of the environment is predicted by a statistical study on human-player psychology. The aim is to synthesize a controller microprocessor for slot machines such that the specifications are met. Thus, the machine is modeled by an MDP, the choices of the microprocessor are the strategic choices and the human choices are resolved randomly based on the probability distributions obtained by statistical experiments. Moreover, having a distributed network of connected slot machines, designers may offer a new exciting large prize, usually called “jackpot”. For instance, in a group of connected machines, the jackpot can be triggered when all connected slot machines bring the same

10 combination of symbols on their reels at the same time. The exciting news is that when jackpot happens, every player wins the total amount bet by all players. When analyzing the MDP of the distributed machines, the global state represent the internal states of all distributed machines, and the jackpot is a kind of synchronization when the MDP is synchronized into the set of states representing identical symbols on the reels of all machines. The aim of the designer is usually to synthesize an optimal strategy that allows for such jackpots infinitely often, and only when the amount of the total bet is smaller than a threshold. In this thesis, we study the introduced synchronizing conditions in MDPs. We establish the computational complexity of deciding whether a given MDP is always, eventually, weakly or strongly synchronizing by providing matching upper and lower complexity bounds: for each winning mode we show that the problems are in PTIME for the always synchronizing condition, PSPACE-complete for eventually and weak synchronization, and PTIME-complete for strong synchronization. Moreover, we prove that the three winning modes coincide for the always synchronizing condition, though those modes form a strict hierarchy for eventually synchronizing. In particular, there are limit-sure winning MDPs that are not almost-sure winning; this result is unlike the analogue reachability condition in the state-based semantics where the two almost-sure and limit-sure winning modes coincide [dAHK07]. One of the difficult result in this thesis is to show that for weak and strong synchronization the almost-sure and limit-sure modes coincide. It implies that the strong and weak synchronizing conditions are more robust than the eventually synchronizing, and provide conservative approximations of the eventually synchronizing. We complete the picture by providing optimal memory bounds for winning strategies. We will see that all solutions, for the problems of deciding whether an MDP is synchronizing, are constructive, and provide the construction of the winning strategies that answer the synthesis problems. In always synchronizing MDPs, we show that memoryless strategies are sufficient for all three winning modes. In eventually synchronizing MDPs, for sure winning strategies, exponential memory is sufficient and may be necessary, and that in general infinite memory is necessary for almost-sure winning, and unbounded memory is necessary for limit-sure winning. Moreover, exponential memory is sufficient and may be necessary for sure winning in weak synchronization, infinite memory is necessary for almost-sure winning in weak synchronization, and linear memory is sufficient for strong synchronization in all winning modes. Regarding the symbolic-outcomes in MDPs, thep-synchronizing conditions defined on the outcome sequence are using a sum function, i.e., the sum of probabilities assigned to target states is at leastp. We present a variant of synchronization for which the used function is taking the maximum probability assigned to the set of target states. We also establish some reductions for eventually and weakly synchronization such that the tight complexity bounds for sum functions are carried over to the max variant. For strong synchronization, we see that deciding whether the MDP is winning is again PTIME-complete though memoryless strategies are sufficient.

11 The obtained results on synchronization in MDPs are summarized in Tables 4.1, 5.1 and 6.1 on pages 67, 90 and 116, respectively.

1.3.1 Relations to games and to alternatingfinite automata Some results in this thesis rely on insights related to games and alternating automata that are of independent interest. First, the sure-winning problem for eventually synchronizing in MDPS is equivalent to a two-player game with a synchronized reachability condition, where the goal for thefirst player is to ensure that a target state is reached after a number of steps that is independent of the strategy of the opponent (and thus this number can befixed in advance by thefirst player). This condition is stronger than plain reachability, and while the winner in two-player reachability games can be decided in polynomial time, deciding the winner for synchronized reachability is PSPACE-complete. This result is obtained by turning the synchronized reachability game into a one-letter alternating automaton for which the emptiness problem (i.e., deciding if there exists a word accepted by the automaton) is PSPACE-complete [Hol95, JS07]. Second, the PSPACE lower bound for the limit-sure winning problem in eventually synchronizing uses a PSPACE-completeness result that we establish for the universalfiniteness problem, which is to decide, given a one-letter alternating automata, whether from every state the accepted language isfinite.

1.4 Synchronizing problems in probabilistic automata

Synchronizing problems werefirst considered forfinite state automata where a synchronizing word is afinite sequence of control actions that can be executed from any unknown or unobservable state of an automaton and brings the automaton to a known specific state (see [Vol08] for a survey of results and applications). The notion of synchronizing automata is motivated by the following natural problem: how can we regain control over a device if we do not know its current state? Since losing the control over a device may happen due to missing the observation on the outputs produced by the system, blind strategies, which arefinite sequences (or words) of input letters, are considered while synchronizing systems. As an example think of remote systems connected to a wireless controller that emits the command via wireless waves but expects the observations via physical connectors (it might be excessively expensive to mount wireless senders on the remote systems), and consider that the physical connection to controller is lost due to technical failure. The wireless controller can therefore not observe the current states of the distributed subsystems. In this setting, emitting a synchronizing word as the command leaves the remote system (as a whole) in one particular state no matter in which state each distributed subsystem started; and thus the controller can again regain control. Synchronizing automata are well-studied in the setting of complete deterministicfinite- state automata [Čer64, Pin78, Epp90, IS95, IS99, San04, Vol08, Mar10, AGV10, OU10, Mar12]. While the existence of a synchronizing word is NLOGSPACE-complete for complete deterministicfinite state automata, extensive research efforts have been devoted to estab-

12 lishing tight bounds on the length of the shortest synchronizing word, which is conjectured to be(n−1) 2 for automata withn states [Čer64, Pin78, Vol08]. Various extensions of the notion of synchronizing word have been proposed for not-complete or non-deterministicfinite state automata and were proved to be PSPACE-complete [IS95, IS99, Mar10, Mar12]. We introduce infinite synchronizing words for non-deterministicfinite state automata, where the always, eventually, weakly and strongly synchronization require that the automaton is always, once, infinitely often and ultimately always synchronized (into a given target set).

Probabilistic automata (PAs) are a generalization of non-deterministicfinite state automata where the non-deterministic choice of successors is resolved with a probabilistic distribution; at the same time, a PA can also be interpreted as an MDP with a blind controller. In this context, an input word for the PA corresponds to the special case of a blind strategy (sometimes called word-strategy) that chooses the control actions in advance. In the example of remote systems, a blind strategy cannot depend on the current states of the distributed subsystems, but only depend on the sequence of wireless commands emitted so far. The set of all winning word-strategies for an always, eventually, weakly and strongly synchronizing PA is respectively the always, eventually, weakly and strongly synchronizing language of the PA. Similar to the MDPs, we study three qualitative modes, sure, almost-sure and limit-sure for synchronizing languages in PAs. A different definition of synchronizing words for probabilistic automata was proposed by Kfoury, but the associated decision problem is undecidable [Kfo70]. In contrast, we show that the emptiness problem of almost-sure strongly synchronizing languages is PSPACE- complete, while the emptiness problem of almost-sure eventually and weakly synchronizing languages are undecidable. Moreover, we show that the emptiness problem of sure languages, for all three eventually, weakly and strongly synchronizations, are PSPACE- complete, though the emptiness problem of limit-sure languages are undecidable. There are easy arguments to establish the complexity bounds for the always synchronizing languages. The obtained results are summarized in Tables 4.2, 5.2 and 6.2 on pages 81, 103 and 124, respectively. We also shortly discuss the universality problems for the synchronizing languages in PAs: we show that the universality problems of sure and almost-sure 6 always synchronizing languages are in PTIME. The universality problems of sure eventually, weakly and strongly synchronizing languages are in PSPACE, while for almost-sure eventually, weakly and strongly synchronizing languages, the universality problems are PSPACE-complete. In addition, we show that the results from synchronizing in PAs can carry over to the non- deterministicfinite state automata. 6. The universality problem of synchronizing languages in PAs cannot be considered for limit-sure winning mode.

13 1.5 Synchronizing problems in timed and weighted automata

A further contribution of this thesis is introducing synchronization in systems whose behaviors depend on quantitative constraints. We study two classes of such systems, timed automata and weighted automata. We introduce the synchronization problem for timed and weighted automata in several variants to include the quantitative aspects of those models as well as some safety condition while synchronizing. The main challenge is that we are facing automata with inﬁnite state-spaces and inﬁnite branching (e.g. delays in a timed automaton).

For timed automata the synchronization problems are shown to be PSPACE-complete in the deterministic case, and undecidable in the non-deterministic case. The obtained results are summarized in Table 7.1 on 142. In the deterministic case, synchronizing the classical region abstraction is not sound. We propose an algorithm which reduces the (uncountably) infinite set of configurations into afinite set (with at most the number of locations in the automaton), and then pairwise synchronizes the obtainedfinite set of states.

For weighted automata, states are composed of locations and quantitative values as weights. As weights are merely accumulated in this setting, it is impossible to synchronize to a single state. Instead we search for a location-synchronizing word, that is a word after which all states will agree on the location. In addition, we add a safety condition insisting that during synchronization the accumulated weight (energy) is safe, e.g. a non- negative safety condition (or energy constraints) that requires a system to never run out of power while synchronizing. Considering the safety condition is what distinguishes our setting from the one presented in [FV13]; moreover, in that work weighted automata are restricted to only have non-negative weights on transitions. For deterministic weighted automata, we provide PSPACE-completeness of the synchronization problem under energy constraints, and the membership in 3-EXPSPACE under general safety constraints. The obtained results are summarized in Table 8.1 on 158.

1.6 Structure of this manuscript

This thesis is organized as follows.

Chapter 2 Preliminaries. We introduce deﬁnitions needed throughout the thesis such as MDPs and PAs. We also recall some classical problems and results on such models.

The results of the following four chapters on synchronization in MDPs and PAs are mostly disseminated in these publications [DMS11a, DMS11b, DMS12, DMS14a, DMS14b]:

14 Chapter 3 Synchronizing Problems. We recall the definition offinite synchronizing words forfinite state automata, and survey decision problems and conjectures about such words. We provide definitions of infinite synchronizing words for non-deterministicfinite state automata. We introduce variants of synchronizations in MDPs and PAs by considering synchronizing strategies and synchronizing words. The relations to two-player games and to one-letter alternatingfinite automata are also discussed.

Chapter 4, Chapter 5 and Chapter 6 Always and Eventually Synchronizing Con- dition, Weakly Synchronizing Condition and Strongly Synchronizing Condition. We show tight complexity bounds of the membership problem for always, eventually, weakly and strongly synchronizing conditions in MDPs, and provide the memory requirement of winning strategies. Moreover, we establish tight complexity bounds for the emptiness problem of all synchronizing languages for PAs while the universality problem is also shortly discussed.

The results of the following two chapters on synchronization in timed and weighted automata has been published in [DJL+14]:

Chapter 7 Synchronization in Timed Automata. We introduce synchronizing and location-synchronizing words for timed automata. We focus on deciding the existence of a synchronizing and location-synchronizing word for timed automata, proving tight complexity bounds of the problems for deterministic timed automata, and proving undecidability for non-deterministic timed automata.

Chapter 8 Synchronization in Weighted Automata. We introduce synchronizing and location-synchronizing words for weighted automata. We deﬁne safe synchronization for weighted automata where the aim is to synchronize the set of safe states while forbidding the automaton to visit states outside the safety-set during synchronization. We establish complexity results for deciding the existence of synchronizing and location-synchronizing words for weighted automata under diﬀerent kind of safety conditions.

Chapter 9 Conclusion.

Selected publications by the author Most of the contributions in this manuscript have been already published in international conferences. The following publications partially contain the results of this thesis: [DJL+14] Laurent Doyen, Line Juhl, Kim G. Larsen, Nicolas Markey, and Mahsa Shir- mohammadi. Synchronizing words for weighted and timed automata. In Proceed- ings of the 34th Conference on Foundations of Software Technology and Theoretical Computer Science, FSTTCS 2014, Leibniz International Proceedings in Informatics. Leibniz-Zentrum für Informatik, 2014. To appear.

15 [DMS14b] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Robust synchronization in Markov decision processes. In CONCUR 2014 - Concurrency Theory - 25th International Conference, CONCUR 2014, Rome, Italy, September 2-5, 2014. Proceed- ings, volume 8704 of Lecture Notes in Computer Science, pages 234–248. Springer, 2014. [DMS14a] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Limit synchronization in Markov decision processes. In Foundations of Software Science and Com- putation Structures - 17th International Conference, FOSSACS 2014, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2014, Grenoble, France, April 5-13, 2014, Proceedings, volume 8412 of Lecture Notes in Computer Science, pages 58–72. Springer, 2014. [DMS12] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Inﬁnite synchronizing words for probabilistic automata (erratum). CoRR, abs/1206.0995, 2012. [DMS11a] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Inﬁnite synchronizing words for probabilistic automata. In Mathematical Foundations of Computer Science 2011 - 36th International Symposium, MFCS 2011, Warsaw, Poland, August 22-26, 2011. Proceedings, volume 6907 of Lecture Notes in Computer Science, pages 278–289. Springer, 2011. [DMS11b] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Synchronizing objectives for Markov decision processes. In Proceedings International Workshop on Interactions, Games and Protocols, iWIGP 2011, Saarbrücken, Germany, 27th March 2011., volume 50 of EPTCS, pages 61–75, 2011.

16 Preliminaries2

First sight. In this chapter we introduce definitions and recall results needed throughout the thesis. Wefirst introducefinite automata, Markov decision processes and probabilistic automata. We then mention classical acceptance and winning conditions to define languages and winning regions. We conclude the chapter with the last section devoted to the classical problems and results for each model.

Contents 2.1 Finite automata and words ...... 18 2.2 Markov decision processes and strategies ...... 20 2.2.1 Markov decision processes ...... 21 2.2.2 Markov chains ...... 22 2.2.3 Strategies ...... 23 2.3 Probabilistic Automata and randomized word ...... 25 2.4 Winning and acceptance conditions ...... 26 2.4.1 Languages of NFAs ...... 27 2.4.2 Path-outcomes of MDPs ...... 28 2.4.3 Languages of PAs ...... 29 2.5 Decision Problems ...... 29 2.5.1 Problems for NFAs ...... 29 2.5.2 Problems for MDPs ...... 30 2.5.3 Problems for PAs ...... 34

17 def We denote byZ the set of integer numbers and byN ≡{n≥0|n∈Z} the set of def natural numbers. We denote byR the set of real numbers and byR ≥0 ≡{x≥0|x∈R} the set of non-negative real numbers. We assume that the reader is familiar with the basic concepts of sets, relations, graphs theories and automata theory [Ros93, Die12, Sip97].

2.1 Finite automata and words

Finite automata are defined overfinite alphabets to accept input words which are sequences of concatenated letters of the alphabet. Such words could befinite, such as w1 =a·b·a·b·b, or could be infinite such asw 2 =a·b·b·b··· wherea andb are some input letters. Given an alphabetA, we denote byA ∗ the set of allfinite words over this alphabet, and byA ω the set of all infinite words. The length of afinite wordw∈A ∗ is denoted by|w|; for example|w 3|=3 for the prefixw 3 =a·b·b ofw 2. The empty wordϵ is of length zero, andA + is the set of all non-emptyfinite words.

Regular andω-regular languages. A language offinite words over the alphabetA is a subset ofA ∗ and a language of infinite words is a subset ofA ω. A language (offinite words) is regular if there is a regular expression to express it. Given afinite alphabetA, the following constants are defined as regular expressions: –∅ denotes the empty language, –ϵ denotes the language{ϵ} which contains only the empty wordϵ of length zero, –a∈A denotes the language{a} containing only the worda of length1. ∗ Given two regular expressionsr 1 andr 2, the expressionsr 1 +r 2,r 1 ·r 2 andr 1 are regular expressions where – (union operator)r 1 +r 2 denotes the language that is obtained by the union of languages expressed byr 1 andr 2, – (concatenation operator)r 1 ·r 2 denotes the language of all words made by concatenation of a word from the language expressed byr 1 followed by a word fromr 2, ∗ – (Kleene star operator)r 1 denotes the smallest language that includesϵ andr 1, and that is closed under concatenation operator. A language (of infinite words) isω-regular if there is anω-regular expression to express it. Given a regular expressionr expressing a regular language not containing the empty wordϵ, the expressionr ω that denotes a language of infinite words made by concatenating consecutively the words ofr infinitely many times, isω-regular. Given a regular expression r1 and twoω-regular expressionsr 2 andr 3, the following expressions areω-regular: – (concatenation operator)r 1 ·r 2 denotes the language of all infinite words made by concatenation of afinite word from the language expressed byr 1 followed by an infinite word fromr 2. – (union operator)r 2 +r 3 denotes the language that is obtained by the union of languages expressed byr 2 andr 3.

18 a b q0 q1 b

a, b a

q q 3 a 2 b

Figure 2.1: The NFA described in Example 2.2 with initial stateq 0.

Example 2.1. The regular expressiona·b ∗ ·a is expressing the language{a·b n ·a|n∈N} whereb n is the word consists ofn consecutiveb. Theω-regular expressiona·b ∗ ·a ω is expressing the language{a·b n ·a·a·a···|n∈N}. ▹

Finite automata.Aﬁnite automaton recognizes a regular or anω-regular language:

Definition 2.1 (Finite automata). A non-deterministicfinite automaton (NFA)N= ⟨Q,A,∆⟩ consists of afinite setQ of states, afinite alphabetA of input letters and a transition function∆:Q×A→2 Q.

Given a set of initial statesQ 0 and an input wordw=a 0 ·a 1 ··· , the behavior of an NFA is as follows. One of the initial statesq 0 ∈Q is chosen non-deterministically. The automaton starts in that initial stateq 0 and reads theﬁrst lettera 0 of the input wordw. Inputting the lettera at the time the automaton is inq (that is lettera 0 when it initially is inq 0), two cases happen: (1) either∆(q, a)≠ ∅ when the automaton moves to the next stateq ′, that is chosen non-deterministically among the states of∆(q, a);(2) or∆(q, a) =∅ when the automaton “fails” to move and becomes blocked. As long as the automaton is not blocked, in each move the next letter ofw=a 0a1 ··· , from left to right, is read. Example 2.2. An NFA is depicted in Figure 2.1, which is a running example used throughout the thesis. The automaton has four statesq 0, q1, q2 andq 3, and two lettersa andb. The transition function∆ is deﬁned as follows. For all states, thea-transitions shift once the automaton in the loopq 0q1q2q3; formally,

∆(qi, a) ={q j} wherej≡i + 1 (mod 4) for all0≤i≤3.

Theb-transitions in the statesq 0, q1, q2 are self-loops,∆(q i, b) ={q i} for all0≤i<3; the next successor onb-transition inq 3, however, isq 0. ▹ In Example 2.2, the transition function maps each pair of stateq and lettera only to sets of size at most1, that implies this automaton is in fact deterministic. A deterministic

19 finite automaton (DFA) is afinite automaton 1 where each stateq and input lettera have at most one successor state:|∆(q,a)|≤1. On the other hand, all states and letters of the automaton of Example 2.2 have at least one successor that means this automaton is complete too. A completefinite automaton is an automaton where the function∆(q, a) is non-empty for each stateq and input lettera. A stateq is absorbing if∆(q, a) ={q} for all lettersa∈A (all transitions are self-loops onq). a We denote byq −→q ′ ana-transition that goes fromq toq ′:q ′ ∈∆(q, a). Let the relation a →⊆Q×Q be such thatq→q ′ if there exists a lettera whereq −→q ′. We thus define the + i i transitive closure→ of this relation, that is∪ 1≤i → where→ is thei-th power of→ by the composition operation in relations. The notationq→ + q′ thus means that there exists afinite wordw=a 0 ···a n−1 such that after inputtingw, the automaton initiating inq ends to move inq ′. The wordw, actually, induces afinite run over the automaton; afinite ′ ai run is a sequence of statesq 0q1 ···q n whereq 0 =q,q n =q andq i −→q i+1 for all0≤i

2.2 Markov decision processes and strategies

Probability distribution.#A probability distribution over aﬁnite setS is a function d:S→ [0, 1] such that s∈S d(s) = 1. The support ofd is the set Supp(d) ={s∈ 1 S|d(s)>0}. For example, given the alphabetA={a, b}, the functiond(a) = 4 and 3 d(b) = 4 is a probability distribution overA; we usually denote probability distributions 1 2 over an alphabet withd. Another example is the functionX(q 1) = 3 ,X(q 2) = 3 and X(q3) = 0 that is a probability distributionX over setS={q 1, q2, q3} of states, where Supp(X)={q 1, q2}. In this thesis, we usually denote probability distributions over states withX. We denote byD(S) the set of all probability distributions# overS. Given a set T⊆S and a probability distributionX overS, letX(T)= s∈T X(s). 1 ForT≠ ∅, the uniform distribution onT assigns probability |T| to every state inT . For 1 example, the distributionX(q 1) =X(q 2) = 2 is a uniform distribution on the set{q 1, q2}.

1. A DFA is equipped with only one initial stateq 0.

20 1 a: 2 , b a a, b 1 a: 2 b a, b q0 q1 q2 q3

Figure 2.2: The MDP described in Example 2.3 with the initial Dirac distributionq 0.

Givens∈S, the Dirac distribution ons assigns probability1 tos, and by a slight abuse of notation, we denote it simply bys.

2.2.1 Markov decision processes Markov decision processes are a kind of stochastic games:

Definition 2.2 (Markov decision processes). A Markov decision process (MDP)M= ⟨Q,A,δ⟩ consists of afinite setQ of states, afinite setA of actions and a probabilistic transition functionδ:Q×A→D(Q).

Given an initial distributionX 0 ∈D(Q), the behavior of an MDP is described as a one- player stochastic game played in rounds. The game starts in the stateq with probability X0(q). In all rounds, the player chooses an actiona∈A, and if the game is in stateq, the next round starts in the successor stateq ′ with probabilityδ(q, a)(q ′).

Example 2.3. An MDP is depicted in Figure 2.2 where the states areQ={q 0, q1, q2, q3} and the alphabet isA={a, b}. The probabilistic transition functionδ is deﬁned as follows. Botha-transitions andb-transitions inq 1, q2, q3 are deterministic, that isδ(q i, a) andδ(q i, b) are Dirac distributions: for0

δ(q1, a) =q 1,δ(q 1, b) =q 2 andδ(q 2, c) =δ(q 3, c) =q 3 forc∈{a, b}.

The only probabilistic transition is thea-transition inq 0 whereδ(q 0, a) is a uniform distribution on the set{q 0, q1}. Theb-transition inq 0 is a deterministic self-loop:δ(q 0, b) =q 0. ▹

In Example 2.3, we say that the stateq 3 is absorbing sinceδ(q 3, c) is the Dirac distribution onq 3 for all actionsc∈A. Givenq∈Q anda∈A, denote by post(q,a) the set Supp(δ(q, a)) which contains all states that can be chosen with a positive probability as the next successor of thea-transition inq. In Example 2.3, post(q 0, a) ={q 0, q1}. LetS⊆Q ′ andA ⊆A. We denote by post(S,a) the set of successors∪ q∈S post(q,a) of all statesq in ′ S and the actiona; and similarly by post(S,A ) the set of all successors∪ a∈A′ post(S,a) of states inS and actions inA ′. GivenT⊆Q, let Pre(T)={q∈Q|∃a∈A: post(q, a)⊆T} be the set of states from which the player has an action to ensure that the successor state is inT . Fork>0, k k−1 0 let Pre (T)= Pre(Pre (T)) with Pre (T)=T . In Example 2.3, forT={q 0, q3} one

21 1

1 3 1 q1 3

q0 1 3 1 q2 q3 1

Figure 2.3: The Markov chain of Example 2.4 with the initial Dirac distributionq 0.

2 k can verify that Pre(T)={q 0, q2, q3} and Pre (T)={q 0, q1, q2, q3}. Moreover, Pre (T)= Pre2(T) for allk≥2 showing that the sequence of predecessors is ultimately periodic forT. Since the set of statesQ isfinite, the sequence of predecessors Pre 0(T), Pre 1(T), Pre 2(T),··· is, indeed, ultimately periodic for all setsT⊆Q. A play (or simply a path) in an MDPM is an infinite sequenceπ=q 0q1q2 ... such that for alli≥0 there exists an actiona such thatq i+1 ∈ post(qi, a).Afinite prefix ρ=q 0q1 . . . qn of a path has length|ρ|=n and last state Last(ρ) =q n. In Example 2.3, the ω playπ=q 0q1q2(q3) has the prefixρ=q 0q1q2 where|ρ|=2 and Last(ρ) =q 2. We denote by Plays(M) and Pref(M) the set of all plays andfinite paths inM, respectively.

2.2.2 Markov chains Aﬁnite-state Markov chain is a special kind of MDPs where the player’s choice has no role.

Definition 2.3 (Markov chains).Afinite-state Markov chainM=⟨Q,δ⟩ consists of afinite setQ of states and a probabilistic transition functionδ:Q→D(Q).

We see that the probabilityδ(q)(q ′) of moving from one stateq to the successorq ′ is fixed independently of the actions and the states that the Markov chain went through. The Markov chains defined in Definition 2.3 are sometimes referred as homogeneous (discrete time) Markov chains [KS83, Nor98, Ser13]. In this manuscript, we only consider the homogeneous Markov chains and thus omit the word homogeneous as a qualifier. Example 2.4. A Markov chains is depicted in Figure 2.3 where the states areQ= {q0, q1, q2, q3} and the probabilistic transition functionδ is defined as follows. The next 1 successor inq 0 is eitherq 0, q1 orq 2, each with probability 3 . The stateq 1 is an absorbing state:δ(q 1)(q1) = 1. The transitions inq 2 andq 3 are deterministic:δ(q 2)(q3) = 1 and δ(q3)(q2) = 1. ▹ Since Markov chains are a subset of MDPs, we analogously define paths andfinite prefixes here.

22 The states offinite Markov chains can be classified in recurrent and transient states; for that, we define a bidirectional reachability relation∼ on their state spaceQ: given the Markov chainM, we say thatq andq ′ are related, denoted byq∼q ′, if there are two finite paths inM, one fromq toq ′ and another fromq ′ toq. In Example 2.4, we see thatq 0 ∼q 0,q 2 ∼q 3 andq 1 ̸∼q 2. The relation∼ is an equivalence relation since it is reflexive, symmetric and transitive. We denote by[Q] ∼ the partition (equivalence classes) induced by the relation∼. For the Markov chain described in Example 2.4, the equivalence classes are{q 0},{q 1} and{q 2, q3}. Intuitively, each of these classes is a strongly connected component (SCC) in the digraph induced by the Markov chain. A stateq is recurrent if fromq there is no path to a state outside the class containingq; formally, a stateq∈c ′ ′ wherec∈[Q] ∼ is recurrent if there is nofinite path to a stateq whereq ̸∈c. Otherwise the stateq is transient. The definition results in the states of a class being all recurrent or all transient. The Markov chain of Example 2.4 has two recurrent classes{q 1} and{q 2, q3} and one transient class{q 0}. In the graph view, a recurrent class is a bottom SCC in the digraph induced by the Markov chain. Roughly speaking, from a recurrent stateq there is no possibility of going to a state q′ from which there can be no return toq. Therefore, if a Markov chain ever enters in the stateq of a recurrent classc, it always stays inc and eventually returns toq with probability1, and thus keeps returning inq infinitely often. To formalize this observation, letM be a Markov chain andX 0 be an initial distribution; for technical reason, we assume that for all recurrent statesq, there is a path from some initial stateq 0 ∈ Supp(X0) toq (otherwise we omit the recurrent stateq). For alln∈N, let the probability distributionM n over the states of the Markov chainM give the# probability of being in various states ′ ′ at stepn. Thus,M 0 =X 0 andM n+1(q) = q′∈Q Mn(q )·δ(q )(q) for alln∈N. Considering the sequenceM 0M1M2 ··· in a Markov chainM, a distributionX∈D(Q)

is stationary if there is an infinite sequencen 1n2n3 ··· such that limk→∞ Mnk exists and is equal toX. It is known that allfinite homogeneous (discrete time) Markov chains always have a non-empty set of stationary distributions [Nor98]. Given a Markov chain withm stationary distributionsX 1,···,X m, letS i = Supp(Xi) be the support of the stationary distributionX i where1≤i≤m. For all recurrent statesq, there exists at least one supportS i such thatq∈S i; and moreover, none of the supportsS i contains some transient state. We refer to the setsS 1,···,S m as periodic supports of recurrent states. For instance, the periodic supports of recurrent states for the Markov chain of Example 2.4 areS 1 ={q 1, q2} andS 2 ={q 1, q3}. As a result, the probabilityM n(q) in all transient statesq vanishes whenn→∞, and all recurrent states are visited infinitely often with probability1.

2.2.3 Strategies For an MDP, a strategy is a recipe (a program) that determines the player’s choice of action in all diﬀerent situations of the game played on the MDP. The strategy observes the ﬁnite sequence of states visited so far in the game and suggests the player a probability distribution over a designated set of good actions.

23 Deﬁnition 2.4 (Strategies). A randomized strategy for an MDPM (or simply a strategy) is a functionα: Pref(M)→D(A) that, given aﬁnite pathρ, returns a probability distribution# α(ρ) over the action set, used to select a successor stateq ′ ofρ with prob- ′ ability a∈A α(ρ)(a)·δ(q, a)(q ) whereq= Last(ρ).

A strategyα is pure if for allρ∈ Pref(M), there exists an actiona∈A such that α(ρ)(a) = 1; and memoryless ifα(ρ) =α(ρ ′) for allρ,ρ ′ such that Last(ρ) = Last(ρ′). We view pure strategies as functionsα: Pref(M)→A, and memoryless strategies as functions α:Q→D(A). Example 2.5. LetM be the MDP described in Example 2.3. The strategyα is a randomized strategy forM, that is, given a preﬁxρ=p 0p1 ···p n in Pref(M), deﬁned by ⎧ ⎨ the Dirac distribution ona ifn<4 andp n ∈{q 0, q1} α(ρ) = ⎩ the uniform distribution on{a, b} ifn<4 andp n ∈{q 2, q3} the Dirac distribution onb otherwise,

The strategyα purely playsa for thefirst four rounds, if the MDP is inq 0 orq 1; otherwise 1 in those rounds, it plays eithera orb, each with probability 2 . For all next rounds,α purely playsb. The strategyβ, defined for a given prefixρ∈ Pref(M) by ' a ifn<4, β(ρ) = b otherwise, is a pure strategy forM; andγ is a memoryless strategy whereγ(q 0) =a andγ(q i) =b for all0

24 For instance, we see that the strategyα described in Example 2.5 is actually afinite- memory strategy that can be represented by the transducer⟨Memory,m 0,α u,α n⟩ where Memory={m 0, m1,···,m 4}. The update function isα u(m4, q)=m 4 andα u(mi, q)=m i+1 for all0≤i<4 and statesq∈Q. The next-move functionα n is defined as follows. ⎧ ⎨ the Dirac distribution ona if0≤i<4 andq∈{q 0, q1} αn(mi, q)= ⎩ the uniform distribution on{a, b} if0≤i<4 andq∈{q 2, q3} the Dirac distribution onb otherwise. For thefinite-memory strategyα, the obtained Markov chain isM(α)=⟨Q ′,δ ′⟩ where ′ Q ={m 0,···,m 4} × {q0,···,q 3} and the transition function is defined as follows. The transitions in⟨m i, q0⟩ is a uniform distribution on{⟨m i+1, q0⟩,⟨m i+1, q1⟩} for all0≤i<4 and the transition in⟨m 4, q0⟩ is a self-loop. The transitions in⟨m i, q1⟩ are deterministic and go to⟨m i+1, q1⟩ for all0≤i<4 while the transition in⟨m 4, q1⟩ is the Dirac distribution on⟨m 4, q2⟩. Finally, for all modesm i, the transitions in two states⟨m i, q2⟩ and⟨m i, q3⟩ are directed to the same state: to⟨m i+1, q3⟩ if0≤i<4, and to⟨m 4, q3⟩ otherwise.

2.3 Probabilistic Automata and randomized word

A natural extension for non-deterministic automata are probabilistic automata where the non-deterministic choice of successors for a pair of states and actions is resolved by ran- domness. Reactive systems, by nature, respond to environmental actions. Finite-state automata and probabilistic automata are used to model a kind of reactive systems where the reactions to the environment arefixed by a controller before the system execution. An automaton inputs an infinite word where the letters are, one-by-one, read as the controller’s choice among possible reactions. Observing the system execution by controllers is a way to en- hance modeling reactive systems; in this case, the controller provides a strategy that is able to choose an action (or a letter) according to the sequence of states visited along the system execution. Considering the power that controller gains by observing the execution, MDPs can be viewed as a generalization of probabilistic automata. An infinite word can be viewed as a pure blind strategyα:N→A that chooses actions (or letters) independently of any observation on the system execution but only depending on the number of previous actions that it has chosen so far. We sometimes consider randomized words that are se- quencesd 0d1d2 ··· of probability distributionsd i ∈D(A) over the letters, i.e., randomized blind strategiesα:N→D(A).

Definition 2.5 (Probabilistic automata). A probabilistic automaton (PA)P= ⟨Q,A,δ⟩ consists of afinite setQ of states, afinite alphabetA of input letters and a probabilistic transition functionδ:Q×A→D(Q).

As we may compare the deﬁnitions 2.2 and 2.5, the syntax of PAs and MDPs are the same, but the semantics (the behaviors) are diﬀerent. Given an initial distribution

25 X0 ∈D(Q), the behavior of a PA is as follows. The automaton starts in the stateq with probabilityX 0(q) and reads thefirst letter of the input pure wordw (or if the input word w=d 0d1d2 ··· is randomized, it reads the lettera that is chosen amonga∈ Supp(d 0) ′ with probabilityd 0(a)). The automaton moves to the next stateq that is chosen with probabilityδ(q, a)(q ′) if the automaton is in the stateq and the inputted letter isa. Then, automaton reads the next letter (or the next probability distribution) ofw. Since PAs are an extension of NFAs, and since input infinite words for PAs are indeed blind strategies for MDPs, all definitions such as runs and prefixes are analogously defined for PAs.

Example 2.6. Consider the automaton depicted in Figure 2.2 on page 21 as an instance of a PA. This automaton is described in Example 2.3 as an MDP. Letw=(a·b) ω be an input pure word that can be described as the pure blind strategyα as follows. For all preﬁxesρ with lengthn, ' a ifn≡ 0 (mod 2), α(ρ) = b ifn≡ 1 (mod 2).

ω ∗ ω The wordw results in the following set of inﬁnite runs over the PA:q 0 +q 0(q0q0) q1q2(q3) . ▹

2.4 Winning and acceptance conditions

A Borel set 2 ϕ⊆Q ω is a set of infinite sequences of states (an infinite run or a play) ω in the Cantor topology onQ . For an infinite sequence of statesπ=q 0q1q2 ··· , define Inf(π) ={q∈Q| for alli≥0 there existsj≥i such thatq j =q} which is the set of states visited infinitely often alongπ. An important subclass of the Borel sets are theω- regular ones, that areω-regular languages overQ. We mention reachability, safety, Büchi and coBüchi condition from this subclass:

Reachability and safety conditions. A reachability setT⊆Q for a system requires that some target state inT be visited along the system execution. The set ω ♦T={q 0q1q2 ···∈Q | there existsi≥0 such thatq i ∈T} is, thus, a set of winning plays or accepting inﬁnite runs for the system with the reachability condition on the target setT. In the same way, a safety setT⊆Q for a system requires that only safe states inT be visited ω along the system execution. The set!T={q 0q1q2 ···∈Q | for alli≥0 we haveq i ∈T} is, thus, a set of winning plays or accepting inﬁnite runs for the system with the safety condition onT. 2. Generally speaking, a Borel set is any set in a topological space that is built on open sets (or equivalently on closed sets) by countable unions, countable intersections and relative complement. We refer the reader to [Mar90, Mar98] for more details.

26 Büchi and coBüchi conditions. A Büchi setT⊆Q for a system requires that some state inT be visited along the system execution infinitely often. The set!♦T={π∈ Qω | Inf(π)∩T≠ ∅} is, thus, a set of winning plays or accepting infinite runs for the system with the Büchi condition on the target setT . In the same way, a coBüchi set T⊆Q for a system requires that only states inT be visited infinitely often along the system execution; all states outsideT can only be visitedfinitely many times. The set ♦!T={π∈Q ω | Inf(π)⊆T} is, thus, a set of winning plays or accepting infinite runs for the system with the coBüchi condition onT.

Note that the set of winning plays of safety and reachability conditions are dual, that is♦T=Q ω \!(Q\T); the same holds for the set of winning plays of Büchi and coBüchi conditions:!♦T=Q ω \♦!(Q\T).

Example 2.7. Consider the target setT 1 ={q 2} andT 2 ={q 1, q2} for the MDPM in Example 2.3 on page 21. The set of winning plays (or accepting runs ifM behaves ∗ ω ω as a PA) is♦T 1 =Q ·q 2 ·Q and is!T 1 =q 2 for reachability and safety condition onT 1, respectively. As a result, the set of inﬁnite plays inM that achieves the reachability ∗ ∗ ω condition♦T 1 isq 0q1q2(q3) . This set is an empty set for the safety, Büchi and coBüchi ∗ ω ∗ ∗ ω conditions. Similarly, the set of plays inM that win♦T 2 isq 0(q1) +q 0q1q2(q3) and it is empty for the safety condition. The sets of plays inM that achieves Büchi condition!♦T 2 ∗ ω and coBüchi condition♦!T 2 is the same set:q 0(q1) . ▹

2.4.1 Languages of NFAs A regular language (a subset ofL∈A ∗) is recognizable by afinite automatonN defined over the alphabetA and augmented with an initial setQ 0 of states and afinite acceptance conditionF⊆Q, if for allfinite wordsw of the language, there is an accepting run overN. Afinite runq 0q1 ···q n is accepting for the automatonN if it starts in some stateq 0 ∈Q 0 of the initial set and if the last state of the run visits the acceptance conditionsq n ∈F; the regular language of the automaton is, thus,

∗ w L(N)={w∈A | there existsq 0, q such thatq 0 −→q whereq 0 ∈Q 0 andq∈F}.

Anω-regular language (a subset ofL∈A ω) is recognizable by afinite automatonN defined over the alphabetA and augmented with an initial setQ 0 of states and anω- regular acceptance condition with the setΩ of accepting runs (acceptance conditions such as reachability withΩ=♦T and Büchi condition withΩ=!♦T ) if for all infinite wordsw of the language, there is an accepting infinite run inN . Thus,

ω LΩ(N)={w∈A |w induces an inﬁnite runq 0q1 ··· such thatq 0 ∈Q 0 andq 0q1 ···∈Ω}.

27 Example 2.8. Letq 0 be the unique initial state of the automaton described in Example 2.2 on page 19. For theﬁnite acceptance conditionF={q 0}, we have L(N)=b ∗ ·(a·b ∗ ·a·b ∗ ·a·(a+b)·b ∗)∗

ω that we nameL 1. LetT={q 0} be the target set as well, thenL !T (N)=b that we nameL 2. The language for the coBüchi condition ofT isL ♦!T (N)=L 1 ·L2. ▹ A Finite automaton with a Büchi or coBüchi acceptance condition is said to be a Büchi or coBüchi automata, respectively.

2.4.2 Path-outcomes of MDPs

Given an initial distributionX 0 ∈D(Q) and a strategyα in an MDPM, a path- outcome is an infinite pathπ=q 0q1 ... inM such thatq 0 ∈ Supp(X0) and for alli≥0, the stateq i+1 ∈ post(qi, a) for some actiona∈ Supp(α(q 0 . . . qi)). The probability of a finite prefixρ=q 0q1 . . . qn of the playπ is

n(−1 ) X0(q0)· α(q0 . . . qj)(a)·δ(q j, a)(qj+1). j=0 a∈A

We denote by Outcomes(X 0,α) the set of all path-outcomes fromX 0 under strategyα. As an example, letX 0 be the Dirac distribution onq 0 for the MDPM in Example 2.3. Consider the pure memoryless strategyγ that playsa inq 0, and that playsb in all statesq≠ Outcomes ω ∗ ω q0 (presented in Example 2.5). Thus, (q 0,γ)=(q 0) +q 0q0q1q2(q3) , and the 1 1 probability of thefinite prefixq 0q0 is 4 and the probability ofq 0q0q0q1q2 is 8 . The winning setsΩ of plays for theω-regular conditions (the setΩ is also called an event in this context) are measurable sets of paths, and thus given an initial distributionX 0 and a strategyα, the probabilities Pr α(Ω) of eventsΩ are uniquely defined [Var85]. Hence, Prα(♦T) is the probability to reachT under strategyα. For the MDPM of Exam- ple 2.3, the Dirac initial distributionq 0 and the strategyγ: the probability of reachingq 2 γ γ is Pr (♦{q2}) = 1 whereas the probability of the safety condition Pr (!{q0}) = 0.

Deﬁnition 2.6 (Winning modes for theω-regular conditions). Given an initial distri- butionX 0 and an eventΩ, we say that an MDPM is: – sure winning if there exists a strategyα such that all path-outcomes are winning: Outcomes(X0,α)⊆Ω; – almost-sure winning if there exists a strategyα such that Pr α(Ω) = 1; α – limit-sure winning if supα Pr (Ω) = 1.

γ For the MDPM of Example 2.3, even though Pr (♦{q2}) = 1 (fromq 0) implies that ω M is almost-sure winning for the reachability condition♦{q 2}, the inﬁnite play(q 0) is

28 a counterexample witness proving thatM is not sure-winning for this condition. The deﬁnitions result inM being limit-sure winning for the reachability condition♦{q 2} since M is almost-sure winning for that condition. Actually, almost-sure and limit-sure winning modes coincide for reachability in MDPs [dAHK07].

2.4.3 Languages of PAs The classical languages of PAs are defined on pure words: since pure words for probabilistic automata can be viewed as pure blind strategies for MDPs, the set of runs of a finite word in a PA define a measurable event. Given an initial distributionX 0 for a PAP, the probability that afinite runρ=q 0q1 ···q n of afinite pure wordw=a 0 ·a 1 ···a n−1 inP happens, is n(−1 w Pr (ρ) =X 0(q0)· δ(qj, aj)(qj+1). j=0 For thefi#nite acceptance conditionF, the probability Pr(w) of the wordw to be accepted w is thus ρ∈Runs(w,F) Pr (ρ) where Runs(w,F) is the set of accepting runs over thefinite wordw inP; recall that a runρ=q 0q1 ···q n is accepting ifq n ∈F. Given a threshold 0≤λ≤1, the language of a PAP over thefinite words is L(P)={w∈A ∗ | Pr(w)≥λ}. For infinite words, we follow all definitions presented for the MDPs. Given an initial distributionX 0 ∈D(Q) and the infinite wordw in the PAP, the set of infinite runs overw inP is Outcomes(X 0, w) in the MDPM where the behavior ofP is interpreted as an MDP. This way, the probabilities Prw(Ω) ofω-regular acceptance conditions with the setΩ of accepting runs are uniquely defined [Var85]. Theω-language

ω w LΩ(P)={w∈A | Pr (Ω)>0}. is recognizable by a PAP augmented with the initial distributionX 0 and theω-regular acceptance condition with the setΩ of accepting runs.

2.5 Decision Problems

2.5.1 Problems for NFAs We recall two classical decision problems for NFAs.

Decision problem(s). The emptiness problem asks, given an NFAN with an acceptance condition, whether the accepted language ofN is empty; and universality problem asks whether the accepted language isA ∗ for regular languages or isA ω for ω-regular languages.

29 Finite acceptance condition. The emptiness problem for NFAs withfinite acceptance conditionsF is reducible tos−t connectivity problem in digraphs, that can be determined in NLOGSPACE in the size of the graph (and thus the automaton) [SVW87, Sip97]. To decide the emptiness problem, while performing a depth-first search algorithm that visits all accessible states for which there is a directed path from some initial states, one can check whether one of those accessible states is inF. The universality problem for an NFAN is PSPACE-complete [IRS76]. The proof of the PSPACE-hardness is by a reduction from the space-bounded tiling problem that is known to be PSPACE-complete; and since co-PSPACE is equal to PSPACE, the (N)PSPACE upper bound follows from a non-deterministic algorithm that decides co-universal problem: guess one-by-one letters of afinite wordw (with|w|≤2 n wheren is the number of states inN), and check whetherw is not accepted by the automaton. The algorithm uses polynomial space to keep track of visited states along the runs induced by inputtingw to see if the last state of some of those runs do not visitF.

ω-regular acceptance condition. The emptiness problem for an NFAN with a reachability condition♦T reduces to the same problem with the Büchi condition!♦T where the statesq∈T are transformed to absorbing states. On the other hand, safety and coBüchi conditions are accordingly duals of reachability and Büchi conditions, thus we only mention the complexity of classical decision problems for NFA with a Büchi acceptance condition. The emptiness problem for NFA with a Büchi condition!♦T is solved by means of graph algorithms. Such algorithmsﬁrst run a SCC decomposition, and then see if there is a directed path from an initial state to some SCC that contains at least one cycle and that contains at least one stateq∈T . It is proved that the emptiness problem for Büchi automata is NLOGSPACE-complete. The universality problem for Büchi automata is though PSPACE-complete [SVW87].

2.5.2 Problems for MDPs We recall following decision problems for MDPs.

Decision problem(s). The membership problem of {sure, almost-sure, limit-sure} modes forω-regular winning conditionΩ asks, given an MDPM and given an initial distributionX 0, whetherM is {sure, almost-sure, limits-sure} winning forΩ fromX 0, respectively.

For the MDPM and the winning conditionΩ, the set of all initial distributions from whichM is winning, is the winning region. It is known that for reachability, safety, Büchi and coBüchi winning conditions, and for all three winning modes, the membership problems are decidable in PTIME [dAH00, dAHK07, CH12]. For all these winning conditions, it is shown that the randomization is not necessary. Moreover, for all distributionsX 0 in the winning region of all these conditions, all Dirac distributions on all the statesq∈ Supp(X 0)

30 are winning too. We say that such states are winning states. Moreover for reachability and safety conditions, it is known that memoryless strategies are suﬃcient for all three winning modes. The membership problems of almost-sure and limit-sure modes for the winning conditions in MDPs, are mostly studied as qualitative analysis of MDPs [BK98b, BK08, CDH10, CHJS13]. In the sequel, we give some hints on how to decide the membership problem for the diﬀerent winning modes of all reachability, safety, Büchi and coBüchi winning conditions.

Reachability condition. LetM be an MDP with state spaceQ and letT⊆Q. The membership problem of sure winning for reachability condition♦T is reducible to reachability in a turn-based two-player game, where the probabilistic choice of the environment is simulated by an adversary called Player-2 (since the exact values of transitions do not matter). Initially, Player-2 chooses a stateq∈ Supp(X 0) where the game starts in. For each round, Player-1 chooses an actiona; and then the adversary chooses a successor state among the states in post(q,a). The problem of deciding whether Player-1 has a strategy to ensure reaching a target setT , is known to be PTIME-complete [CH12]. The following fix-point computation is a naive algorithm to compute the set of winning states for sure reachability. For alln≥1, compute win n = Pre(winn−1)∪ win n−1 where win0 =T . We see that win0 ⊆ win1 ⊆ win2 ⊆··· and sinceQ isfinite (and all win i ⊆Q) then this sequence will stabilize (in at most|Q| iterations) into a set win that is the set of winning states. The membership problem for almost-sure winning is more tricky than what it is for sure winning mode, see [dAH00, dAHK07, CH12]. We give few hints on how the set of winning states is computed for almost-sure reachability condition♦T in an MDPM=⟨Q,A,δ⟩. The computation of winning states for almost-sure reachability condition has the solution of positive reachability in the main core: positive reachability problem asks whether the probability Pr(♦T) is (strictly) positive. The positive reachability problem can be solved by a simplefix-point computation. To use this problem for reachability though we define a variant of positive reachability that is reaching a target set with some positive probability while keeping the remaining part of probability mass in the winning region (for positive probability condition). GivenX,Y⊆Q, let APre(X,Y):{q∈Q|∃a∈A:(post(q, a)∩ X≠ ∅ and post(q,a)⊆Y)} be the set of states from which the player has some actiona to ensure reachingX with some positive probability, and moreover the setY is not left with probability1. A nestedfix-point computation gives us the set of winning states win. Let win0 =Q, and for allm≥0, iterate the following two steps: (1) for alln≥1 computeX n = APre(Xn−1, winm)∪X n−1 whereX 0 =T , until the computation stabilizes intoX (i.e.,X n =X n+1 =X for somen), and then as the next step let (2) winm+1 =X. SinceQ isfinite and win m+1 ⊆ winm then this sequence will stabilize (in at most|Q| iterations) into a set win that is the set of winning states from which there is a positive probability to reachT in at most|Q| rounds, and since win is never left, the probability of

31 eventually reachingT is 1. Thus, win is the set of winning states for almost-sure reachability condition♦T. We study another approach, which might be more intuitive, to compute the winning set for almost-sure reachability. GivenS⊆Q, let Pre some(S) ={q∈Q|∃a∈A: post(q,a)∩S≠ ∅} be the set of states from which the player has some action to ensure reachingS with some (strictly) positive probability; note that Pre some(S) = APre(S,Q). Let Preall(S)={q∈Q|∀a∈A: post(q,a)∩S≠ ∅} be the set of states from which the player has no choice unless going toS with some positive probability. An algorithm to compute the set of winning states for positive reachability condition♦T is the followingfix-point computation. For alln≥1, compute win n = Presome(winn−1)∪ winn−1 where win0 =T . We see that win 0 ⊆ win1 ⊆ win2 ⊆··· and sinceQ isfinite (and all wini ⊆Q) then this sequence will stabilize (in at most|Q| iterations) into a set win some(♦T) that is the set of winning states for positive reachability♦T ; in other words, the set of states from which the player has some strategy to ensure positive reachability♦T . In a similar way, the followingfix-point computation, win n = Preall(winn−1)∪win n−1 for alln≥1 where win0 =T , gives the set of states win all(♦T) where all strategies of the player are winning for the positive reachability♦T. We are now equipped to solve almost-sure reachability♦T : letM 0 =M be the MDP andQ 0 =Q be the state space. For alli≥0, compute the set

Qi+1 =Q i \ winall(♦(Qi \ winsome(♦T )));

where winsome and winall are computed on the MDPM i. For alli≥1, the MDPM i is obtained fromM i−1 where the state space isQ i ∪{sink} and the transition functionδ i is deﬁned as follows. For all statesq and actionsa such that post(q,a)⊆Q i, letδ i(q, a) = δi−1(q, a). Otherwise,δ i(q, a) is the Dirac distribution on sink. The state sink is an absorbing state. Intuitively, the algorithmﬁrst computes all states win some(♦T) from whichM i has a strategy to reachT with some positive probability. Thus, the setQ i \ winsome(♦T) contains all states from which there is no chance to reachT , even with a very small probability. Another important point is, as soon asM i enters in some state of the set Qi \ winsome(♦T) , regardless of the player strategy, it always remains in that set with some positive probability. Thus, all states from which player has no choice unless going toQ i \ winsome(♦T) with some positive probability, must be avoided. We remove such states, that are states in winall(♦(Qi \ winsome(♦T)), fromM i (and all in-going or outgoing transitions) to obtainM i+1. We repeat this computation until there is no such state to be removed from the MDP, and as a result the set of winning states for almost-sure reachability♦T is computed. All the computations can be done in PTIME. We refer the reader to [BK08] for more details and the correctness of both algorithms. The set of limit-sure winning states for reachability condition coincide with the set of almost-sure winning states [dA97]. The PTIME-hardness results of the membership problems for reachability condition, is by a reduction from monotone Boolean circuit problem, that is known to be PTIME-complete [GR88].

32 Safety condition. LetM be an MDP with state spaceQ and letT⊆Q. In analogy to reachability, the membership problem of sure winning for safety condition!T is reducible to safety in a turn based two-player game. A naive algorithm to compute the set of winning states for sure safety condition!T in the MDPM is the followingﬁx-point computation. For alln≥1, compute win n = Pre(winn−1)∩win n−1 where win0 =T and Pre(S)={q∈Q| ∃a∈A: post(q,a)⊆S}. We see that win n ⊆ winn−1, therefore this sequence will stabilize (in at most|T| iterations) into a safe set win that is the set of winning states. It is known for safety conditions!T in MDPs that the three winning modes coincide [CH12].

Büchi and coBüchi conditions. LetM be an MDP with state spaceQ and let T⊆Q. Analogous to reachability, the membership problem of sure winning for Büchi condition!♦T and coBüchi condition♦!T are reducible to Büchi and coBüchi in a turn based two-player game. A well-known iterative algorithm to compute the set of winning states for Büchi condition!♦T in a turn-based two player games, has the following logic. For alli=1,2,···,|Q|, the algorithm performs two steps: (1) computes the set wini of states where the Player-1 has a strategy to ensure a visit toT . Next, (2) it computes the set losei of states where the adversary has a strategy to ensure a visit toQ\ win i. The set losei is removed from the game. The states that are remaining in the game after the computation are all sure winning for Büchi condition!♦T . We refer the reader to [CHP08] for the detailed algorithm.

For an MDPM, a setC⊆Q is closed if for every stateq∈C, there existsa∈A such that post(q,a)⊆C. For eachq∈C, letD C (q) ={a∈A| post(q,a)⊆C}. The digraph induced byC isM#C=(C,E) whereE is the set of edges⟨q, q ′⟩ ∈C×C ′ such thatq ∈ post(q,a) for somea∈D C (q). An end component is a closed setU such that the digraphM#U is strongly connected. An end componentU⊆Q is terminal if post(U,A)⊆U. Let endComp(M) be the set of all end components. For the MDPM of Example 2.3, the set of all end components is endComp(M) ={{q 0},{q 1},{q 3}}. The following remarks are folklore results about end components: Remark1. Given an MDPM, for all end componentsU∈ endComp(M) and all initial distributionsX 0 where Supp(X0)⊆U, there exists a (finite-memory) strategy such that Prα(!U)=1 and Pr α(!♦{q}) = 1 for allq∈U. Roughly speaking, this remark says that for all end componentsU there is a (finite- memory) strategy that almost surely ensures thatM stays forever in the end componentU and ensures that all states of the end componentU are visited, infinitely often. The second folk remark is: Remark2. For an MDPM, letΩ={π∈ Plays(M)| Inf(π) =U for someU∈ endComp(M)}. Then Prα(Ω) = 1 under all strategiesα and from all initial distribu- tionsX 0.

Proof (sketch). Consider an initial distributionX 0 and a strategyα for the MDPM. For all inﬁnite pathsπ=q 0q1q2 ··· of the MDPM such thatq 0 ∈ Supp(X0), and allq∈ Inf(π),

33 there is some actiona that is played byα, infinitely often, for the prefixesρ of the play with Last(ρ) =q (prefixes that ends inq). Given the condition that thea-transition inq is taken infinitely often along the playπ, for allq ′ ∈ post(q,a), the conditional probability thatM moves fromq toq ′ for onlyfinitely many times is zero. Therefore, given that thea-transition inq is taken infinitely often along the playπ, all successors post(q,a) of thea-transition inq are almost-surely visited infinitely often alongπ. Hence, for all statesq∈ Inf(π) there is an actiona such that post(q,a)∈ Inf(π) meaning that Inf(π) almost-surely is (strongly connected and) included in an end component. !

These two remarks enable us to reduce the membership problem of almost-sure mode for Büchi and coBüchi conditions, to the membership problem of almost-sure for reachability. For the Büchi condition!♦T , letU good be the union of all end compo- nentsU∈ endComp(M) such thatU∩T≠ ∅; the membership problem of almost-sure for Büchi condition!♦T is reducible to the membership problem of almost-sure mode for reachability condition♦U good. The reduction is correct since as soon as the MDPM enters the end componentU whereU∩T≠ ∅, there is a strategy that ensures, infinitely often, the visits toT. The following modifications are necessary when reducing the membership problem of almost-sure coBüchi condition♦!T to the membership problem of almost-sure reachability condition. The reason for the following modifications is because an end componentU∈ endComp(M) of an MDPM whereT⊆U, may include some bad stateq ̸∈T where there is no strategy to avoid visiting that bad state whereas it may have such a state but there is some strategy which visits only the states ofU∩T , infinitely often. LetM ′ be a copy ofM where all statesq ′ ̸∈T are replaced with an absorbing state sink, and for all states q∈T and all actionsa∈A, if the successor set of thea-transition inq is not included inT , i.e., post(q,a) ̸⊆T , then thea-transition is redirected to sink. LetU good be the union of all end componentsU ′ ∈ endComp(M′) of theM ′ such thatU ′ ⊆T . The membership problem of almost-sure for coBüchi condition♦!T is reducible to the membership problem of almost-sure mode for reachability condition♦U good in the same MDPM. As the almost-sure and limit-sure winning states for reachability condition coincide in MDPs, the sets of winning states of almost-sure and limit-sure are the same for Büchi condition too. The same result holds for coBüchi condition.

2.5.3 Problems for PAs Similar to NFAs, the emptiness problems is a classical decision problem for PAs. We recall that the classical languages of PAs are deﬁned on pure words.

Decision problem(s). The emptiness problem asks, given a PAP with an acceptance condition, whether the accepted language ofP is empty.

34 The language offinite words for a PAP is defined for a given threshold0≤λ≤1; recall thatL λ(P)={w∈A ∗ | Pr(w)≥λ}. The emptiness problem asks, thus, the existence of afinite wordw∈A ∗ such that Pr(w)≥λ. The strict emptiness problem is to decide the existence of afinite wordw∈A ∗ such that Pr(w)>λ. For the extreme valueλ=0, the answer to the emptiness problem is always yes; and the strict emptiness problem reduces to emptiness problem for NFAs. For the extreme valueλ=1, emptiness problem is reducible to the universality problem for NFAs; and the answer to the strict emptiness problem is always no. For0<λ<1, both emptiness problem and strict emptiness problem are undecidable [GO10]. It is known [BBG08, CT12] that the emptiness problem for PAs with positive Büchi acceptance conditions is undecidable, whereas it is PSPACE-complete for almost-sure Büchi acceptance conditions. The emptiness problem for PAs with almost-sure safety and reachability conditions are PSPACE-complete, too. An interesting problem for PAs with afinite acceptance condition is the value1 problem, which is used to establish several complexity results throughout this thesis. The value of a PAP augmented with thefinite acceptance conditionF, is val(P) = sup w∈A∗ Pr(w).

Decision problem(s). The value1 problem asks, given a PAP and aﬁnite acceptance condition, whether val(P)=1.

Intuitively, it asks whether some words exist with the acceptance probability arbitrarily close to1. The value1 problem for PAs is proved undecidable in [GO10]. Subsection 6.2.5 on page 134 provides some details about this problem.

35 36 Synchronizing3 Problems

First sight. In this chapter we define the synchronizing problems we are mainly concerned about in this thesis. We recall the definition offinite synchronizing words for NFAs, and we survey decision problems and conjectures aboutfinite synchronizing words. We then provide definitions of infinite synchronizing words for NFAs. We introduce variants of synchronizations in MDPs and PAs by considering synchronizing strategies and (finite and infinite) synchronizing words. We study the relation between one-letter finite alternating automata and MDPs. We also briefly discuss the synchronization in two-player turn-based games.

Contents 3.1 Synchronization in NFAs ...... 38 3.1.1 Finite synchronization in NFAs ...... 38 3.1.2 Inﬁnite synchronization in NFAs ...... 42 3.2 Synchronization in MDPs ...... 45 3.3 Synchronization in PAs ...... 51 3.4 Relation to one-letter alternatingﬁnite automata ...... 52 3.4.1 Problems for 1L-AFA ...... 54 3.4.2 Connection with MDPs...... 56 3.5 Relation to two-player turn-based games...... 58 3.6 Discussions ...... 59

37 3.1 Synchronization in NFAs

Thefinite synchronizing words are well-studied forfinite automata, for example see [Čer64, Pin78, Epp90, IS95, IS99, San04, Vol08, Mar10, AGV10, OU10, Mar12]; such a word drives the automaton from an unknown or unobservable state to a known specific state. As an example think of remote systems connected to a wireless controller that emits the command via wireless waves but expects the observations via physical connectors (it might be excessively expensive to mount wireless senders on the remote systems), and consider that the physical connection to controller is lost due to some technical failure. The wireless controller can therefore not observe the current states of the distributed subsystems. In this setting, emitting a synchronizing word as the command leaves the remote system (as a whole) in one particular state no matter in which state each distributed subsystem started; and thus the controller again regain control. For synchronizing words, called directable or reset words as well, applications in planning, control of discrete event systems, bio-computing, and robotics [BAPE+03, Vol08, DMS11a].

3.1.1 Finite synchronization in NFAs Aﬁnite automaton is synchronizing if there exists an input word, called synchronizing word, that brings the automaton from every state to the same state.

Deﬁnition 3.1 (Finite synchronizing words). For a complete NFAN=⟨Q,A,∆⟩, a ﬁnite synchronizing word is a wordw∈A ∗ for which there exists a stateq∈Q such w thatQ −→{q}.

The natural decision problem forﬁnite synchronization in NFAs is as follows.

Decision problem(s). Theﬁnite synchronizing problem asks, given an NFA, whether aﬁnite synchronizing word exists.

A straightforward approach to decidefinite synchronizing problem is a reduction to the emptiness problem in the subset construction of the NFA. For a complete NFAN= ⟨Q,A,∆⟩, the subset construction which is usually defined to determinize the automaton, consists in⟨Q ′,A,∆ ′⟩ where the state spaceQ ′ = 2Q \{∅} is the set of all cells, i.e., subsets of states inN , and the transition function∆ ′ :Q ′ ×A→Q ′ defined as∆ ′(c, a) =c ′ a wherec −→c ′ anda∈A. Thefinite synchronizing problem forN can be encoded as the emptiness problem for the deterministic subset construction ofN equipped with the initial cellc 0 =Q and the setF={{q} |q∈Q} of accepting cells. By this reduction, one can prove that the language offinite synchronizing words for an NFAN , that is Lsynch ={w∈A ∗ |w is afinite synchronizing word forN}, is a regular language.

38 a, b q0 b {q0, q1, q2} b b a a a b b q2 q1 a a {q0, q1} {q0} b a, b

Figure 3.1: The NFA described in Exam- Figure 3.2: The subset construction of ple 3.1 with synchronizing worda·a·b. the NFA in Example 3.1 (restricted to the part showing the run overa·a·b).

Example 3.1. The NFAN depicted in Figure 3.1 has three statesq 0, q1, q2 and has two lettersa andb. Thea-transitions inq 0 is non-deterministic and bringN intoq 0 orq 1: a {q0} −→{q 0, q1}; thea-transitions inq 1 andq 2 however deterministically go toq 1. The b b-transition inq 2 spreads the current state of the automaton into the state space:{q 2} −→ {q0, q1, q2} whereas theb-transitions inq 0 andq 1 are deterministic and moves the automaton a·a·b intoq 0. The worda·a·b is synchronizing since{q 0, q1, q2} −−→{q 0}; it can be verified by Figure 3.2 that illustrates some part of the subset construction to show the run from the initial cell{q 0, q1, q2} to the accepting cell{q 0} over the worda·a·b. We see that all words b∗ ·a·a ∗ ·b·a ∗ are synchronizing too. ▹ The reduction to emptiness problem for subset construction provides the PSPACE membership of thefinite synchronizing problem for NFAs. A reduction from thefinite automata intersection that is PSPACE-complete [Koz77, GJ79, San04], has established the matching lower bound to prove thatfinite synchronizing problem is indeed that costly, even if only one transition in the NFA is non-deterministic [IS99, Mar10, Mar12]. The same complexity results hold for thefinite synchronizing problem of not-complete DFAs, called partialfinite automata (PFAs), even if the transition function is undefined only for one state-action pair [Mar10, Mar12]. When the DFA is complete, thefinite synchronizing problem lies in NLOGSPACE. Below, we see that the reason of the lower complexity for DFAs is the pair-wise synchronizing technique whose idea comes from a Lemma proved byČerný in the early 1960’s [Čer64].

Finite synchronizing word for DFAs. For a given complete deterministicfinite au- tomatonN=⟨Q,A,∆⟩, there exists afinite synchronizing word if, and only if, for all pairs w of statesq 1, q2 ∈Q, there exists a stateq and a wordw such that{q 1, q2} −→{q}. This result is by a Lemma fromČerný [Čer64, Epp90, Vol08]; one direction of the statement is w trivial, by monotonicity for all statesq, ifQ −→{q} implying thatw is a synchronizing word w forN then{q 1, q2} −→{q} for all pairs of statesq 1 andq 2. The reverse direction is provable by an easy observation about DFAs: when runs (over any arbitrarily word) starting in two statesq 1 andq 2 cross each other in some stateq, those runs stay merged/synchronized forever. In other words, all wordsw·v∈A ∗ are synchronizing whenw is afinite synchronizing

39 a b a {q1, q3} {q0, q2} {q0, q3} b a the automaton of Example 2.2 b a b a a a {q0, q1} {q1, q2} {q2, q3} q0 q1 b b b b a, b a

q q 3 a 2 b

Figure 3.3: The subset construction of the NFA in Example 2.2 on page 19 (where the state space is restricted only to cells of size at most2). word followed byv∈A ∗. This observation does not hold in a non-deterministic setting. As a result in DFAs, the state space can be synchronized by consecutive synchronizations of pairs of states. The NLOGSPACE upper bound for DFAs follows from an algorithm repeating pair-wise synchronization as follows: (1) choose a pair of statesq 1 andq 2 from the setS of must-be-synchronized states, whereS is initially set to the state spaceQ. w (2)find a wordw for which there existsq such that{q 1, q2} −→{q}; if such a word does not exists, return “no synchronizing word exists”. Otherwise, update the set of must-be- w synchronized states: compute the successor setS ′ after inputting the wordw:S −→S ′ and letS=S ′. (3) ifS is a singleton, return “the automaton is synchronizing”. The termination of this algorithm is guaranteed byČerný’s Lemma. In fact by each iteration, the setS shrinks to a strictly smaller set and the algorithm terminates in at most|Q| iterations. The algorithm is thus in NLOGSPACE that comes from the second step which isfinding a wordw that merges runs from two statesq 1 andq 2; this can be reduced to the membership problem in the subset construction of the automaton where the state space (of the subset construction) is restricted only to cells of size at most2 (equivalently to the reachability problem in the product of two copies of the DFA). Figure 3.3 shows such subset construction, i.e., restricted to cells of size at most2, for the deterministic automaton described in Example 2.2 on page 19. We see that for all pairs of states, there is a run to the accepting cell{q 0}.Afinite synchronizing word for this automaton is b·a 3 ·b·a 3 ·b. This algorithm constructs a synchronizing word for the input DFA, if any, that is not always the shortest one.

The famousČerný conjecture claims that the length of shortest synchronizing word for a DFA withn>1 states is at most(n− 1) 2. Despite all attempts in last decades, this conjecture has not been proved or disproved.Černý provided a family of the automaton (over the alphabet{a, b}) where the shortest synchronizing word is exactly the extreme value(n− 1) 2. Ann-states automaton in this family has following structure: there is a loop going throughq 0q1 ···q n−1 and closing inq 0. For all states, thea-transitions shift

40 once the automaton in this loopq 0q1 ···q n−1q0. For all statesq≠ q n−1, theb-transition is a self-loop onq; theb-transition inq n−1 takes the automaton toq 0. See Example 2.2 for a formal description for the transition function of the4-state automaton instance of this family. The shortest synchronizing word for the automaton in Example 2.2 isb·a 3 ·b· a3 ·b and for then-state automaton in this family is(b·a n−1)n−2 ·b. This family is the only known family satisfying the extreme bound on the length of shortest synchronizing word [Vol08, AGV10]. The best known upper bound for the shortest synchronizing words 1 3 of ann-state automaton is 6 (n −n− 6), proved by Pin [Pin78, Vol08]. Trahtman recently 1 3 2 tried to provide a better upper bound 48 (7n +6n −16n) where the proof was unfortunately erroneous [Tra11, Tra14]. The related decision problem which asks, given a DFA and the boundk, whether the DFA has a synchronizing word with length less thank, is complete for the classDP [OU10], the closure ofNP∪coNP underﬁnite intersections.

Synchronization from a subset in NFAs. A variant ofﬁnite synchronization in NFAs is synchronization from a subsetS of states.

Decision problem(s). Given an NFAN=⟨Q,A,∆⟩ and a subsetS⊆Q, theﬁnite synchronizing problem from a subsetS asks whether there exists aﬁnite wordw∈A ∗ w and some stateq such thatS −→{q}.

Thefinite synchronizing problem from a subset is PSPACE-complete, even if the automaton is complete and deterministic [San04]. In DFAs, the pair-wise synchronizing technique provides a NLOGSPACE algorithm for the synchronizingfinite problem. This technique fails when synchronizing from a subset is asked because a pair of statesq 1, q2 ∈S are not guaranteed to be merged in a stateq from which (and the states of the must-be-synchronized set) there is a synchronizing word. Thus, the order in which the pairs of state are chosen for pair-wise synchronization matters. Note that, an NFA might not have a synchronizing word but synchronizing words fromS exist for several subsetsS of the states. The PSPACE upper bound for the synchronization from a subset comes trivially from the membership problem of the subset construction. Tofind a synchronizing word fromS, we equip the subset construction with the initial cellc 0 =S and the setF={{q} | q∈Q} of accepting cells. The PSPACE-hardness is by a reduction fromfinite automata intersection [Koz77, GJ79, San04]; thefinite automata intersection asks, givenn DFAs equipped with initial states and accepting sets, is there a commonfinite word that is accepting by all DFAs. Forn automatonN 1,N 2,···,N n over the common alphabetA, let q1, q2,···,q n be the initial states andF 1,F 2,···,F n be the accepting sets. LetN be the automaton that has one copy of each automatonN i (1≤i≤n) and two new absorbing states synch and sink. For a new letter#, let#-transitions in all accepting statesq∈F 1 ∪ ···∪F n of all automata go to the state synch whereas all#-transitions in all non-accepting statesq ̸∈F 1 ∪···∪F n go to the state sink. From the subsetS={q 1, q2,···,q n, synch}, a synchronizing word cannot escape synch because synch is an absorbing state. Letw be the shortest synchronizing word fromS. As a resultw must end with#:w=v·# where

41 ∗ v∈A ; it holds since the only way that runs starting in the initial statesq 1, q2,···,q n arrives in synch is inputting# at the right time when those runs are in some accepting states. One can verify thatv is an accepting word for allN i (1≤i≤n).

3.1.2 Infinite synchronization in NFAs We provide four definitions of infinite synchronizing words for NFAs to define languages of infinite synchronizing words, in analogy to safety, reachability, Büchi and coBüchi acceptance conditions. For an infinite wordw=a 0 ·a 1 ·a 2 ··· , letw 0 =ϵ be the empty word and for alli≥1, letw i denote the prefixa 0 ···a i−1 ofw consisting in thefirsti letters.

Definition 3.2 (Infinite synchronizing words). For a complete NFAN=⟨Q,A,∆⟩, an infinite wordw∈A ω is w • always synchronizing if for alli∈N there exists some stateq such thatQ −→i {q}, • eventually (or event) synchronizing if there existsi∈N and some stateq such w thatQ −→i {q}, • weakly synchronizing if for alln∈N there existsi≥n and some stateq such w thatQ −→i {q}, • strongly synchronizing if there existsn∈N such that for alli≥n there exists some w stateq whereQ −→i {q}

wherew 0 =ϵ and fori≥1, the wordw i is the preﬁx ofw consisting in theﬁrsti letters.

Intuitively, the always and eventually synchronization condition require that the NFA is always synchronized or at least once; though the weakly requires infinitely often synchronization, and strongly requires always synchronization except forfinitely many times. We say that an NFAN is {always, event, strongly, weakly} synchronizing if there exists a(n) {always, event, strongly, weakly} synchronizing infinite word forN.

For an NFAN , let the languageL always ,L event ,L strongly andL weakly be the set of all always, eventually, strongly and weakly synchronizing words, respectively. As an immediate result of definitions: Remark3. For an NFAN , we haveL always ⊆L strongly ⊆L weakly ⊆L event . These inclusions, in general, are strict; see Example 3.2. Note that the languageL always is an empty set for all NFAs that have at least two states. In the sequel, we present variations of these definitions where the always synchronization is possible. For a DFA defined over an alphabetA, we haveL strongly =L weakly =L event whereL event is the set of all words w.v constructed by afinite synchronizing wordw concatenated with an infinite wordv∈A ω.

42 a b

q1 b a, b a a q1 b b b a q2 q3 q0 b a b

q0 a, b a q2

Figure 3.4: The NFAN 1 of Example 3.2 Figure 3.5: The NFAN 2 of Example 3.2 with the strongly and eventually syn- with weakly synchronizing word(a·b) ω chronizing wordsb·a ω andb·a·b ω. and strongly synchronizing worda ω.

Example 3.2. The NFAN 1 in Figure 3.4 has four statesq 0, q1, q2, q3 and two lettersa a andb. Thea-transition inq 0 is non-deterministic{q 0} −→{q 0, q1}; but it is deterministic a a in other states:q 2 −→q 3,q 3 −→q 2 and it is a self-loop onq 1. Theb-transitions inq 0 and q1 take the automaton toq 2, and it is a self-loop onq 2 itself. Theb-transition inq 3 is b non-deterministic:{q 3} −→{q 2, q3}. The wordw=b·a is aﬁnite synchronizing word forN 1, thus any inﬁnite word initiated withw is an eventually synchronizing word, such ω ω weakly event asb·a·b . The wordb·a·b is a witness to show thatL ⊂L forN 1. A strongly and weakly synchronizing word, here, isb·a ω. The NFAN 2 depicted in Figure 3.5 is an example to show that there are NFAs for which the inclusionL strongly ⊂L weakly is strict. Thea-transitions in this automaton synchronizes a N inq 0:q i −→q 0 for all0≤i≤2. Theb-transitions inq 1 andq 2 are self-loops; and b {q0} −→{q 1, q2}. The languages of strongly and weakly synchronizing words are

Lstrongly = (a+b) ∗aω andL weakly =L strongly + (a+b) ∗(b·b ∗ ·a·a ∗)ω. ▹

Decision problem(s). For allλ∈ {always, event, strongly, weakly}, the emptiness problem forL λ asks, given an NFAN , whether no inﬁniteλ-synchronizing words exist forN meaning thatL λ =∅.

Similar tofinite synchronization, we can defineλ-synchronization from an initial sub- ω setQ 0 of states. For example, a strongly synchronizing word fromQ 0 is a wordw∈A for which there existsn∈N such that for alli≥n there exists some stateq∈Q such wi thatQ 0 −→{q}. Note that ifQ 0 is a singleton, all infinite words are trivially eventually synchronizing. In this case, we are mostly interested in nontrivial synchronization (when synchronization after at least one input is of interest). For the automatonN 2 described in Example 3.2, since there is a weakly synchronizing word fromQ, there are weakly synchronizing words from all subsetsQ 0 ofQ.

For an NFAN with an initial setQ 0, the set of all inﬁnite runs over an inﬁnite wordw= a0 ·a1 ·a2 ··· can be visualized as an acyclic graph for which there is|Q 0| roots: all statesq 0,j

43 c4 ={q 2, q3}

q3,2 q4,2 q5,2 ···

q0,0 q1,2 q2,3

q3,3 q4,3 q5,3 ···

ω Figure 3.6: The acyclic graph of runs induced by the wordb·a·b over the automatonN 1 described in Example 3.2.

whereq j ∈Q 0. The set of vertices is * wi V={q 0,j |q j ∈Q 0}∪ {qi+1,j |q j ∈S whereQ 0 −→S} i∈N wherew i =a 0 ···a i is the prefix ofw consisting in thefirsti+1 letters. There is an edge ai fromq i,j towardsq i+1,k, denoted byq i,j →q i+1,k, ifq j −→q k,q j, qk ∈Q. For alli∈N, definec i ={q j |q i,j ∈V}, that is the set of states where the automaton may arrive in after inputting thefirsti letters. Thus,c 0 =Q 0 andc 1 =∆(Q 0, a0). In fact, the sequence wi c0c1c2 ··· is nothing else than the sequence of sets wherec 0 =Q 0 andQ 0 −→c i+1. Each infinite path starting in a root, on this acyclic graph is indeed an infinite run induced ω by the wordw. As an example, consider the infinite wordb·a·b over the NFAN 1 in Example 3.2. The acyclic graph of runs starting fromQ 0 ={q 0} is drawn in Figure 3.6.

Now, we can re-formalize the deﬁnitions of inﬁnite synchronizing words for the NFAs. LetN be an NFA andQ 0 be an initial set. For example, a strongly synchronizing word ω fromQ 0 is a wordw∈A for which there existsn∈N such that for alli≥n we have ω |ci|=1; a weakly synchronizing word fromQ 0 is a wordw∈A such that for alln∈N ω there existsi≥n where|c i|=1. For the NFAN 1 in Example 3.2 the wordw=b·a·b is neither strongly nor weakly synchronizing fromQ 0 ={q 0}: because|c i|≥2 for all i≥3 (the acyclic graph induced byw is drawn in Figure 3.6). This wordw is however a (nontrivial) eventually synchronizing word:|c 1|=1.

We introduce two other variants of inﬁnite synchronization for an NFA. In these variants, the states where the automaton arrives in at synchronization’s time is under ques- Q tion. For a target setT , we deﬁne the Boolean functions sub T : 2 →{true, false} and Q memT : 2 →{true, false} where for allc⊆Q, we have sub T (c) = true if∅̸=c⊆T and memT (c) = true if|c|=1 andc⊆T . Intuitively, the function sub T only decides if the input set is a non-empty subset ofT (subset inclusion ofc) though the function mem T decides ifc is a singleton such asc={q}, and ifq∈T (membership ofq).

Definition 3.3 (Infinite synchronizing words into target sets). For a complete NFAN=⟨Q,A,∆⟩ with an initial setQ 0, a target setT andf∈{sub T , memT }, ω an infinite wordw∈A fromQ 0 according tof is

44 • always synchronizing iff(c i) for alli∈N,

• eventually (or event) synchronizing iff(c i) for somei∈N,

• weakly synchronizing if for alln∈N there existsi≥n such thatf(c i),

• strongly synchronizing if there existsn∈N such that for alli≥n we havef(c i).

As an instance, consider the automatonN 1 described in Example 3.2 and the unique ω initial stateq 0. The wordb·a·b where the induced graph of runs is drawn in Figure 3.6, is mem not strongly synchronizing fromq 0 according to the function {q1,q2,q3} but it is strongly sub synchronizing according to {q1,q2,q3}.

Given an NFAN with the initial setQ 0 of states and the target setT , deﬁne the λ-synchronizingω-language forN according tof to be

λ L (f)={the set of all inﬁniteλ-synchronizing words forN fromQ 0 according tof}

whereλ∈ {always, event, strongly, weakly} andf∈{sub T , memT }.

Decision problem(s). The emptiness problem of theλ-synchronizingω-language λ according tof asks, given an NFAN and an initial set of statesQ 0, whetherL (f)=∅.

For the automatonN 2 of Example 3.2 with all possible initial setQ 0 ⊆Q and for all λ mem λ sub λ∈ {event, strongly, weakly}, we haveL ( {q1,q2}) =∅ whereasL ( {q1,q2 })≠ ∅. For the emptiness problem ofλ-synchronizingω-languages according to a functionf, without loss of generality we can assume that the initial set is a singleton set: given an NFAN and an initial setQ 0, we construct a copy ofN with a new initial stateq init from which the successor set on all actions isQ 0 (for eventually synchronizing, we need to consider nontrivial synchronization and for always synchronizing, we need to consider the target set T=T∪{q init}). We, thus, always equip the automaton with an initial stateq init and study synchronization from that unique initial state. λ For all singleton target setsT , we have sub T = memT , thus we haveL (subT ) = λ λ L (memT ). For the sake of simplicity we usually denote byL (q) those sets whenT={q}.

3.2 Synchronization in MDPs

For probabilistic systems, a natural extension of words is the notion of strategy that reacts and chooses actions according to the sequence of states visited along the system execution. In this context, we deﬁne synchronizing problems in MDPs regarding to strategies (and not words). To this aim, we consider a symbolic-outcome of MDPs viewed as generators of sequences of probability distributions over states [KVAK10]. In contrast to the

45 traditional path-outcomes presented in subsection 2.4.2, the symbolic-outcome of MDPs recently has received attention [KVAK10, CKV+11, AAGT12, HKK14]. Let usfirst define two variants of the reachability condition for MDPs: given a set T⊆Q of target states andk∈N, the event of reachingT after exactlyk steps is k ♦ T={q +0q1q2 ···∈ Plays(M)|q k ∈T}, and the event of reachingT withink steps is ≤k j ♦ T= j≤k ♦ T . In the same way of reachability condition, these events are measurable α k provided the strategyα and the initial distributionX 0; thus, the probabilities Pr (♦ T) and Prα(♦≤k T) of the events are uniquely defined. As an example, consider the MDPM described in Example 2.3 on page 21 and the memoryless strategyγ whereγ(q 0) =a and γ(q)=b for all statesq≠ q 0 (defined in Example 2.5 on page 24). LetX 0 be the Dirac distribution onq 0, then fromX 0 γ k 1 k k−1 ω – Pr (♦ {q1}) = 2k for allk≥1: it holds since♦ {q1}=(q 0) q1Q and since on 1 the actiona, the probability thatM stays inq 0 or leavesq 0 to move intoq 1 is 2 p if M is inq 0 with probabilityp. γ ≤k 1 – Pr (♦ {q3}) = 1− 2k−2 for allk≥3. Sinceq 3 is absorbing, whenM arrives inq 3 γ ≤k γ k it can never leave outq 3. Hence, Pr (♦ {q3}) = Pr (♦ {q3}).

Symbolic-outcome of MDPs. LetM be an MDP andX 0 ∈D(Q) be an initial distribution. Under a strategyα, the symbolic-outcome ofM fromX 0 is the infinite sequence α α α α α k M0 M1 M2 ··· of probability distributions defined byM k (q)=Pr (♦ {q}) for allk≥0 α andq∈Q. Hence,M k is the probability distribution over states afterk steps under α strategyα. Note thatM 0 =X 0. As an example, the infinite sequence ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ 1 1 1 1 1 q0 1 ⎜ ⎟ ⎜ 2 ⎟ ⎜ 4 ⎟ ⎜ 8 ⎟ ⎜ 16 ⎟ ⎜ 32 ⎟ ⎜ ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ q1 ⎜ 0 ⎟ ⎜ 2 ⎟ ⎜ 4 ⎟ ⎜ 8 ⎟ ⎜ 16 ⎟ ⎜ 32 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ··· ⎝ ⎠ ⎝ ⎠ ⎝ 1 ⎠ ⎝ 1 ⎠ ⎝ 1 ⎠ ⎝ 1 ⎠ q2 0 0 2 4 8 16 1 3 7 q3 0 0 0 2 4 8 is the symbolic-outcome for the MDPM described in Example 2.3 under the memoryless strategyγ whereγ(q 0) =a andγ(q)=b for all statesq≠ q 0 (mentioned in Example 2.5).

Synchronization. Informally, synchronizing conditions require that the probability of some target state (or some group of target states) tends to1 in the se- α α α quenceM 0 M1 M2 ··· , either always, once, inﬁnitely often, or always after some point. sum max Given a target setT⊆Q, we deﬁne the# functions T :D(Q)→ [0, 1] and T : sum max D(Q)→ [0, 1] that compute T (X)= q∈T X(q) and T (X) = maxq∈T X(q). The function max T (X) is the quantitative version of the function mem(Supp(X)), and sum T (X) is the quantitative version of subT (Supp(X)).

46 Always Eventually α α Sure ∃α∀nM n(T)=1 ∃α∃nM n(T)=1 α α Almost-sure ∃α infn Mn(T)=1 ∃α supn Mn(T)=1 α α Limit-sure supα infn Mn(T)=1 supα supn Mn(T)=1 Weakly Strongly α α Sure ∃α∀N∃n≥NM n(T)=1 ∃α∃N∀n≥NM n(T)=1 α α Almost-sure ∃α lim supn→∞ Mn(T)=1 ∃α lim infn→∞ Mn(T)=1 α α Limit-sure supα lim supn→∞ Mn(T)=1 supα lim infn→∞ Mn(T)=1

α Table 3.1: Winning modes and synchronizing conditions (whereM n(T) denotes the probability that under strategyα, aftern steps the MDPM is in a state ofT).

Deﬁnition 3.4 (Synchronized probability distributions). For the functionf∈ {sumT , max T } andp∈ [0, 1], we say that a probability distributionX isp-synchronized according tof iff(X)≥p.

We say that a sequence X¯=X 0X1 ... of probability distributions is: (a) alwaysp-synchronizing ifX i isp-synchronized for alli≥0; (b) event (or eventually)p-synchronizing ifX i isp-synchronized for somei≥0; (c) weaklyp-synchronizing ifX i isp-synchronized for inﬁnitely manyi’s; (d) stronglyp-synchronizing ifX i isp-synchronized for all butﬁnitely manyi’s.

Forp=1, these deﬁnitions are analogous to the traditional safety, reachability, Büchi, and coBüchi conditions [dAH00]. Similar to traditional semantics, the following winning modes can be considered:

Deﬁnition 3.5 (Synchronizing MDPs). LetM=⟨Q,A,δ⟩ be an MDP with the initial distributionX 0 and letf∈{sum T , max T }. For the {always, eventually, weakly, strongly} synchronizing condition,M is:

• sure winning if there exists a strategyα such that the symbolic-outcome ofα fromX 0 is {always, eventually, weakly, strongly}1-synchronizing according tof; • almost-sure winning if there exists a strategyα such that for allϵ>0 the symbolic- outcome ofα fromX 0 is {always, eventually, weakly, strongly} (1−ϵ)-synchronizing according tof; • limit-sure winning if for allϵ>0, there exists a strategyα such that the symbolic- outcome ofα fromX 0 is {always, eventually, weakly, strongly} (1−ϵ)-synchronizing according tof.

47 We often useX(T) instead of sum T (X), as in Tables 3.1 where the deﬁnitions of the various winning modes and synchronizing conditions forf= sum T are summarized, and we use∥X∥ T instead of max T (X).

Forf∈{sum T , max T } andλ∈ {always, event(ually), weakly, strongly}, the winning λ region ⟨⟨1⟩⟩sure (f) is the set of initial distributions such thatM is sure winning forλ- synchronization (we assume thatM is clear from the context). We deﬁne analogously the λ λ sum winning regions ⟨⟨1⟩⟩almost (f) and ⟨⟨1⟩⟩ limit (f). For a singletonT={q}, we have T = max λ T , and we simply write ⟨⟨1⟩⟩µ(q) whereµ∈ {sure, almost, limit}. Example 3.3. Consider the MDPM described in Example 2.3 on page 21 and the memoryless strategyγ whereγ(q 0) =a andγ(q)=b for all statesq≠ q 0 (deﬁned in Example 2.5 on page 24). The outcome from the initial Dirac distributionq 0 is as follows ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ 1 1 1 1 1 1 q0 1 k ⎜ ⎟ ⎜ 2 ⎟ ⎜ 4 ⎟ ⎜ 8 ⎟ ⎜ 16 ⎟ ⎜ 32 ⎟ ⎜ 2 ⎟ ⎜ ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ q1 0 k ⎜ ⎟ ⎜ 2 ⎟ ⎜ 4 ⎟ ⎜ 8 ⎟ ⎜ 16 ⎟ ⎜ 32 ⎟ ··· ⎜ 2 ⎟ ··· ⎜ ⎟ ⎜ ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎜ 1 ⎟ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ ⎝ − ⎠ q2 0 0 2 4 8 16 2k 1 1 3 7 1 q3 0 0 0 2 4 8 1− 2k−2

γ 1 wherek≥3 is the number of steps. We have sup k∈N Mk(q3) = supk∈N(1− 2k−2 ) = 1 event showing thatq 0 ∈ ⟨⟨1⟩⟩almost (q3). Consider the pure memoryless strategyβ that always playsb in all states:β(q) =b β for all statesq∈Q. The outcome is such that for allk∈N, the probabilityM k (q0) = 1 always fromq 0 which trivially givesq 0 ∈ ⟨⟨1⟩⟩sure (q0). The MDPM is sure winning for always synchronizing condition since in all stepsk, there is a deterministic transition ensuring the β existence of a unique successor, and thus ensuring that the probability distributionM k+1 over states remains a Dirac distribution. By definition, ifq 0 is sure winning for always synchronizing condition in{q 0}, it is sure winning for eventually synchronizing condition event max too. Thus, ⟨⟨1⟩⟩sure ( T )≠ ∅ for the target setT={q 0, q1}. Now, let us modify the MDPM as follows. Theb-transition inq 0 is not deterministic 1 anymore:δ(q 0, b)(q0) =δ(q 0, b)(q1) = 2 (shown in Figure 3.7). We can easily verify that always always q0 ̸∈ ⟨⟨1⟩⟩sure (q0) andq 0 ̸∈ ⟨⟨1⟩⟩sure (q1) neither. This two results are sufficient to prove always max thatq 0 ̸∈ ⟨⟨1⟩⟩sure ( T ) for the target setT={q 0, q1}. However, a pure memoryless always sum strategy that always playsa for all statesq∈Q, is a witness to proveq 0 ∈ ⟨⟨1⟩⟩sure ( T ). ▹ The definitions of synchronization result in following hierarchies:

Remark4. Given an MDPM, for all functionsf∈{sum T , max T }, for allλ∈ {always, event, weakly, strongly}, andµ∈ {sure, almost, limit}, we have: λ λ λ – ⟨⟨1⟩⟩sure (f) ⊆ ⟨⟨1⟩⟩ almost (f) ⊆ ⟨⟨1⟩⟩ limit (f), always strongly weakly event – ⟨⟨1⟩⟩µ (f) ⊆ ⟨⟨1⟩⟩ µ (f) ⊆ ⟨⟨1⟩⟩ µ (f) ⊆ ⟨⟨1⟩⟩ µ (f)

48 1 a, b: 2 a a, b 1 a, b: 2 b a, b q0 q1 q2 q3

event event λ Figure 3.7: An MDPM such that ⟨⟨1⟩⟩ almost (q2)≠ ⟨⟨1⟩⟩ limit (q2) and ⟨⟨1⟩⟩sure (q1)≠ λ ⟨⟨1⟩⟩almost (q1) for allλ∈ {event, weakly, strongly}.

For the always synchronizing condition we will provide robustness result saying that the always always always three winning modes coincide: ⟨⟨1⟩⟩sure (f)= ⟨⟨1⟩⟩ almost (f)= ⟨⟨1⟩⟩ limit (f) (See Lemma 4.1 on page 64). For the eventually synchronizing condition as the least robust synchroniza- event event event tion, we will prove that the inclusions ⟨⟨1⟩⟩sure (f) ⊂ ⟨⟨1⟩⟩ almost (f) ⊂ ⟨⟨1⟩⟩ limit (f) are in general strict (See Lemma 3.1 on page 49). The strong and weak synchronization are more robust than eventually synchronization, but less than always synchronization: we λ λ see that ⟨⟨1⟩⟩sure (f) ⊂ ⟨⟨1⟩⟩ almost (f) whereλ∈ {strongly,weakly}; and we will prove that the almost-sure and limit-sure winning modes coincide (See Lemma 3.1 , Theorem 5.3 and Corollaries 6.1 and 6.2 on pages 49, 100, 119 and 123, respectively.).

Lemma 3.1. For allλ∈ {event, weakly, strongly}, there exists an MDPM and two statesq 1, q2 such that: λ λ (i) ⟨⟨1⟩⟩sure (q1)! ⟨⟨1⟩⟩ almost (q1) and, event event (ii) ⟨⟨1⟩⟩almost (q2)! ⟨⟨1⟩⟩ limit (q2).

Proof. Consider the MDPM with statesq 0, q1, q2, q3 and actions a, b as shown in Figure 3.7. All transitions are deterministic except fromq 0 where on all actions, the successor isq 0 or 1 q1 with probability 2 . The deterministica andb-transitions inq 1, q2, q3 are as follows,

δ(q1, a) =q 1,δ(q 1, b) =q 2 andδ(q 2, c) =δ(q 3, c) =q 3 forc∈{a, b}.

Let the initial distributionX 0 be the Dirac distribution onq 0. strongly weakly event To establish(i), since ⟨⟨1⟩⟩ µ (q1) ⊆ ⟨⟨1⟩⟩µ (q1) ⊆ ⟨⟨1⟩⟩µ (q1) for allµ∈ {sure, strongly event almost}, it is suﬃcient to prove thatX 0 ∈ ⟨⟨1⟩⟩almost (q1) andX 0 ̸∈ ⟨⟨1⟩⟩sure (q1), as it λ λ implies thatX 0 ∈ ⟨⟨1⟩⟩almost (q1) andX 0 ̸∈ ⟨⟨1⟩⟩sure (q1) for allλ∈ {event, weakly, strongly}. strongly To prove thatX 0 ∈ ⟨⟨1⟩⟩almost (q1), consider the pure strategy that always playsa. The 1 outcome is such that the probability to be inq 1 afterk steps is1− 2k , showing thatM is almost-sure winning for the strongly synchronizing condition in the target set{q 1} (from event X0). On the other hand,X 0 ̸∈ ⟨⟨1⟩⟩sure (q1) because for all strategiesα, the probability in α q0 remains always positive, and thus inq 1 we haveM n(q1)<1 for alln≥0, showing that M is not sure winning for the eventually synchronizing condition inq 1 (fromX 0). To establish(ii), for allk≥0 consider a strategy that playsa fork steps, and then 1 playsb. Then the probability to be inq 2 afterk+1 steps is1− 2k , showing that this strategy 1 is eventually (1− 2k )-synchronizing in the target set{q 2}. Hence,M is limit-sure winning for the eventually synchronizing condition inq 2 (fromX 0). Second, for all strategies, since

49 1 1 p1 10 2

p a 1 1 ℓ1 1 2 2 10 a: 1 p 1 5 2 2 ℓ 4 ⇒ 1 a: 5 1 10 2 1 4 4 ℓ2 10 a q 5 5 1 2 q

Figure 3.8: State duplication ensures that the probability mass can never be accumulated in a single state except inq (we omit actiona for readability).

the probability inq 0 remains always positive, the probability inq 2 is always smaller than1. Moreover, if the probabilityp inq 2 is positive aftern steps (p >0), then after any number m > n of steps, the probability inq 2 is bounded by1−p. It follows that the probability in{q 2} is never equal to1 and cannot tend to1 form→∞, showing thatM is not almost-sure winning for the eventually synchronizing condition in{q 2} (fromX 0). !

Decision problem(s). Forλ∈ {always, event, strongly,weakly} andµ∈ {sure, almost, limit}, the membership problem ofλ-synchronization is to decide, given an λ MDPM and an initial probability distributionX 0, whetherX 0 ∈ ⟨⟨1⟩⟩µ(f).

Remark5. Forλ∈ {eventually, weakly} synchronizing condition and each winning mode, we show that the membership problem with function max T is polynomial-time equivalent sum ′ ′ to the membership problem with function + T with a singletonT . First, forµ∈ λ max λ {sure, almost, limit}, we have ⟨⟨1⟩⟩µ( T ) = q∈T ⟨⟨1⟩⟩µ(q), showing that the membership problems for max are polynomial-time reducible to the corresponding membership problem ′ ′ for sumT ′ with singletonT . The reverse reduction is as follows. Given an MDPM , a ′ ′ singletonT ={q} and an initial distributionX 0, we can construct an MDPM and initial ′ λ λ max distributionX 0 such thatX 0 ∈ ⟨⟨1⟩⟩µ(q) if and only ifX 0 ∈ ⟨⟨1⟩⟩µ( T ) whereT=Q is ′ ′ the state space ofM. The idea is to constructM andX 0 as a copy ofM andX 0 where all states exceptq are duplicated, and the initial and transition probabilities are evenly distributed between the copies (see Figure 3.8). Therefore if the probability tends to1 in some state, it has to be in the stateq. Remark6. Forλ∈ {always, event, strongly, weakly}, for the membership problem ofλ- synchronization it is suﬃcient to consider Dirac initial distributions (i.e., assuming that

50 MDPs have a single initial state) because the answer to the general membership problem for an MDPM with initial distributionX 0 can be obtained by solving the membership problem for a copy ofM with a new initial stateq init from which the successor distribution on all actions isX 0. We, thus, always equip the MDP with an initial Dirac distributionq init and study synchronization fromq init. Note that in the case of always synchronizing MDPs, the target set is replaced withT∪{q init}, and in the case of eventually synchronizing MDPs the non-trivial synchronization is of interest.

3.3 Synchronization in PAs

Since words for probabilistic automata can be viewed as blind strategies for MDPs, we define symbolic-runs of PAs as the symbolic outcomes of MDPs with blind strategies. In the context of synchronization, we consider randomized words for PAs. For an infinite randomized wordw=d 0d1d2 ··· whered i ∈D(A) for alli∈N, the w w w symbolic-runP 0 P1 P2 ··· of the PAP from the initial distributionX 0 is defined by w P0 =X 0, and for alln∈N andq∈Q, by ) ) w w ′ ′ Pn+1(q)= Pn (q )·d n(a)·δ(q , a)(q). a∈A q′∈Q

By replacing the path-outcomes with run-outcomes and strategies with randomized words, PAs inherit all definitions of synchronizations (such as sure weakly synchronizing condition) from MDPs. For allλ∈ {always, eventually, strongly, weakly} and f∈{max T , sumT }, we thus define {sure, almost}λ-synchronizing languages according λ λ tof for PAsP, asL sure (f) andL almost (f), that are the set of all randomized words (blind strategies) thatP is sure and almost-sure winning for theλ-synchronizing condition. We λ λ also define the limit-sureL limit (f)λ-synchronizing languages, denoted byL limit (f), that is the set of all families of limit-sure words such that in each family, for allϵ there is a wordw over whichP is (1−ϵ)-synchronizing forλ-synchronizing condition.

Decision problem(s). Forλ∈ {always, event, weakly, strongly} andf∈ {sumT , max T }, the emptiness problem for {sure, almost, limit}λ-synchronizing languages according tof asks, given a PAP and an initial distributionX 0, whether the language is empty.

By Remark 6, without loss of generality we assume thatX 0 is a Dirac distribution on the stateq init.

Example 3.4. Consider the MDP drawn in Figure 3.9 with the initial stateq init. All transitions are deterministic except inq init where on all actions, the successors areq 1 andq 2 each 1 with probability 2 . Inq 1 andq 2, theb-transitions are self-loops whereas thea-transitions leave the states toq 2 andq 3, respectively. The stateq 3 is an absorbing state.

51 b

1 a, b: 2 q1

qinit a a, b

1 a q2 q3 a, b: 2

event Figure 3.9: An MDP such thatq init ∈ ⟨⟨1⟩⟩sure (q2) implying that the MDP is sure eventually synchronizing in{q 2} by some strategy. However, there is no blind strategy under event which the MDP is sure eventually synchronizing in{q 2}, and thusL sure (q2) =∅ fromq init when interpreting the model as a PA.

event We prove thatq init ∈ ⟨⟨1⟩⟩sure (q2) implying that there is some strategy forM to win sure eventually synchronizing condition in{q 2}, and we prove that there is no blind strategy to event win this condition. This implies thatL sure (q2) =∅ fromq init when interpreting the model as a PA. Consider the pure memoryless strategyα that playsa in all states exceptq 2 where it playsb:α(q 2) =b andα(q)=a for all statesq≠ q 2. The symbolic-outcome ofM underα is ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ qinit 1 0 0 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ 1 ⎟ ⎜ ⎟ q1 ⎜ 0 ⎟ ⎜ 2 ⎟ ⎜ 0 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ··· ⎝ ⎠ ⎝ 1 ⎠ ⎝ ⎠ q2 0 2 1

q3 0 0 0 event meaning thatq init ∈ ⟨⟨1⟩⟩sure (q2) in the MDPM. Considering the model as the PAP, theﬁrst input letter brings the automaton inq 1 1 1 orq 2 with equal probability 2 . From now on, ifP always inputsb, the probability 2 inq 1 never leaks down toq 2, andP can never be inq 2 with probability1. On the other hand, 1 as soon asP inputs oncea, the automaton moves fromq 2 toq 3 with probability 2 . Since q3 is absorbing, the automaton never comes back fromq 3 toq 2; thus, the probability inq 2 1 event is never more than 2 andL sure (q2) =∅ fromq init. ▹

3.4 Relation to one-letter alternatingﬁnite automata

LetB +(Q) be the set of positive Boolean formulas overQ, i.e., Boolean formulas built from elements inQ using∧ and∨. A setS⊆Q satisﬁes a formulaϕ∈B +(Q) (denoted S|=ϕ) ifϕ is satisﬁed when replacing inϕ the elements inS by true, and the elements in Q\S by false.

52 q0 δA(q0) =q 0 ∨(q 1 ∧q 2)

q1 q2 δA(q1) =q 2 ∧q 3 δA(q2) =q 2

q3 δA(q3) =q 1 ∧q 2

Figure 3.10: An example of 1L-AFA where the languageL(A q0 ) is notﬁnite.

Definition 3.6 (One-letter alternatingfinite automaton).A one-letter alternatingfinite automaton (1L-AFA) is a tupleA=⟨Q,δ A,F⟩ whereQ is afinite set of states, + δA :Q→B (Q) is the transition function, andF⊆Q is the set of accepting states.

We assume that the formulas in transition function are in disjunctive normal form. Note that the alphabet of the automaton is omitted, as it has a single letter. In the language of a 1L-AFA, only the length of words is relevant. For alln≥0, deﬁne the set Acc A(n,F)⊆Q of states from which the word of lengthn is accepted byA as follows: – AccA(0,F)=F; – AccA(n,F)={q∈Q| Acc A(n−1,F)|=δ(q)} for alln>0. The setL(A q) ={n∈N|q∈ Acc A(n,F)} is the language accepted byA from initial stateq.

Example 3.5. Consider the 1L-AFA depicted in Figure 3.10 which has four statesq 0, q1, q2 andq 3. The transition function is deﬁned as follows. Inq 0, there is a self-loop and another transition going toq 1 andq 2:δ A(q0) =q 0 ∨(q 1 ∧q 2). The stateq 2 has a self-loop. Inq 1 and q3, there are only∧-transitions:δ A(q1) =q 2 ∧q 3 andδ A(q3) =q 1 ∧q 2. LetF={q 2, q3} be the set of accepting states. By deﬁnition, we have AccA(0,{q 2, q3}) ={q 2, q3} and

AccA(1,F)={q 1, q2} and AccA(2,F)={q 0, q2, q3}.

Sinceq 0 has a self-loop, thenq 0 ∈ AccA(n,F) for all next iterationsn≥2. It implies that the languageL(A q0 ) ofA starting inq 0 is notﬁnite. ▹

Q Forﬁxedn, we view Acc A(n,·) as an operator on2 that, given a setF⊆Q computes the set AccA(n,F). Note that Acc A(n,F) = Acc A(1, AccA(n−1,F)) for alln≥1. Denote by PreA(·) the operator AccA(1,·). Then for alln≥0 the operator Acc A(n,·)

53 n coincides with PreA(·), then-th iterate of Pre A(·). For the 1L-AFA of Example 3.5, we 3 4 have PreA({q2, q3}) ={q 0, q1, q2} and PreA({q2, q3}) ={q 0, q2, q3}. For the next iterations, we see that for alln≥1

2n+1 3 2n+2 2 PreA ({q2, q3}) = PreA({q2, q3}) and PreA ({q2, q3}) = PreA({q2, q3}).

0 1 2 Thus, the inﬁnite sequence PreA(F)Pre A(F)Pre A(F)··· is ultimately periodic. This se- n+k quence is ultimately periodic for all 1L-AFAs: there exists n, k∈N such that Pre A (F)= n PreA(F).

3.4.1 Problems for 1L-AFA We present classical decision problems for alternating automata, namely the emptiness andﬁniteness problems, and we introduce a variant of theﬁniteness problem that will be useful to give the complexity of synchronizing problems for (two player games and) MDPs.

Decision problem(s). Given a 1L-AFAA and an initial stateq, the emptiness problem for 1L-AFAs is to decide whetherL(A q) =∅, and thefiniteness problem is to decide whetherL(A q) isfinite. The universalfiniteness problem is to decide, given a 1L-AFAA, whetherL(A q) isfinite for all statesq.

In sequel, we discuss all of these three decision problems.

n Emptiness problem. Given a 1L-AFAA, the sequence Pre A(F) is ultimately periodic: |Q| |Q| k n for alln>2 there existsk≤2 such that PreA(F)= Pre A(F). Therefore, a simple PSPACE algorithm to decide the emptiness problem is to compute on theﬂy the sets n |Q| PreA(F) forn≤2 , and check membership ofq. The emptiness problem can be solved n by checking whetherq∈ Pre A(F) for somen≥0. It is known that the emptiness problem is PSPACE-complete, even for transition functions in disjunctive normal form [Hol95, JS07].

Finiteness problem. This problem can be solved in (N)PSPACE by guessing n, k≤2 |Q| n+k n n such that PreA (F)= Pre A(F) andq∈ Pre A(F). Theﬁniteness problem is PSPACE- complete by a simple reduction from the emptiness problem: from an instance(A, q) of the emptiness problem, construct(A ′, q′) whereq ′ =q andA ′ =⟨Q,δ ′,F⟩ is a copy of A=⟨Q,δ,F⟩ with a self-loop onq (formally,δ ′(q) =q∨δ(q) andδ ′(r) =δ(r) for all ′ r∈Q\{q}). It is easy to see thatL(A q) =∅ if and only ifL(A q′ ) isﬁnite.

Universalfiniteness problem. This problem can be solved in PSPACE by checking n |Q| n m whether PreA(F)=∅ for somen≤2 . Note that if PreA(F)=∅, then Pre A (F)=∅ for allm≥n. Given the PSPACE-hardness proofs of the emptiness andfiniteness problems, it is not easy to see that the universalfiniteness problem is PSPACE-hard. Lemma 3.2. The universalfiniteness problem for 1L-AFA is PSPACE-hard.

54 1L-AFAB

1 ′ c2 A from Figure 3.12 x 1 c1

1 ... C1 c0 C2 Cn

Figure 3.11: Sketch of reduction to show PSPACE-hardness of the universalﬁniteness problem for 1L-AFA.

Proof. The proof is by a reduction from the emptiness problem for 1L-AFA, which is PSPACE-complete [Hol95, JS07]. The language of a 1L-AFAA=⟨Q,δ,F⟩ is non-empty i i ifq 0 ∈ PreA(F) for somei≥0. Since the sequence Pre A(F) is ultimately periodic, it is i |Q| sufficient to compute PreA(F) for alli≤2 . FromA, we construct a 1L-AFAB=⟨Q ′,δ ′,F ′⟩ with setF ′ of accepting states such i ′ i |Q| that the sequence PreB(F ) inB mimics the sequence Pre A(F) inA for2 steps. The ′ i ′ automatonB contains the state space ofA, i.e.,Q⊆Q . The goal is to have PreB(F )∩Q= i |Q| i i PreA(F) for alli≤2 , as long asq 0 ̸∈ PreA(F). Moreover, ifq 0 ∈ PreA(F) for some j ′ i≥0, then Pre B(F ) will containq 0 for allj≥i (the stateq 0 has a self-loop inB), and if i j ′ q0 ̸∈ PreA(F) for alli≥0, thenB is constructed such that Pre B(F ) =∅ for sufficiently largej (roughly forj>2 |Q|). Hence, the language ofA is non-empty if and only if the j ′ sequence PreB(F ) is not ultimately empty, that is if and only if the language ofB is infinite from some state (namelyq 0). The key is to letB simulateA for exponentially many steps, and to ensure that the |Q| simulation stops if and only ifq 0 is not reached within2 steps. We achieve this by definingB as the gadget in Figure 3.11 connected to a modified copyA ′ ofA with the same state space. The transitions inA ′ are defined as follows, wherex is the entry state of the gadget (see Figure 3.12): for allq∈Q let(i)δ B(q)=x∧δ A(q) ifq≠ q 0, and(ii) δB(q0) =q 0 ∨(x∧δ A(q0)). Thus,q 0 has a self-loop, and given a setS⊆Q in the automatonA, ifq 0 ̸∈S, then PreA(S)= PreB(S∪{x}) that is Pre B mimics PreA whenx is in the argument (andq 0 has not been reached yet). Note that ifx ̸∈S (andq 0 ̸∈S), then Pre B(S)=∅, that is unless q0 has been reached, the simulation ofA byB stops. Since we need thatB mimicsA for |Q| ′ ′ i ′ 2 steps, we define the gadget and the setF to ensure thatx∈F and ifx∈ Pre B(F ), i+1 ′ |Q| thenx∈ Pre B (F ) for alli≤2 . 1 2 n In the gadget, the statex has non-deterministic transitionsδ B(x) =c 0 ∨c 0 ∨···∨c 0 ton

55 1L-AFAA ′

q0 q0 δA(q0) =q 1 δB(q0) =q 0 ∨(x∧δ A(q0)) ⇒ q1 x q1 δA(q1) =q 2 ∧q 3 δB(q1) =x∧δ A(q1)

q2 q3 q2 q3 1L-AFAA 1L-AFAA

Figure 3.12: Detail of the copyA ′ obtained fromA in the reduction of Figure 3.11.

i i components with state spaceC i ={c 0, . . . , cpi−1} wherep i is the(i + 1)-th prime number, 1 i i and the transitions δB(c2j) =x∧c j+1 form a loop in each component (i=1, . . . , n). We # n |Q| choosen such that#p n = i=1 pi >2 (taken=|Q|). Note that the number of states in n 2 the gadget is1+ i=1 pi ∈O(n logn) [BS96] and hence the construction is polynomial in the size ofA. i By construction, for all setsS, we havex∈ Pre B(S) whenever theﬁrst statec 0 of some i i componentC i is inS, and ifx∈S, thenc j ∈S impliesc j−1 ∈ PreB(S). Thus, ifx∈S, the operator PreB(S) ‘shifts’ backward the states in each component; and,x is in the next i iteration (i.e.,x∈ Pre B(S)) as long asc 0 ∈S for some componentC i. ′ i Now, deﬁne the set of accepting statesF inB in such a way that+ all statesc 0 disappear # ′ i simultaneously only afterp n iterations. LetF =F∪{x}∪ 1≤i≤n(Ci \{cpi−1}), thus F ′ contains all states of the gadget except the last state of each component. It is easy to i ′ check that, irrespective of the transition relation inA, we havex∈ Pre B(F ) if and only if # i j ′ 0≤i

p n , we have PreB(F ) =∅ for alli>p n . This shows that the language ofA is non-empty if and only if the language ofB is inﬁnite from some state (namelyq 0), and establishes the correctness of the reduction. !

3.4.2 Connection with MDPs. The underlying structure of a Markov decision processM=⟨Q,A,δ⟩ is an alternating graph, where the successorq ′ of a stateq is obtained by an existential choice of an actiona and a universal choice of a stateq ′ ∈ Supp(δ(q, a)). Therefore, it is natural that some questions related to MDPs have a corresponding formulation in terms of alternating automata. We show that such connections exist between synchronizing problems for

i 1. In expressionc j, we assume thatj is interpreted modulop i.

56 a

q0 q0 b: 1 δA(q0) =q 0 ∨(q 1 ∧q 2) 2 b: 1 a 2

1 b: 2 q1 q2 q1 q2 a δA(q1) =q 0 ∨(q 2 ∧q 3) δA(q2) =q 2 ∨q 3 1 b: 2 b 1 1 b: 2 b: 2 q3 a q3 a. δA(q3) =q 3 ∨(q 1 ∧q 2) b.

Figure 3.13: An example of 1L-AFAA with the connected MDPM A.

MDPs and language-theoretic questions for alternating automata, such as emptiness and universalﬁniteness. Given a 1L-AFAA=⟨Q,δ A,F⟩, assume without loss of generality that the transition functionδ A is such thatδ A(q) =c 1 ∨···∨c m has the same number m of conjunctive clauses for allq∈Q. FromA, construct the MDPM A =⟨Q,A,δ M⟩ whereA={a 1, . . . , am} andδ M(q, ak) is the uniform distribution over the states occurring in thek-th clausec k inδ A(q), for allq∈Q anda k ∈A. Then for alln≥0, we have n AccA(n,F) = Pre (F) in the MDPM A. Similarly, from an MDPM and a setT of states, we can construct a 1L-AFAA=⟨Q,δ A,F⟩ withF3=T such4 that for alln≥0 we n ′ have AccA(n,F)= Pre (T) in the MDPM (letδ A(q)= a∈A q′∈post(q,a) q for allq∈Q).

Example 3.6. Consider the 1L-AFAA depicted in Figure 3.13.a which has four statesq 0, q1,q 2 andq 3. The transition function is deﬁned as follows. In all states, the transitions have2 conjunctive clauses. Inq 0, there is a self-loop and another transition going toq 1 andq 2:δ A(q0) =q 0 ∨(q 1 ∧q 2). Inq 1, the transition function isδ A(q1) =q 0 ∨(q 2 ∧q 3). The statesq 2 andq 3, both, have a self-loop and a second transition:δ A(q2) =q 2 ∨q 3 and δA(q3) =q 3 ∨(q 1 ∧q 2). FromA, we construct the MDPM A with two letters a, b and the same four states (see Figure 3.13.b). Thea-transitions inq 0, q2, q3 are all self-loops (the Dirac distribution on itself); anda-transition inq 1 is the Dirac distribution onq 0. The b-transition inq 2 deterministically goes toq 3. Theb-transitions inq 0, q1, q3 are respectively uniform distributions over the successors sets{q 1, q2},{q 2, q3} and{q 1, q2}. ▹ Several decision problems for 1L-AFA can be solved by computing the sequence of sets AccA(n,F) of states from which the word of lengthn is accepted, and we show that some synchronizing problems for MDPs require the computation of the sequence Pren(F) in the MDPM. Therefore, the above relation between 1L-AFA and MDPs establishes bridges that we use in Chapter 4 to transfer complexity results from 1L-AFA to MDPs.

57 3.5 Relation to two-player turn-based games.

Remark 7 follows from the characterizations and reductions presented in Chapters 4, 5 and 6 to establish the upper bound of computational complexity for the synchronizing problems in MDPs . Remark7. For all three winning modes and all membership problems of {always, event, weakly, strongly} synchronization in MDPs, only the support of the probability distributions in the transition function of the system is relevant (i.e., the exact value of the positive probabilities does not matter). As a result, we can encode an MDP as anA-labelled transition system(Q, R) with R⊆Q×A×Q such that(q, a, q ′)∈R is a transition ifq ′ ∈ post(q,a).

TheA-labeled transition system(Q, R) of an MDP can be interpreted as a game arena. A turn-based two player gameG, with Player-1 and adversary Player-2, can be played in rounds. The game starts in an initial stateq init ∈Q. For each round, Player-1 chooses an actiona to play in the current stateq of the game. Next, Player-2 chooses a successor state among the statesq ′ where(q, a, q ′)∈R, and the game moves toq ′. The definitions of paths and prefixes follow from the definitions in MDPs. A pathq 0q1 ··· is an infinite sequence of states whereq 0 =q init and for alli∈N, there exists an actiona such that(q i, a, qi+1)∈R; thefinite prefix of a path is a prefix or simply afinite path. Let Pref(G) be the set of allfinite paths in the game. A strategy for Player-1 is a functionα: Pref(G)→A that, given afinite pathρ, returns an action. A strategy for adversary Player-2 is a functionβ: Pref(G)×A→Q that, given afinite pathρ and an actiona, returns the successor state. Given the strategyα for Player-1, we say that afinite pathq 0q1 ···q n is feasible if there exists some strategyβ for the adversary such thatq 0 =q init and(q i, a, qi+1)∈R wherea=α(q 0q1 ···q i) andq i+1 =β(q 0q1 ···q i, a) for alli∈N. Given a strategyα for Player-1, the sequencec 0c1 ··· of feasible current sets is defined as follows. For alli∈N, let

ci ={Last(ρ)|ρ=q 0q1 ···q i ∈ Pref(G) is a feasible path of lengthi forα}

where Last(ρ) =q i is the last stateq i visited along the pathρ=q 0q1 ···q i. Then,c 0 ={q init} andc 1 = post(qinit, a) wherea=α(q init). Now, we can deﬁne synchronizing winning conditions of Player-1 in gamesG.

Deﬁnition 3.7 (Synchronizing games). LetG be a two-player turn-based game played on⟨Q, R⟩,q init be the initial state,T a target set andf∈{sub T , memT }. The gameG fromq init according tof is

• always synchronizing iff(c i) for alli∈N,

• eventually (or event) synchronizing iff(c i) for somei∈N,

• weakly synchronizing if for alln∈N there existsi≥n such thatf(c i),

58 • strongly synchronizing if there existsn∈N such that for alli≥n we havef(c i)

wherec 0c1c2 ··· is the sequence of feasible current sets for some strategyα of the good player.

The following decision problems are of interest.

Decision problem(s). For allλ∈{always, event, weakly, strongly} and the target setT , theλ-synchronizing problem with functionf∈{max T , sumT } in games asks, given a gameG and an initial stateq init, whetherG isλ-synchronizing fromq init according tof.

As a result of Remark 7, we have Corollary 3.1. Corollary 3.1. Forλ∈ {always, event, weakly, strongly}, theλ-synchronizing problem in turn-based two player games is polynomial-time equivalent with the membership problem of sureλ-synchronization in MDPs. We thus focus on establishing the computation complexities of synchronizing problems only in MDPs, and carry up the results to turn-based two player games.

3.6 Discussions

Unlike for Remark 7 stating that the exact value of transitions does not matter while synchronizing in an MDP, this result in general does not hold for emptiness problem of synchronizing languages in PAs. For instance in Chapter 4, we will establish undecidability result of the emptiness problem for almost-sure eventually synchronizing languages in PAs by a reduction from value1 problem in PAs: the undecidability of value1 is obtained by providing a PA (see Example 6.2 on page 135) where the PA has value1 if and only if 1 the probability of some transition is strictly greater than 2 . However, Remark 8 follows from the characterizations and reductions presented in Chapters 4, 5 and 6 to establish the upper bound of computational complexity of emptiness problem for sure eventually synchronizing languages in PAs. Remark8. For the emptiness problems of sure {always, event, weakly, strongly} synchronizing languages in PAs, only the support of the probability distributions in the transition function of the system is relevant (i.e., the exact value of the positive probabilities does not matter). Since PAs are a generalization of NFAs where non-deterministic choices of successor are resolved with probabilistic distributions and by Remark 8, Corollary 3.2 follows. Corollary 3.2. Forλ∈ {always, event, weakly, strongly}, the emptiness problem forλ- synchronizing languages in NFAs is polynomial-time equivalent with the emptiness problem of sureλ-synchronizing languages in PAs.

59 a, b a, b

a b q1 qinit q2

Figure 3.14: Randomization is necessary to decide the universality problems of synchronizing languages according to max in PAs.

We thus focus on establishing only the computation complexities of emptiness problems for synchronizing languages in PAs, and carry up the results to NFAs. A further observation that can be deduced from the constructions and reductions presented in Chapters 4, 5 and 6 is that in order to decide emptiness problems of synchronizing languages in PAs it is suﬃcient to consider only pure words, and the computational complexities would not be more costly than considering randomized words.

The second classical decision problem for languages that one can study is the universality problem.

Decision problem(s). Forλ∈ {always, event, weakly, strongly} andf∈ {sumT , max T }, the universality problem for {sure, almost}λ-synchronizing languages according tof asks, given a PAP and an initial distributionX 0, whether the language λ ω is universal, e.g. whetherL sure (f)=D(A) .

This problem cannot be considered for limit-sure winning mode in PAs. The universality problems of synchronizing languages in PAs are mostly obtained by easy arguments, we will discuss these problems shortly, in Sections 4.4, 5.3 and 6.3, and give the main ideas and intuitions. Unlike for the emptiness problem, in general, it is not suﬃcient to consider only pure words to decide universality of synchronizing languages. For the max function, Lemma 3.3 provides a PA and target sets where all pure words are {always, eventually, weakly, strongly} synchronizing whereas there is a randomized word that is not {always, eventually, weakly, strongly} synchronizing. An inﬁnite randomized word is the uniformly randomized word over the alphabetA denoted byw u =d 0d1d2 ··· whered i is the uniform distribution overA for alli∈N.

Lemma 3.3. For allλ∈ {event, weakly, strongly} andµ∈ {sure, almost}, there exists a ′ PAP with an initial stateq init and two target setsT={q init, q1, q2} andT ={q 1, q2} such that: ω always max always max (i)A ⊆L µ ( T ) butw u ̸∈L µ ( T ) and,

ω λ max ′ λ max ′ (ii)A ⊆L µ( T ) butw u ̸∈L µ( T ).

60 qinit

b a a, b a, b q3 q2 q1 a, b

Figure 3.15: Randomization is necessary to decide the universality problems of {event, weakly} synchronizing languages according to sum in PAs.

Proof. Consider the PAP with statesq init,q 1 andq 2 and actions a, b as shown in Fig- ure 3.14. All transitions are deterministic; two statesq 1 andq 2 are absorbing and each is reached deterministically fromq init by playing one of the actions: post(qinit, a) ={q 1} and post(qinit, b) ={q 2}. To establish(i), since sure always synchronizing implies almost-sure always synchronizing, it is suﬃcient to show that all pure words are sure always synchronizing whereas the uniformly randomized word is not almost-sure always synchronizing. After reading the ﬁrst letter of an input pure wordw, since post(q init, a) ={q 1} and post(qinit, b) ={q 2}, if ω w∈a·{a, b} thenP is inq 1 with probability1; andP is inq 2 with probability1 otherwise. Since the initial distribution is Dirac and since both statesq 1 andq 2 are absorbing, we see that all pure words are sure always synchronizing according to max T withT={q init, q1, q2}. On the other hand, by inputting the uniformly randomized wordw u both statesq 1 andq 2 1 wu 1 would be reached with probability 2 showing that∥P n ∥= 2 for alln>1. It proves that always max wu ̸∈L almost ( T ).

strongly max ′ weakly max ′ eventually max ′ To establish(ii), sinceL sure ( T )⊆L sure ( T )⊆L sure ( T ) and since sureλ-synchronization implies almost-sureλ-synchronization, it is suﬃcient to show that ω strongly max ′ event max ′ ω λ max ′ A ⊆L sure ( T ) andw u ̸∈L almost ( T ), as it implies thatA ⊆L µ( T ) but λ max ′ wu ̸∈L µ( T ) for allλ∈ {event, weakly, strongly} and allµ∈ {sure, almost}. To prove ω strongly max ′ ω always max A ⊆L sure ( T ), the same argument used to provedA ⊆L sure ( T ) is valid: we only need to relax the condition on the initial distribution. To complete the proof, we see that the uniformly randomized wordw u is not almost-sure eventually synchronizing according to max T ′ again due to the fact that both absorbing statesq 1 andq 2 are reached 1 with probability 2 afterﬁrst input letter. The proof is complete. ! For the sum function, a similar result holds for eventually and weakly synchronization. Lemma 3.4 provides a PA and some target state where all pure words are {eventually, weakly} synchronizing whereas there is a randomized word that is not {eventually, weakly} synchronizing. Lemma 3.4. For allλ∈ {event, weakly} andµ∈ {sure, almost}, there exists a PAP with an initial stateq init such that:

61 ω λ (i)A ⊆L µ(q3), but λ (ii)w u ̸∈L µ(q3).

Proof. Consider the PAP depicted in Figure 3.15 with statesq init,q 1, q2 andq 3 and actions a, b. All transitions are deterministic; in the stateq init thea-transition goes toq 1 and the b-transition goes toq 2, both deterministically. All transitions inq 1 are redirected toq 2. Two statesq 2 andq 3 make a deterministic loop: post(q2, c) ={q 3} and post(q3, c) ={q 2} for all actionsc∈{a, b}. To establish(i), since all sure weakly synchronizing words are sure eventually synchronizing, and since sureλ-synchronization implies almost-sureλ-synchronization, it is suﬃcient to show that all pure words are sure weakly synchronizing in{q 3}. Since all transitions are deterministic, by inputting a pure word the probability mass would stay in ω a single state in all steps. For all wordsw∈a·{a, b} , the PAP is1-synchronized in{q 3} in all even steps; and similarly, it is1-synchronized in{q 3} in all odd steps for all words ω w∈b·{a, b} . Thus, all pure words are sure weakly synchronizing in{q 3}.

To establish(ii), it is suﬃcient to show that the uniformly randomized wordw u is not almost-sure eventually synchronizing in{q 3}. The symbolic-outcome ofP overw u is ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ qinit 1 0 0 0 0 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ 1 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ q1 ⎜ 0 ⎟ ⎜ 2 ⎟ ⎜ 0 ⎟ ⎜ 0 ⎟ ⎜ 0 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ··· ⎝ ⎠ ⎝ 1 ⎠ ⎝ 1 ⎠ ⎝ 1 ⎠ ⎝ 1 ⎠ q2 0 2 2 2 2 1 1 1 q3 0 0 2 2 2

wu 1 One can verify thatP n (q3)≤ 2 for alln∈N, and thusw u is neither almost-sure (and thus sure) eventually nor almost-sure (and thus sure) weakly synchronizing in{q 3}. !

This result, in general, does not hold for the universality of always and strongly synchronizing languages according to sum. Discussions of Chapters 4, 5 and 6 are devoted to the universality problems of synchronizing languages, where we will show that the universality problem of sure and almost-sure always synchronizing languages, are in PTIME. The universality problems of sure eventually, weakly and strongly synchronizing languages are in PSPACE, while for almost-sure eventually, weakly and strongly synchronizing languages, the universality problems are PSPACE-complete.

62 Always and Eventually Synchronizing4 Condition

First sight. In this chapter we study the always and eventually synchronizing conditions in MDPs and PAs. We provide tight complexity bounds of membership problems for each winning modes {sure, almost-sure and limit-sure} in MDPs, and the memory requirement of winning strategies. We also establish the tight complexity bounds for the emptiness problems of always synchronizing languages for PAs, and for the emptiness problems of sure eventually synchronizing language. We prove undecidability results for the emptiness problem of {almost-sure, limit-sure} synchronizing languages in PAs. Table 4.1 and Table 4.2 on pages 67 and 81 summarize the results presented in this chapter.

Contents 4.1 Always synchronizing condition in MDPs and PAs ...... 64 4.2 Eventually synchronizing condition in MDPs ...... 66 4.2.1 Sure eventually synchronization ...... 67 4.2.2 Almost-sure eventually synchronization ...... 69 4.2.3 Limit-sure eventually synchronization ...... 73 4.3 Eventually synchronization in PAs ...... 80 4.3.1 Sure eventually synchronization ...... 81 4.3.2 Almost-sure eventually synchronization ...... 83 4.3.3 Limit-sure eventually synchronization ...... 85 4.4 Discussion ...... 85

63 4.1 Always synchronizing condition in MDPs and PAs

Weﬁrst provide a lemma stating that for always synchronizing condition, the winning regions of the three winning modes {sure, almost-sure, limit-sure} coincide in MDPs. The result also holds in PAs meaning that from an initial state and a target setT , the sure always-synchronizing language is empty if, and only if, the almost-sure (and limit-sure) always-synchronizing language is empty. We also observe that for always synchronization in probabilistic settings, it only matters whether the value of a transition is non-zero (for the sum function) or whether a transition is deterministic (for the max function); deterministic here means there exists a unique successor with probability1 for the transition. By this observation and provided Lemma 4.1 and Corollary 4.1, the solutions for MDPs and PAs carry up to their counterpart non-probabilistic settings, i.e. two player games and NFAs. Results presented in this section are listed in Tables 4.1 and 4.2.

Lemma 4.1. LetT be a set of states for an MDPM. For all functionsf∈{max T , sumT }, always always always we have ⟨⟨1⟩⟩sure (f)= ⟨⟨1⟩⟩ almost (f)= ⟨⟨1⟩⟩ limit (f).

always always Proof. It follows from the definition of winning modes that ⟨⟨1⟩⟩sure (f) ⊆ ⟨⟨1⟩⟩ almost (f)⊆ always always always ⟨⟨1⟩⟩limit (f). Hence it suffices to show that ⟨⟨1⟩⟩ limit (f) ⊆ ⟨⟨1⟩⟩ sure (f), that is for all initial distributionsX 0, ifM is limit-sure always synchronizing fromX 0 (i.e.X 0 ∈ always always ⟨⟨1⟩⟩limit (f)), thenM is sure always synchronizing fromX 0 (i.e.X 0 ∈ ⟨⟨1⟩⟩sure (f)). Forf= max T , considerϵ smaller than the smallest positive probability in the initial distributionX 0 and in the transitions of the MDPM. Then, given an always (1−ϵ)- synchronizing strategyα (consider a blind strategy for the PA), it is easy to show by α induction onk that the distributionsM k are Dirac for allk≥0. Thus,α is an always 1-synchronizing strategy that proves the statement. A similar argument forf= sum T shows that for sufficiently smallϵ, an always (1−ϵ)- β β synchronizing strategyβ must produce a sequenceM 0 M1 ··· of distributions with support β contained inT , meaning thatS k ⊆T whereS k = Supp(Mk ) for allk≥0, until some support repeats in the sequence. Such supports exist becauseQ isfinite, and so there must be two numbersm>n such thatS n =S m. This naturally induces an always 1-synchronizing strategy proving the statement. !

The proof of Lemma 4.1 is valid even if we restrict the controller in the MDPs to use only blind strategies.

Corollary 4.1. LetT be a set of states for a PAP. For all functionsf∈{max T , sumT } and all initial stateq init, the following three propositions are equivalent: always (i)L sure (f)=∅, always (ii)L almost (f)=∅, always (iii)L limit (f)=∅.

64 1 a, b: q1 a, b 2 a

qinit b a q3

1 b a, b: 2 q2

Figure 4.1: The MDP described in Example 4.1 where for the target setT={q init, q1, q2}, always sum always max we haveq init ∈ ⟨⟨1⟩⟩sure ( T ) butq init ̸∈ ⟨⟨1⟩⟩sure ( T ).

Example 4.1. Consider the MDP depicted in Figure 4.1 that has four statesq init, q1, q2, q3 and two actions a, b. From the initial stateq init, botha-transitions andb-transitions have 1 two successorsq 1 andq 2 each with probability 2 . The stateq 3 is an absorbing state and it is deterministically reachable fromq 1 by the actiona and fromq 2 byb. Theb-transition fromq 1 goes toq 2, and thea-transition moves the MDP back fromq 2 toq 1. LetT= {qinit, q1, q2}. Consider the memoryless strategyα that always playsb inq 1 anda inq 2; this always sum strategy is a witness to prove thatq init ∈ ⟨⟨1⟩⟩sure ( T ). On the other hand, we see that α 1 always max ∥Mi ∥T ≤ 2 for all stepsi≥1 and under all strategiesα. Thus,q init ̸∈ ⟨⟨1⟩⟩sure ( T ). For the likewise structured PA, after inputting thefirst letter both statesq 1 andq 2 are 1 assigned with probability 2 . However, by the second letter, no matter if it isa orb, the 1 PA will be 2 -synchronized in{q 3}. This observation and the fact thatq 3 is absorbing imply always sum always max that both the languagesL sure ( T ) andL sure ( T ) are empty fromq init. ▹ Always synchronizing condition in MDPs. Consider an MDPM=⟨Q,A,δ⟩ and the setT of target states. Letα be an always1-synchronizing strategy according to max T and α α α M0 M1 ··· be the outcome. For allk≥0, letq k ∈T be such thatM k (qk) = 1. For allk, α α sinceM k andM k+1 are both Dirac distributions, then there is a deterministic transition fromq k toq k+1 (there exists an actiona such that the unique successor ofq k anda isq k+1; formally post(qk, a) ={q k+1}). In particularX 0 is Dirac, and letq init ∈T be such that X0(qinit) = 1. It follows that there is an infinite path fromq init in the digraph⟨T,E⟩ where (q, q′)∈E if there exists an actiona∈A such that post(q,a) ={q ′}. The existence of this path entails that there is a loop reachable fromq init in the digraph⟨T,E⟩, and this naturally defines a sure-winning always synchronizing strategy inM. Thus, to decide the membership problem for always synchronizing according to max T (for all winning modes), it is sufficient to check the existence of an infinite path staying inT in the digraph⟨T,E⟩, which can be decided in PTIME. Note that for the functions max T , pure memoryless strategies are sufficient. It follows from the proof of Lemma 4.1 that the winning region for always synchronizing according to sumT coincides with the set of winning initial distributions for the safety condition!T in the traditional semantics in MDPs, which can also be computed in PTIME [CH12]. Thus, for the functions sum T , pure memoryless strategies are sufficient.

65 Theorem 4.1. For always synchronizing condition in MDPs and for all functions f∈{max T , sumT }: 1. (Complexity). The membership problem is in PTIME. 2. (Memory). Pure memoryless strategies are suﬃcient.

Always synchronizing condition in PAs. We have just seen that in MDPs, to decide always synchronizing condition according to max T , it is sufficient tofind an infinite path q0q1q2 ··· such that there is a deterministic transition fromq k toq k+1 for allk≥0 and q0 is the state where the initial distribution is the Dirac distribution onq 0 (qinit =q 0). Such transitions give an actiona k such that post(qk, ak) ={q k+1} for allk≥0. A blind strategyα that playsa k for allk∈N is thus always synchronizing according to max T . It means that pure blind strategies are sufficient to decide the membership problem of always synchronizing condition according to max T . It follows that the always synchronizing language according to max T from an initial stateq init for the PAP is not empty if and only ifq init is winning for the always synchronizing condition in the likewise structured MDPM. As a result, the emptiness problem of always synchronizing language in PAs according to max T can be decided in PTIME. By a similar argument to the case for MDPs, the always synchronizing language according to sumT in PAs is equal to theω-language with safety condition!T in the traditional semantics in PAs, where the emptiness problem is PSPACE-complete [CT12].

Theorem 4.2. For always synchronizing language in PAs:

1. The emptiness problem forf= max T is in PTIME.

2. The emptiness problem forf= sum T is PSPACE-complete.

4.2 Eventually synchronizing condition in MDPs

In this section, we show the PSPACE-completeness of the membership problem for eventually synchronizing conditions in the three winning modes. By Remarks 5 and 6 (on page 50), we consider the membership problem with function sum and Dirac initial distributions (i.e., single initial state). The presented algorithms to solve the membership event sum problem for ⟨⟨1⟩⟩µ ( T ) for arbitraryT andµ∈ {sure, almost, limit}, can be used for event max the special case of singletonT , which solves the membership problem for ⟨⟨1⟩⟩ µ ( T ). Furthermore, we establish PSPACE-completeness of the problems, even in the special case whenT is a singleton, which implies PSPACE-completeness of the membership problems event max for ⟨⟨1⟩⟩µ ( T ) too. Results presented in this section are listed in Table 4.1.

66 Always Eventually Complexity Required memory Complexity Required memory Sure PSPACE-complete exponential Almost-sure PTIME memoryless PSPACE-complete inﬁnite Limit-sure PSPACE-complete unbounded

Table 4.1: Computational complexity of the membership problem of always and eventually synchronization in MDPs, and memory requirement for the winning strategies (for always synchronizing, the three modes coincide).

4.2.1 Sure eventually synchronization Given a target setT in an MDP, the membership problem for sure-winning eventually synchronizing condition inT 1 can be solved by computing the sequence Pren(T) of iterated predecessors. A stateq init is sure-winning for eventually synchronizing inT if and only if n qinit ∈ Pre (T) for somen≥0.

Lemma 4.2. LetM be an MDP andT be a target set. For all statesq init, we have event sum n qinit ∈ ⟨⟨1⟩⟩sure ( T ) if and only if there existsn≥0 such thatq init ∈ PreM(T).

Proof. We prove the following equivalence by induction (on the lengthi): for all initial statesq init, there exists a strategyα sure-winning ini steps fromq init (i.e., such that α i Mi (T)=1) if and only ifq init ∈ Pre (T). The casei=0 trivially holds since for all α strategiesα, we haveM 0 (T)=1 if and only ifq init ∈T. Assume that the equivalence holds for alli

1. We sometimes refer to synchronizing condition according to sum T by synchronizing condition inT.

67 1 a: 2 , b

1 a, b: q1 1 a, b 2 a: 2

qinit q3

1 a, b a, b: 2 q2

Figure 4.2: The MDP described in Example 4.2 where for the target setT={q 1, q3}, we event sum event haveq init ∈ ⟨⟨1⟩⟩sure ( T ) butq init ̸∈ ⟨⟨1⟩⟩sure (q2).

Example 4.2. Consider the MDPM shown in Figure 4.2 that has four statesq init, q1, q2, q3 and two actions a, b. From the initial stateq init, botha-transitions andb-transitions have 1 two successorsq 1 andq 2 each with probability 2 . Theb-transition on the stateq 1 is a self- loop, and thea-transition is a uniform distribution on{q 1, q3}. Thea andb-transitions inq 2 are both the Dirac distribution onq 3; andq 3 is an absorbing state. For the target n set{q 2}, the predecessor is Pre ({q2}) =∅ for alln≥1. The MDPM is not sure eventually synchronizing in{q 2} fromq init. For the target setT={q 1, q3}, the predecessor sequence is ω {q1, q3}{q1, q2, q3}({qinit, q1, q2, q3}) . 2 event sum We see thatq init ∈ Pre (T) which givesq init ∈ ⟨⟨1⟩⟩sure ( T ). ▹ By Lemma 4.2, the membership problem for sure eventually synchronizing is equivalent to the emptiness problem of 1L-AFA, and thus PSPACE-complete. Moreover ifq init ∈ n PreM(T), aﬁnite-memory strategy withn modes that at modei in a stateq plays an actiona such that post(q,a)⊆ Pre i−1(T) is sure winning for eventually synchronizing. There exists a family of MDPsM n (n∈N) over alphabet{a, b} that are sure winning for eventually synchronization, and where the sure winning strategies require exponential memory. The MDPM 2 is shown in Figure 4.3. The structure ofM n consists of an initial stateq init,n componentsH 1,...,Hn and two statesq T andq ⊥. The goal of construction is to be sure eventually synchronizing in the target set{q T }. Each componentH i is a i i i cycle of lengthp i, thei-th prime number. We name the states ofH i withq 1q2 ···q pi i i where the initial state in this cycle isq 1 and the last state isq pi . On all actions inq init, the i probabilistic transition is a uniform distribution on the initial statesq 1 of then components i i H1,...,Hn. On actiona, the next state in the cycleH n is reached: post(qj, a) ={q j+1} for i i all1≤j

2 while 2 the size ofM n is inO(n logn) [BS96]. It can be proved by standard pumping arguments # that no strategy of size smaller thanp n is sure winning. The following theorem summarizes the results for sure eventually synchronizing in MDPs.

68 a H1 1 1 q1 q2 a, b: 1 a 2 b a, b

a, b qinit qT q⊥ H2 a 2 q2 1 a, b: 2 2 q1 a b

2 a q3

Figure 4.3: The MDPM 2. All not-drawn transitions are directed toq ⊥, for example 1 2 b-transitions inq 1 andq 1.

Theorem 4.3. For sure eventually synchronizing condition in MDPs and for all functionsf∈{max T , sumT }: 1. (Complexity). The membership problem is PSPACE-complete. 2. (Memory). Exponential memory is necessary and suﬃcient for both pure and randomized strategies, and pure strategies are suﬃcient.

4.2.2 Almost-sure eventually synchronization This subsection starts with an example of almost-sure eventually synchronizing MDPs where infinite memory is necessary to win the almost-sure eventually synchronizing condition. Example 4.3 describes this MDP and provide a winning strategy that uses infinite memory to win almost-sure eventually synchronizing condition; next in Lemma 4.3, we prove that nofinite-memory strategy wins this condition.

Example 4.3. Consider the MDP in Figure 4.4 with three statesq init, q1, q2 and two actions a, b. The only probabilistic transitions are in the initial stateq init where on both 1 actionsa andb, the next successor is itself orq 1 each with probability 2 . Thea-transition inq 1 is a self-loop and theb-transition goes toq 2 where all transitions are determinis- event tically directed toq init. We are interested to study whetherq init ∈ ⟨⟨1⟩⟩almost (q2). To this aim, we construct a strategy that is almost-sure eventually synchronizing in{q 2}, showing event thatq init ∈ ⟨⟨1⟩⟩almost (q2) and suggesting that winning strategies for almost-sure eventually synchronization may require inﬁnite memory. First, observe that for allϵ>0 we can have

69 a, b

1 a, b: 2 b qinit q1 q2

1 a a, b: 2 Figure 4.4: An MDP where inﬁnite memory is necessary for almost-sure eventually synchronizing strategies.

probability at least1−ϵ inq 2 afterﬁnitely many steps: playingn timesa and thenb leads 1 to probability1− 2n inq 2. Thus the MDP is limit-sure eventually synchronizing in the target set{q 2}. Moreover the remaining probability mass is inq init andq 1; which we recall with the MDP being limit-sure eventually synchronizing in{q 2} with support{q init, q1, q2}. It turns out that from any (initial) distribution with support{q init, q1, q2}, the MDP is again limit-sure eventually synchronizing in the target set{q 2}, and with support in{q init, q1, q2} (by strategies with the same logic: playingﬁnitely many consecutivea and thenb). There- fore we can take a smaller value ofϵ and play a strategy to have probability at least1−ϵ inq 2, and repeat this forϵ→0. This strategy ensures almost-sure eventually synchronizing in the target set{q 2}. ▹

The next Lemma shows that inﬁnite memory is indeed necessary for almost-sure eventually synchronizing strategies for the MDP of Example 4.3.

Lemma 4.3. There exists an almost-sure eventually synchronizing MDP for which all almost-sure eventually synchronizing strategies require inﬁnite memory.

Proof. Consider the MDPM of Example 4.3 which is shown in Figure 4.4. We argued in event Example 4.3 thatq init ∈ ⟨⟨1⟩⟩almost (q2) and we now show that infinite memory is necessary for almost-sure eventually synchronization in{q 2} fromq init. Assume towards contradiction that there exists afinite-memory strategyα that is almost-sure eventually synchronizing in the target set{q 2}. Consider the Markov chain M(α) obtained by the product of the MDPM with thefinite-state transducer definingα as explained in Section 2.2.3. To simply the notation, byX 0 X1 X2 ··· we denote the sequence of probability distributions generated byM(α). We know that the probability Xk((m, q)) in transient states(m, q) vanishes fork→∞ [FV97, Nor98, Ser13]. A state (m, q) inM(α) is called aq-state. Sinceα is almost-sure eventually synchronizing (but is not sure eventually synchronizing) inq 2, there is aq 2-state in the set of recurrent states ofM(α). Since on all actionsq init is a successor ofq 2, andq init is a successor of itself, it follows that there is a recurrentq init-state inM(α), and that all periodic supports of recurrent states inM(α) contain aq init-state. Hence, in each stationary distribution there

70 is aq init-state with a (strictly) positive probability, and therefore the probability mass in qinit is bounded away from zero. It follows that the probability mass inq 2 is bounded away from1 thusα is not almost-sure eventually synchronizing inq 2, a contradiction. !

It turns out that in general, almost-sure eventually synchronizing strategies can be constructed from a family of limit-sure eventually synchronizing strategies if we can also ensure that the probability mass remains in the winning region (as in the MDP of Example 4.3). We present a characterization of the winning region for almost-sure winning based on an extension of the limit-sure eventually synchronizing condition with exact support. This condition requires to ensure probability arbitrarily close to1 in the target setT , and moreover that after the same number of steps the support of the probability distribution is event sum contained in a given setU. Formally, given an MDPM, let ⟨⟨1⟩⟩ limit ( T ,U) forT⊆U be the set of all initial distributions such that for allϵ>0 there exist a strategyα and α α α n∈N such thatM n(T)≥1−ϵ andM n(U)=1 (i.e. the support ofM n is contained inU). We say thatα is limit-sure eventually synchronizing inT with support inU. We will present an algorithmic solution to limit-sure eventually synchronizing condition with exact support in Section 4.2.3. Our characterization of the winning region for almost- sure winning is as follows.

Lemma 4.4. LetM be an MDP andT be a target set. For all statesq init, we have event sum qinit ∈ ⟨⟨1⟩⟩almost ( T ) if and only if there exists a setU such that: event sum –q init ∈ ⟨⟨1⟩⟩sure ( U ), and event sum –X U ∈ ⟨⟨1⟩⟩limit ( T ,U) whereX U is the uniform distribution overU.

event sum Proof. First, ifq init ∈ ⟨⟨1⟩⟩almost ( T ), then there is a strategyα such that α α α supn∈N Mn(T)=1. Then eitherM n(T)=1 for somen≥0, or lim sup n→∞ Mn(T)=1. α IfM n(T)=1, thenq init is sure winning for eventually synchronizing inT , thus event sum qinit ∈ ⟨⟨1⟩⟩sure ( T ) and we can takeU=T . Otherwise, for alli>0 there exists α −i ni ∈N such thatM ni (T)≥1−2 , and moreovern i+1 > ni for alli>0. Let α α si = Supp(Mni ) be the support ofM ni . Since the state space isﬁnite, there is a setU that occurs inﬁnitely often in the sequences 0s1 ... , thus for allk>0 there existsm k ∈N such α −k α thatM mk (T)≥1−2 andM mk (U)=1. It follows thatα is sure eventually synchro- event sum nizing inU fromq init, henceq init ∈ ⟨⟨1⟩⟩sure ( U ). MoreoverM with initial distribution α X1 =M m1 is limit-sure eventually synchronizing inT with exact support inU. Since event sum Supp(X1) =U= Supp(X U ), it follows by Corollary 4.3 thatX U ∈ ⟨⟨1⟩⟩limit ( T ,U). event sum To establish the converse, note that sinceX U ∈ ⟨⟨1⟩⟩limit ( T ,U), it follows from Corollary 4.3 that from all initial distributions with support inU, for allϵ>0 there αϵ αϵ exists a strategyα ϵ and a positionn ϵ such thatM nϵ (T)≥1−ϵ andM nϵ (U)=1. We construct an almost-sure limit eventually synchronizing strategyα as follows. Since event sum qinit ∈ ⟨⟨1⟩⟩sure ( U ), play according to a sure eventually synchronizing strategy from −i qinit until all the probability mass is inU. Then fori=1,2,... andϵ i = 2 , repeat the

71 MDPN MDPM MDPM q ˆq ⇒ q ˆq # #

A′ sink qT A′

Figure 4.5: Sketch of the reduction to show PSPACE-hardness of the membership problem for almost-sure eventually synchronizing in MDPs. following procedure: given the current probability distribution, select the corresponding strategyα ϵi and play according toα ϵi forn ϵi steps, ensuring probability mass at least 1−2 −i inT , and since after that the support of the probability mass is again inU, play α according toα ϵi+1 forn ϵi+1 steps, etc. This strategyα ensures that sup n∈N Mn(T)=1 event sum fromq init, henceq init ∈ ⟨⟨1⟩⟩almost ( T ). !

Note that from Lemma 4.4, it follows that counting strategies are suﬃcient to win almost-sure eventually synchronizing condition (a strategy is counting ifα(ρ) =α(ρ ′) for all preﬁxesρ,ρ ′ with the same length and Last(ρ) = Last(ρ′)). As we show in Section 4.2.3 that the membership problem for limit-sure eventually synchronizing with exact support can be solved in PSPACE, it follows from the characterization in Lemma 4.4 that the membership problem for almost-sure eventually synchronizing is in PSPACE, using the following (N)PSPACE algorithm: guess the setU, and check that event sum event sum qinit ∈ ⟨⟨1⟩⟩sure ( U ), and thatX U ∈ ⟨⟨1⟩⟩limit ( T ,U) whereX U is the uniform distribution overU (this can be done in PSPACE by Theorem 4.3 and Theorem 4.5). We present a matching lower bound. event sum Lemma 4.5. For MDPs, the membership problem for ⟨⟨1⟩⟩almost ( T ) is PSPACE-hard even ifT is a singleton.

Proof. The proof is by a reduction from the membership problem for sure eventually synchronization, which is PSPACE-complete by Theorem 4.3. Given an MDPM=⟨Q,A,δ⟩, ′ ′ ′ an initial stateq init ∈Q, and a stateˆq∈Q, we construct an MDPN=⟨Q ,A ,δ ⟩ and a ′ event event stateq T ∈Q such thatq init ∈ ⟨⟨1⟩⟩sure (ˆq) inM if and only ifqinit ∈ ⟨⟨1⟩⟩almost (qT ) inN . The MDPN is a copy ofM with two new statesq T and sink reachable only by a new action# ′ ′ (see Figure 4.5). Formally,Q =Q∪{q T , sink} andA =A∪{#}, and the transition func- tionδ ′ is deﬁned as follows, for allq∈Q:δ ′(q, a) =δ(q, a) for alla∈A,δ ′(q, #)(sink) = 1 if ′ ′ ′ ′ q≠ ˆq, andδ (ˆq, #)(Tq) = 1;ﬁnally, for alla∈A , letδ (qT , a)(sink) =δ (sink, a)(sink) = 1. The goal is thatN simulatesM until the action# is played inˆq to move the probability mass fromˆq toq T , ensuring that ifM is sure-winning for eventually synchronizing in{ˆq},

72 thenN is also sure-winning (and thus almost-sure winning) for eventually synchronizing in{q T }. Moreover, the only way to be almost-sure eventually synchronizing in{q T } is to have probability1 inq T at some point, because the stateq T is transient under all strategies, thus the probability mass cannot accumulate and tend to1 inq T in the long run. It follows that (from all initial statesq init)M is sure-winning for eventually synchronizing in{ˆq} if and only ifN is almost-sure winning for eventually synchronizing in{q T }. It follows from this reduction that the membership problem for almost-sure eventually synchronizing condition is PSPACE-hard. !

The results of this section are summarized as follows.

Theorem 4.4. For almost-sure eventually synchronizing condition in MDPs and for all functionsf∈{max T , sumT }: 1. (Complexity). The membership problem is PSPACE-complete. 2. (Memory). Inﬁnite memory is necessary in general for both pure and randomized strategies, and pure strategies are suﬃcient.

4.2.3 Limit-sure eventually synchronization In this subsection, we present the algorithmic solution for limit-sure eventually synchronizing with exact support in MDPs. Note that the limit-sure eventually synchronizing condition is a special case where the support is the state space of the MDP. Example 4.4. Consider the MDP in Figure 3.7 which is limit-sure eventually synchroniz- i ing in{q 2}, as shown in Lemma 3.1. Fori=0,1,... , the sequence Pre (T) of predecessors 0 i ofT={q 2} is ultimately periodic: Pre (T)={q 2}, and Pre (T)={q 1} for alli≥1. Givenϵ>0, a strategy to get probability1−ϵ inq 2 first accumulates probability mass in the periodic subsequence of predecessors (here{q 1}), and when the probability mass is greater than1−ϵ inq 1, the strategy injects the probability mass inq 2 (through the aperiodic prefix of the sequence of predecessors). ▹ In general, the typical shape of a limit-sure eventually synchronizing strategy is as explained in Example 4.4. To have probability arbitrarily close to1 in a target setT, a limit-sure eventually synchronizing strategy accumulates the probability mass in the periodic subsequence of predecessors until the probability mass is as large as demanded; and then the strategy synchronously moves the probability mass into the target setT via the prefix subsequence of predecessors. Note that in this scenario, the MDP is also limit- sure eventually synchronizing in every set Prei(T) of the sequence of predecessors. A special

73 case is when it is possible to get probability1 in the sequence of predecessors afterﬁnitely many steps. In this case, the probability mass injected inT is1 and the MDP is even sure- winning. The algorithm for deciding limit-sure eventually synchronization relies on the above characterization, generalized in Lemma 4.6 to limit-sure eventually synchronizing with exact support, saying that limit-sure eventually synchronization inT with support inU is equivalent to either limit-sure eventually synchronization in Pre k(T) with support in Prek(U) (for arbitraryk), or sure eventually synchronizing inT (and therefore also inU).

Lemma 4.6. Given an MDPM, for allT⊆U and allk≥0, we have event sum event sum event sum k ⟨⟨1⟩⟩limit ( T ,U) = ⟨⟨1⟩⟩ sure ( T ) ∪ ⟨⟨1⟩⟩limit ( R,Z) whereR= Pre (T) and Z= Pre k(U).

event sum event sum Proof. The proof is in two parts. First we show that ⟨⟨1⟩⟩sure ( T )∪⟨⟨1⟩⟩limit ( R,Z)⊆ event sum event sum ⟨⟨1⟩⟩limit ( T ,U): sinceT⊆U, it follows from the deﬁnitions that ⟨⟨1⟩⟩ sure ( T )⊆ event sum event sum event sum ⟨⟨1⟩⟩limit ( T ,U); to show that ⟨⟨1⟩⟩ limit ( R,Z) ⊆ ⟨⟨1⟩⟩limit ( T ,U) in an MDPM, letϵ>0 and consider an initial distributionX 0 and a strategyα such that for some α α i≥0 we haveM i (R)≥1−ϵ andM i (Z)=1. We construct a strategyβ that plays likeα for theﬁrsti steps, and then sinceR= Pre k(T) andZ= Pre k(U) plays from states inR according to a sure eventually synchronizing strategy with targetT , and from states inZ\R according to a sure eventually synchronizing strategy with the target setU (such strategies exist by the proof of Lemma 4.2). The strategyβ ensures fromX 0 that β β Mi+k(T)≥1−ϵ andM i+k(U)=1, showing thatM is limit-sure eventually synchronizing inT with support inU.

event sum Second we show the converse inclusion, namely that ⟨⟨1⟩⟩limit ( T ,U)⊆ event sum event sum ⟨⟨1⟩⟩sure ( T ) ∪ ⟨⟨1⟩⟩limit ( R,Z). Consider an initial distributionX 0 ∈ event sum 1 ⟨⟨1⟩⟩limit ( T ,U) in the MDPM and forϵ i = i (i∈N) letα i be a strategy and αi αi ni ∈N such thatM ni (T)≥1−ϵ i andM ni (U)=1. We consider two cases.(a) If the set{n i |i≥0} is bounded, then there exists a numbern that occurs infinitely often in the sequence(n i)i∈N, and such that for alli≥0, there exists a strategyβ i such that βi βi Mn (T)≥1−ϵ i andM n (U)=1. Sincen isfixed, we can assume w.l.o.g. that the strategiesβ i are pure, and since there is afinite number of pure strategies over paths of length at mostn, it follows that there is a strategyβ that occurs infinitely often β among the strategiesβ i and such that for allϵ>0 we haveM n(T)≥1−ϵ, hence β Mn(T)=1, showing thatM is sure winning for eventually synchronization inT , that event sum isX 0 ∈ ⟨⟨1⟩⟩sure ( T ).(b) otherwise, the set{n i |i≥0} is unbounded and we can assume w.l.o.g. thatn i ≥k for alli≥0. We claim that the family of strategiesα i ensures limit-sure synchronization inR= Pre k(T) with support inZ= Pre k(U). Essentially this is because if the probability inT is close to1 aftern i steps, thenk steps before k the probability in Pre (T) must be close to1 as well. Formally, we show thatα i is such αi ϵ αi thatM ni−k(R)≥1− ηk andM ni−k(Z)=1 whereη is the smallest positive probability αi ϵ in the transitions ofM. Towards contradiction, assume thatM ni−k(R)<1− ηk , then αi ϵ Mni−k(Q\R)> ηk and from every stateq∈Q\R, no matter which sequence of actions is

74 played byα i for the nextk steps, there is a path fromq to a state outside ofT , thus with k ϵ k probability at leastη . Hence the probability inQ\T aftern i steps is greater than ηk ·η , αi and it follows thatM ni (T)<1−ϵ, in contradiction with the definition ofα i. This shows αi ϵ thatM ni−k(R)≥1− ηk , and an argument analogous to the proof of Lemma 4.2 shows αi event sum thatM ni−k(Z)=1. It follows thatX 0 ∈ ⟨⟨1⟩⟩limit ( R,Z) and the proof is complete. ! Thanks to Lemma 4.6, since membership problem of sure eventually synchronization is already solved in Section 4.2.1, it suffices to solve the membership problem of limit-sure eventually synchronization in the target setR= Pre k(T) and supportZ= Pre k(U) with arbitraryk, instead ofT andU. We can choosek such that both Pre k(T) and Pre k(U) lie in the periodic part of the sequence of pairs of predecessors(Pre i(T), Pre i(U)). We can assume thatk≤3 |Q| since Prei(T)⊆ Pre i(U)⊆Q for alli≥0. For such value ofk the limit-sure problem is conceptually simpler: once some probability is injected in R= Pre k(T), it can loop through the sequence of predecessors and visitR infinitely often (everyr steps, wherer≤3 |Q| is the period of the sequence of pairs of predecessors). It follows that if a strategy ensures with probability1 that the setR can be reached byfinite paths whose lengths are congruent modulor, then the whole probability mass can indeed synchronously accumulate inR in the limit. Therefore, limit-sure eventually synchronization inR reduces to standard limit-sure reachability condition♦R and the additional requirement that the numbers of steps at which the target set is reached be congruent modulor. In the case of limit-sure eventually synchronization with support inZ, we also need to ensure that no mass of probability leaves the sequence Prei(Z). In a stateq∈ Pre i(Z), we say that an actiona∈A isZ-safe at positioni if 2 post(q,a)⊆ Pre i−1(Z). In statesq ̸∈ Pre i(Z) there is noZ-safe action at positioni. To encode the above requirements, we construct an MDPM Z ×[r] that allows only Z-safe actions to be played (and then mimics the original MDP), and tracks the position (modulor) in the sequence of predecessors, thus simply decrementing the position on each transition since all successors of a stateq∈ Pre i(Z) on aZ-safe action are in Pre i−1(Z).

′ ′ Formally, ifM=⟨Q,A,δ⟩ thenM Z ×[r]=⟨Q ,A,δ ⟩ where –Q ′ =Q×{r−1,...,1,0}∪{sink}; intuitively, we expect thatq∈ Pre i(Z) in the reachable states⟨q, i⟩ consisting of a stateq ofM and a positioni in the predecessor sequence; –δ ′ is deﬁned as follows (assuming an arithmetic modulor on positions) for all⟨q, i⟩ ∈ Q′ anda∈A: ifa is aZ-safe action inq at positioni, thenδ ′(⟨q, i⟩, a)(⟨q′, i−1⟩) = δ(q, a)(q′), otherwiseδ ′(⟨q, i⟩, a)(sink) = 1 (and sink is absorbing).

Note that the size of the MDPM Z ×[r] is exponential in the size ofM (sincer is at most3 |Q|).

2. Since Prer(Z)=Z and Pre r(R)=R, we assume a modular arithmetic for exponents of Pre, that is mod 1 1 Prex(·) is deﬁned as Prex r (·). For example Pre− (Z) is Prer− (Z).

75 Lemma 4.7. LetM be an MDP andR⊆Z be two sets of states such that Pre r(R)=R r and Pre (Z)=Z wherer>0. Then a stateq init is limit-sure eventually synchronizing in event sum R with support inZ(q init ∈ ⟨⟨1⟩⟩limit ( R,Z)) if and only if there exists0≤t

Proof. For theﬁrst direction of the lemma, assume thatq init is limit-sure eventually syn- β chronizing inR with support inZ, and forϵ>0 letβ be a strategy such thatM k (Z)=1 β andM k (R)≥1−ϵ for some numberk of steps. Let0≤t≤r such thatt=k modr and letR 0 =R×{0}. We show that from initial state(q init, t) the strategyα in MZ ×[r] that mimics (copies) the strategyβ is limit-sure winning for the reachability condition♦R 0: it follows from Lemma 4.2 thatα plays onlyZ-safe actions, and since α α k β P r (♦R0)≥Pr (♦ R0) =M k (R)≥1−ϵ, the result follows.

For the converse direction, letR 0 =R×{0} and assuming that there exists0≤t0, we construct fromα a pure strategyβ inM that is (1−ϵ)-synchronizing inR with support inZ. Given afinite pathρ=q 0a0q1a1 . . . qn inM (withq 0 =q init), there ′ is a corresponding pathρ =⟨q 0, k0⟩a0⟨q1, k1⟩a1 ...⟨q n, kn⟩ inM Z ×[r] wherek 0 =t and ki+1 =k i −1 for alli≥0. Since the sequencek 0, k1,... is uniquely determined fromρ, there is a clear bijection between the paths inM and the paths inM Z ×[r] that we often kn omit to apply and mention. Define the strategyβ as follows: ifq n ∈ Pre (R), then there kn−1 exists an actiona such that post(q n, a)⊆ Pre (R) and we defineβ(ρ) =a, otherwise letβ(ρ) =α(ρ ′). Thusβ mimicsα (thus playing onlyZ-safe actions) unless a stateq is reached at stepn such thatq∈ Pre t−n(R), and thenβ switches to always playing actions that areR-safe (and thus alsoZ-safe sinceR⊆Z). We now prove thatβ is limit-sure eventually synchronizing in target setR with support inZ. First sinceβ plays onlyZ-safe actions, it follows for allk such thatt−k=0 (modulor), all states reached fromq init with β positive probability afterk steps are inZ. HenceM k (Z)=1 for all suchk. β Second, we show that givenϵ>0 there existsk such thatt−k=0 andM k (R)≥1−ϵ, β thus alsoM k (Z)=1 andβ is limit-sure eventually synchronizing in target setR with α α ≤k support inZ. To show this, recall that Pr (♦R0) = 1, and therefore Pr (♦ R0)≥1−ϵ for all sufficiently largek. Without loss of generality, consider such ak satisfyingt+−k=0 i α ≤k r (modulor). Fori=1, . . . , r−1, letR i = Pre (R)×{i}. Then trivially Pr (♦+ i=0 Ri)≥ r 1−ϵ and sinceβ agrees+ withα on allfinite paths that do not (yet) visit i=0 Ri, given r a pathρ that visits i=0 Ri (for thefirst time), onlyR-safe actions will be played byβ and thus all continuations ofρ in the outcome ofβ will visitR afterk steps (in total). It β =k β follows that Pr (♦ R0)≥1−ϵ, that isM k (R)≥1−ϵ. Note that we used the same strategyβ for allϵ>0 and thusβ is also almost-sure eventually synchronizing inR. !

76 The proof of Lemma 4.7 immediately gives the following corollary:

Corollary 4.2. LetM be an MDP andR be a set of states such that Pre r(R) =R for event sum event sum somer>0. Then, ⟨⟨1⟩⟩ almost ( R) = ⟨⟨1⟩⟩limit ( R).

Since deciding limit-sure reachability is PTIME-complete, it follows from Lemma 4.7 that limit-sure synchronization (with exact support) can be decided in EXPTIME. We show that the problem can be solved in PSPACE by exploiting the special structure of the exponential MDP in Lemma 4.7. We conclude this subsection by showing that limit-sure synchronization with exact support is PSPACE-complete (even in the special case of a trivial support).

Lemma 4.8. For MDPs, the membership problem for limit-sure eventually synchronization with exact support is in PSPACE.

Proof. We present a (non-deterministic) PSPACE algorithm to decide, given an MDPM= ⟨Q,A,δ⟩, an initial stateq init, and two setsT⊆U⊆Q, whetherq init is limit-sure eventually synchronizing inT with support inU. First, the algorithm computes numbersk≥0 andr>0 such that forR= Pre k(T) and Z= Pre k(U) we have Pre r(R)=R and Pre r(Z)=Z. As discussed before, this can be done |Q| event sum event sum by guessingk,r≤3 . By Lemma 4.6, we have ⟨⟨1⟩⟩limit ( T ,U)= ⟨⟨1⟩⟩ limit ( R,Z)∪ event sum ⟨⟨1⟩⟩sure ( T ), and since sure eventually synchronization inT can be decided in PSPACE (by Theorem 4.3), it suffices to decide limit-sure eventually synchronizing inR with support inZ in PSPACE. According to Lemma 4.7, it is therefore sufficient to show that deciding limit-sure winning for the (standard) reachability condition♦(R×{0}) in the MDPM Z ×[r] can be done in polynomial space. As we cannot afford to construct the exponential-size MDPM Z ×[r], the algorithm relies on the following characterization of the limit-sure winning set for reachability conditions in MDPs. It is known that the winning region for limit-sure and almost-sure reachability coincide [dAHK07], and pure memoryless strategies are sufficient. Therefore, we can see that the almost-sure winning setW for the reachability condition♦(R×{0}) satisfies the following property: there exists a memoryless strategy α:W→A such that (1)W is closed, that is post(q,α(q))⊆W for allq∈W , and (2) in the graph of the Markov chainM(α), for every stateq∈W , there is a path (of length at most|W|) fromq toR×{0}. This property ensures that from every state inW , the target setR×{0} is reached within|W| steps with positive (and bounded) probability, and sinceW is closed it ensures thatR×{0} is reached with probability1 in the long run. Thus any setW satisfying the above property is almost-sure winning. Our algorithm will guess and explore on thefly a setW to ensure that it satisfies this property, and contains the state⟨q init, t⟩ for somet

77 Z-safe actions, all successors are of the form(·, j− 1). It follows that the successors of the states inW i × {−i} should lie in the sliceW i+1 × {−i−1}, and we can guess on theﬂy the+ next sliceW i+1 ⊆Q by guessing for each stateq in a sliceW i an actiona q such that post q∈Wi (q,aq)⊆W i+1. Moreover, we need to check the existence of a path from every state inW toR×{0}. AsW is closed, it is suﬃcient to check that there is a path from every state inW 0 × {0} toR×{0}. To do this we guess along with the slicesW 0,W1,... a sequence of setsP 0,P1,... whereP i ⊆W i contains the states of sliceW i that belong to the guessed paths. Formally,P 0 =W 0, and for alli≥0, the setP i+1 is such that ′ ′ ′ post(q,aq)∩P i+1 ≠ ∅ for allq∈P i (whereP i =P i \R ifi is a multiple ofr, andP i =P i otherwise), that isP i+1 contains a successor of every state inP i that is not already in the targetR (at position0 modulor).

We need polynomial space to store theﬁrst sliceW 0, the current sliceW i and the set Pi, and the value ofi (in binary). AsM Z ×[r] has|Q| ·r states, the algorithm+ runs for |Q| ·r iterations and then checks that (1)W |Q|·r ⊆W 0 to ensure thatW= i≤|Q|·r Wi × {i modr} is closed, (2)P |Q|·r =∅ showing that from every state inW 0 × {0} there is a path toR×{0} (and thus also from all states inW ), and (3) the stateq init occurs in some slice Wi. The correctness of the algorithm follows from the characterization of the almost-sure winning set for reachability in MDPs: if some state⟨q init, t⟩ is limit-sure winning, then the algorithm accepts by guessing (slice by slice) the almost-sure winning setW and the paths fromW 0 × {0} toR×{0} (at position0 modulor), and otherwise any set (and paths) correctly guessed by the algorithm would not containq init in any slice. !

It follows from the proof of Lemma 4.7 that all winning modes for eventually synchronizing are independent of the numerical value of the positive transition probabilities.

Corollary 4.3. LetM be an MDP andT⊆U be two sets of states. For all winning modesµ∈{sure, almost, limit}, and for all two pairs of distributions d, d ′ with Supp(d)= ′ event sum ′ event sum Supp(d ), we haved ∈ ⟨⟨1⟩⟩ µ ( T ,U) if and only ifd ∈ ⟨⟨1⟩⟩µ ( T ,U).

To establish the PSPACE-hardness for limit-sure eventually synchronizing in MDPs, we use a reduction from the universalﬁniteness problem for 1L-AFAs. event sum Lemma 4.9. For MDPs, the membership problem for ⟨⟨1⟩⟩limit ( T ) is PSPACE-hard even ifT is a singleton.

Proof. The proof is by a reduction from the universalfiniteness problem for one-letter alternating automata (1L-AFA), which is PSPACE-complete (by Lemma 3.2). It is easy to see that this problem remains PSPACE-complete even if the setT of accepting states of the 1L-AFA is a singleton, and given the tight relation between 1L-AFA and MDP (see Subsection 3.4.2), it follows from the definition of the universalfiniteness problem that

78 MDPN A

qinit # ··· # # #

q1 ... q2 ⇒ q1 ... q2 MDPM T⊆Q MDPM T⊆Q

Figure 4.6: Sketch of reduction to show PSPACE-hardness of the membership problem for limit-sure eventually synchronizing.

n deciding, in an MDPM, whether the sequence Pre M(T)≠ ∅ for alln≥0 is PSPACE- complete.

The reduction is as follows (see also Figure 4.6). Given an MDPM=⟨Q,A,δ⟩ and a ′ ′ ′ ′ singletonT⊆Q, we construct an MDPN=⟨Q ,A ,δ ⟩ with state spaceQ =Q9{q init} n such that PreM(T)≠ ∅ for alln≥0 if and only ifq init is limit-sure eventually synchronizing inT . The MDPN is essentially a copy ofM with alphabetA9{#} and the transition function on action# is the uniform distribution onQ fromq init, and the Dirac distribution onq init from the other statesq∈Q. There are self-loops onq init for all other actionsa∈A. Formally, the transition functionδ ′ is defined as follows, for allq∈Q: ′ ′ –δ (q, a) =δ(q, a) for alla∈A (copy ofM), andδ (q, #)(qinit) = 1; ′ ′ 1 –δ (qinit, a)(qinit) = 1 for alla∈A, andδ (qinit, #)(q)= |Q| . We establish the correctness of the reduction as follows. For thefirst direction, as- n sume that PreM(T)≠ ∅ for alln≥0. Then sinceN embeds a copy ofM it fol- n |Q| lows that PreN (T)≠ ∅ for alln≥0 and there exist numbersk 0, r≤2 such that k0+r k0 k0 PreN (T) = Pre N (T)≠ ∅. Using Lemma 4.6 withk=k 0 andR= Pre N (T) (and ′ event U=Z=Q is the trivial support), it is sufficient to prove thatq init ∈ ⟨⟨1⟩⟩limit (R) to get event qinit ∈ ⟨⟨1⟩⟩limit (T) (inN ). We show the stronger statement thatq init is actually almost-sure eventually synchronizing inR with the pure strategyα defined as follows, for all play prefix ρ (letm=|ρ| modr): – if Last(ρ) =q init, thenα(ρ) = #; – if Last(ρ) =q∈Q, then r−m – ifq∈ Pre N (R), thenα(ρ) plays aR-safe action at positionr−m; – otherwise,α(ρ) = #. The strategyα ensures that the probability mass that is not (yet) in the sequence of n 1 predecessors PreN (R) goes toq init, where by playing# at least a fraction |Q| of it would reach the sequence of predecessors (at a synchronized position). It follows that after2i 1 i steps, the probability mass inq init is (1− |Q| ) and the probability mass in the sequence of

79 1 i predecessors is1− (1− |Q| ) . Fori→∞, the probability in the sequence of predecessors n tends to1 and since Pre N (R) =R for all positionsn that are a multiple ofr, we get α event supn Mn(R)=1 andq init ∈ ⟨⟨1⟩⟩almost (R).

event For the converse direction, assume thatq init ∈ ⟨⟨1⟩⟩limit (T) is limit-sure eventually synchronizing inT . By Lemma 4.6, either (1)q init is limit-sure eventually synchronizing in n n PreN (T) for alln≥0, and then it follows that Pre N (T)≠ ∅ for alln≥0, or (2)q init is sure eventually synchronizing inT , and then since only the action# leaves the stateq init k (and post(qinit, #) =Q), the characterization of Lemma 4.2 shows thatQ⊆ Pre N (T) for somek≥0, and sinceQ⊆ Pre N (Q) and PreN (·) is a monotone operator, it follows that n n Q⊆ Pre N (T) for alln≥k and thus Pre N (T)≠ ∅ for alln≥0. We conclude the proof by n n n noting that PreM(T)= Pre N (T)∩Q and therefore Pre M(T)≠ ∅ for alln≥0. !

The example in the proof of Lemma 4.3 can be used to show that the memory needed by a family of strategies to win limit-sure eventually synchronizing condition (in the target setT={q 2}) is unbounded.

The following theorem summarizes the results for limit-sure eventually synchronization in MDPs.

Theorem 4.5. For limit-sure eventually synchronizing condition (with or without exact support) in MDPs and for all functionsf∈{max T , sumT }: 1. (Complexity). The membership problem is PSPACE-complete. 2. (Memory). Unbounded memory is required for both pure and randomized strategies, and pure strategies are suﬃcient.

4.3 Eventually synchronization in PAs

As discussed in the beginning of Section 4.2, to show the complexity results of the membership problem for eventually synchronizing conditions in PAs, we only consider the emptiness problem of the eventually synchronizing languages according to function sum and from Dirac initial distributions too. Results presented in this section are listed in Table 4.2.

80 Always Eventually sum max Sure PSPACE-complete Almost-sure PSPACE-complete PTIME undecidable Limit-sure undecidable

Table 4.2: Computational complexity of the emptiness problem of always and eventually synchronizing languages in PAs.

4.3.1 Sure eventually synchronization Given a target setT in a PA, the emptiness problem for sure eventually synchronizing language inT can be solved by means of subset construction, as it is solved for NFAs. For ′ a PAP=⟨Q,A,δ⟩, the subset construction consists inN P =⟨Q ,A,∆⟩ where the state spaceQ ′ = 2Q \{∅} is the set of all cells, i.e. subsets of states inP, and the transition function∆:Q ′ ×A→Q ′ deﬁned as∆ ′(c, a) =c ′ where post(c, a) =c ′ anda∈A. We equip the subset construction ofP with the setF={S∈Q ′ |S⊆T} of accepting cells. A stateq init is sure-winning for eventually synchronizing inT if and only if the subset constructionN P , equipped with initial cell{q init} andF, accepts a word.

Lemma 4.10. LetP be a PA andT be a target set. From all statesq init, we have event sum Lsure ( T )≠ ∅ if and only if the language of the subset constructionN P , equipped ′ with the initial cell{q init} and setF={S∈Q |S⊆T} of accepting cells, is non-empty.

Proof. We prove the following equivalence by induction (on the lengthi of accepting words): w for all initial distributionsX 0, there exists a wordw such thatP i (T)=1 (we say that the wordw is sure-winning ini steps fromX 0) if and only if there exists an accepting wordv with length|v|=i for the subset constructionN P equipped with the initial cell Supp(X0) and setF={S∈Q ′ |S⊆T} of accepting cells. The casei=0 trivially holds since w for all wordsw, we haveP 0 (T)=1 if and only ifX 0(T)=1 implying that the initial cell Supp(X0) is an accepting cell in the subset construction. Assume that the equivalence holds for alli

81 PAP N1

′ 1 q1 qF1 DFAN 1 A : 2 # # q1 qF1 ⇒ qinit A′ sink qT A′ DFAN 2 # # ′ 1 N2 q2 qF2 A : 2 q2 qF2

Figure 4.7: Sketch of reduction to show PSPACE-hardness of the emptiness problem for sure eventually synchronizing language in PAs.

sure eventually synchronizing fromX 0 inn steps. It follows that the induction step holds, and thus the statement of lemma holds for all initial distributions. !

By Proof of Lemma 4.10, a PAP is sure eventually synchronizing in the target setT by a pure wordw if and only if there exists afinite wordv such thatw∈v·A ω where the subset ′ constructionN P equipped with the initial cell{q init} and setF={S∈Q |S⊆T} of accepting cells, accepts thefinite wordv. It implies that the set of all pure sure eventually synchronizing words in a PA isω-regular. Moreover, the PSPACE upper bounds follows for the emptiness problem of sure eventually synchronizing languages in PAs. The PSPACE- hardness is by a reduction fromfinite automata intersection based on an idea close to the reduction provided to show PSPACE-hardness forfinite synchronization from a subset in NFAs (in Subsection 3.1.1). event sum Lemma 4.11. For PAs, the emptiness problem forL sure ( T ) is PSPACE-hard even if T is a singleton.

Proof. The proof is by a reduction from thefinite automata intersection for DFAs, which is known to be PSPACE-complete [Koz77, GJ79, San04]. Thefinite automata intersection asks, givenn DFAs equipped with initial states and accepting sets, whether there exists an acceptingfinite word for all DFAs. The reduction is as follows: Givenn DFAsN 1,N 2,···,N n over the common alphabetA whereN i =⟨Q i,A,∆ i⟩ for all1≤i≤n, and where eachN i is equipped with the initial stateq i and the setF i of accepting states, we construct the PAP=⟨Q,A ′,δ⟩ as sketched in Figure 4.7 for the casen=2. We ′ augment the alphabet with a new letter:A =A∪{#}. We add a new initial stateq init, one copy of the state space of each DFAN i, and two new absorbing statesq T and sink: Q=Q 1 ∪···∪Q n ∪{q init, qT , sink}. For the new letter#, let#-transitions in all accepting statesq∈F 1 ∪···∪F n of all automata go to the stateq T whereas all#-transitions in all non-accepting statesq ̸∈F 1 ∪···∪F n go to the state sink. On all letters in initial stateq init, the next successor is randomly chosen from then initial stateq 1, q2,···,q n,

82 1 each with probability n . For all1≤i≤n and all statesq∈Q i and actionsa∈A, let δ(q, a) =∆ i(q, a). Letq T be the target state. We establish the correctness of the reduction as follows. First direction of proof is straightforward. By construction, for allfinite wordsw accepted by all DFAs, the PAP is ω sure eventually synchronizing in{q T } fromq init, by the infinite word#·w·# . event Second, assume thatL sure (qT )≠ ∅ fromq init, then there existsw that is a sure eventually synchronizing word fromq init in{q T }. After thefirst input, regardless the choice of 1 letter, we have probability n in each of the statesq 1, q2,···,q n. Sinceq T is only reachable via#-transitions, thenw has at least one#. Letw=a·v·#·v ′ wherea∈A∪{#}, v is the subword after thefirst letter and up to thefirst following occurrence of# and v′ ∈(A + #) ω. We show thatv is an accepting word for all DFAs. Towards contradiction, assume thatv is not accepted by at least one of the automaton, sayN 1. Thus, the run overv fromq 1 is not ending in an accepting state and the probability to be in sink would 1 be n by inputting the letter# right afterv. It means that the probability inq T will always 1 be bounded by1− n contradicting withw being a sure eventually synchronizing word fromq init in{q T }. !

The following theorem summarizes the results for sure eventually synchronizing in PAs.

Theorem 4.6. For sure eventually synchronizing languages in PAs:

1. (Complexity). For all functionsf∈{max T , sumT }, the emptiness problem is PSPACE-complete.

2. (Regularity). For the function sum T , the set of all pure sure eventually synchronizing words isω-regular.

4.3.2 Almost-sure eventually synchronization In this subsection, we provide undecidability result for the emptiness problem of almost- sure eventually synchronizing languages in PAs.

Theorem 4.7. For all functionsf∈{max T , sumT }, the emptiness problem of almost- sure eventually synchronizing language in PAs is undecidable.

Proof. The proof is by a reduction from the value1 problem for PAs, which is known to be undecidable (See Subsection 2.5.3). From a PAA=⟨Q,A,δ⟩ equipped with an initial ′ ′ ′ stateq init and a setF of accepting states, we construct another PAP=⟨Q ,A ,δ ⟩ and a

83 PAP A∪{#} # PAA A qT qF qF q init ⇒ qinit # # qn

A∪{#} Figure 4.8: Sketch of reduction to show undecidability of the emptiness problem for almost- sure eventually synchronizing languages in PAs.

event target stateq T such that the PAA has value1 if and only ifL almost (qT )≠ ∅ fromq init in the PAP. The PAP is a copy ofA with two new statesq n andq T that are reachable only by a ′ ′ new action#. Formally,Q =Q∪{q T , qn} andA =A∪{#}. The construction is depicted in Figure 4.8. The transition functionδ ′ is defined as follows:δ ′(q, a) =δ(q, a) for all statesq∈Q anda∈A. The#-transition in all accepting statesq∈F goes to the target ′ stateq T :δ (q, #)(qT ) = 1 whereq∈F. The#-transition in all not-accepting statesq ̸∈F ′ goes toq n:δ (q, #)(qn) = 1 whereq ̸∈F. All transitions, in the two new statesq T andq n, ′ ′ reset the PAP into the initial stateq init meaning thatδ (qn, a)(qinit) =δ (qT , a)(qinit) = 1 for all actionsa∈A∪{#}. We establish the correctness of the reduction as follows. First, assume that the PAA has value1. Since the definition of value in PAs is val(A) = sup w∈A∗ Pr(w), if val(A) = 1 then for allϵ>0 there exists afinite wordw ϵ such that Pr(wϵ)≥1−ϵ. Consider the 1 decreasing sequence{ }n≥2 and thefinite words{w 1 }n≥2; and construct the infinite word n n v=w 1 ·#·#·w 1 ·#·#···w 1 ·#·#···· represented asv={w 1 ·#·#} n≥2. We show thatv 2 3 n n is an almost-sure eventually synchronizing word in{q T } fromq init. We repeat the following argument for alln=2,3,··· : the initial (current) distribution is the Dirac distribution 1 1 onq init, next we input the wordw 1 where Pr(w 1 )≥1− . We thus have probability1− n n n n 1 in the setF, and inputting the following# leads the mass of probability1− n to the target set{q T } while the remaining mass of probability is in{q n}. Now, the second#-transitions reset the PAP, and the current distribution will be the initial Dirac distributionq init. This event argument proves thatL almost (qT )≠ ∅ fromq init. event For the reverse direction, assume thatL almost (qT )≠ ∅ fromq init in the PAP. Consider w w an infinite wordw such that sup n Pn (qT ) = 1. Letϵ>0 and letn be such thatP n (qT )≥ 1−ϵ. Sinceq T is only reachable via#-transitions then then-th letter ofw must be#. We pick the subwordv ofw as follows, – if there is no occurrence of# before the(n− 1)-th letter, letv be the subword from the beginning up to(n− 1)-th letter.

84 – otherwise, if the last occurrence of# before the(n−1)-th letter happens as them-th letter, check whether post(qinit, wm) ={q init}: 1. if yes, letv be the subword after them-th letter up to(n− 1)-th letter 2. otherwise, letv be the subword after the(m + 1)-th letter up to the(n− 1)-th letter

wherew m =a 0 ·a 1 ···a m is the prefix ofw consisting in thefirstm+1 letters. So, thefinite wordv does not contain# and it is thus a valid word forA. We show that Pr(v)≥1−ϵ in the PAA (equipped with the setF of accepting states). The prefix subword ofw beforev (thefirstm orm+1 letters, if there is any#) resets the automaton intoq init; note that if post(qinit, wm)≠ {q init}, then we surely have post(qinit, wm) ={q n, qT } and thus post(qinit, wm+1) ={q init}. Next, inputting the wordv followed by# results in having probability1−ϵ in the target stateq T . Sinceq T is only reachable from accepting statesq∈F, so the probability inF must have been1−ϵ after readingv and right before the last#. This argument proves that for allϵ≥0, we are able tofind a subwordv ofw that is valid forA and for which Pr(v)≥1−ϵ. This completes the proof. !

4.3.3 Limit-sure eventually synchronization The emptiness problem for limit-sure eventually synchronizing language in PAs is polynomial-time equivalent with value1 problem in PAs. If a PAP=⟨Q,A,δ⟩, equipped with the initial stateq init and the setF⊆Q of accepting states, has value1, then there exists a family offinite wordsw i for which the acceptance probability is arbitrarily close ω to1. By concatenating any infinite wordv∈A at the end of eachfinite wordw i in this family, a family of limit-sure eventually synchronizing words in the target setF fromq init is constructed. In the same way, from a family of limit-sure eventually synchronizing infinite words in the target setF from initial stateq init we can construct a family offinite words with acceptance probability arbitrarily close to1. It happens since the probability in the target setF will be1−ϵ afterfinitely many steps for the limit-sure eventually synchronizing condition. The following theorem follows from the above observation.

Theorem 4.8. For all functionsf∈{max T , sumT }, the emptiness problem of limit- sure eventually synchronizing language in PAs is undecidable.

4.4 Discussion

In this section, we shortly discuss the universality problems for sure and almost-sure {always, eventually} synchronizing languages in PA.

85 Universality problems of always synchronizing languages. For the function max T , by Proof of Lemma 4.1, we have established that the symbolic-run of a PA over a sure and almost-sure always synchronizing wordw is an always1-synchronizing sequence. Thus, the w states where the probability mass∥P i ∥ is accumulated form an infinite sequenceq 0q1q2 ··· where there is a deterministic transition fromq i toq i+1 for alli∈N. For a PAP with universal sure (and thus almost-sure) always synchronizing language, we claim that there is a unique sequenceq 0q1q2 ··· of states where the probability mass is accumulated along the symbolic runs ofP over all words. Toward contradiction, assume that for all deterministic w sequencesq 0q1q2 ··· inP, there exists a wordw and somen where∥P n ∥<1. The number of deterministic sequences inP arefinite, thus we can enumerate all pair of such words and steps with⟨w 1, n1⟩,···⟨w n, nk⟩ wherek is the number of deterministic sequences inP. 1 1 Thus, the symbolic run of the automaton over the randomized wordv= n w1 +···+ n wn cannot be1-synchronized at stepsn 1,···,n k implying thatv is not sure (and almost-sure) always synchronizing, a contradiction with the fact that the sure (and almost-sure) always synchronizing languages are universal. Thus, to decide the universality problems of always synchronizing languages according to max T , it is sufficient to check for all statesq reachable fromq init, whetherq∈T and|post(q,A)|=1. It implies that the graph underling the PA must be lasso-shaped, which can be checked in PTIME. A similar argument for the function sum T shows that to decide the universality problem of sure (and almost-sure) always synchronizing language, it is sufficient to check whether q∈T for all statesq reachable fromq init, which can be checked in PTIME.

Universality problem of sure eventually synchronizing language. Consider a PAP such that the sure eventually synchronizing language according to max T is universal. w w w By deﬁnition, for all wordsw the symbolic runP 0 P1 P2 ··· is eventually1-synchronized in w a singleton{q} whereq∈T , i.e. there existsn such thatP n (q)=1, and by Lemma 4.10, we know thatn≤2 |Q|. We claim that there exists some stepn≤2 |Q| such that for all w |Q| wordsw, we have∥P n ∥T = 1. Toward contradiction, assume that for all stepsn≤2 , wn there exists a wordw n such that∥P n ∥T ≠ 1. Then the symbolic run of the automaton 1 1 |Q| over the randomized wordv= k w1 +···+ k wk wherek=2 is never1-synchronized, implying thatv is not sure eventually synchronizing according to max T , a contradiction with the fact thatP has a universal sure eventually synchronizing language according to max T . A similar argument proves that the stateq where the probability mass is accumulated at stepn is unique for all words too. Thus, to decide the universality problem of sure eventually synchronizing language according to max T , one can compute the sequences 0s1s2 ··· wheres 0 ={q init} ands i = post(si−1,A) for alli∈N, and check whether there exists somen such thats n ={q} is a singleton andq∈T . The sequences 0s1s2 ··· is ultimately |Q| periodic meaning that there are k,n≤2 such thats n =s n+k; as a result universality problem of sure eventually synchronizing language according to max T can be decided in PSPACE. A similar argument for the function sum T shows that to decide the universality problem of sure eventually synchronizing language, the sequences 0s1s2 ··· wheres 0 ={q init} and

86 si = post(si−1,A) for alli∈N, can be computed and then one can check whether there exists somen such thats n ⊆T , which can be done in PSPACE. We do not provide a matching lower bound for the universality problem of sure eventually synchronizing languages. However, there is a reduction from tautology problem proving that deciding whether all pure words are sure eventually synchronizing according to the function sumT , is co-NP-hard.

Universality problem of almost-sure eventually synchronizing language. We

recall that ifx = sup n xn for an infinite sequencex 0x1x2 ··· of bounded values, thenx is equal or greater than allx i and either there exists a certainx n such thatx=x n or x = lim supn→∞ xn, For the function max T , consider a PAP that the almost-sure eventually synchronizing w language is universal. By definition, supn∈N∥Pn ∥T = 1 for all wordsw. There are three cases: w – either for all wordsw, there existsn such that∥P n ∥T = 1 meaning that the universality problem of almost-sure eventually synchronizing language reduces to the universality problem of sure eventually synchronizing language. w – or lim supn→∞∥Pn ∥T = 1 for all wordsw, meaning that the universality problem of almost-sure eventually synchronizing language reduces to the universality problem of almost-sure weakly synchronizing language, which is discussed in Section 5.3. ′ w – or there must be two wordsw andw such that there existsn such that∥P n ∥T = 1 but w w′ w′ lim supn→∞∥Pn ∥T <1, and∥P i ∥T ≠ 1 for alli∈N but lim sup n→∞∥Pn ∥T = 1. 1 1 ′ We see that for the randomized wordv= 2 w+ 2 w , there exists non such that v v ∥Pn∥T = 1 and also lim supn→∞∥Pn∥T <1, a contradiction with the fact thatP has a universal almost-sure eventually synchronizing language. It proves that the third case never happens. A similar argument for the function sum T shows that to decide the universality problem for almost-sure eventually synchronizing language, it is sufficient to check if the sure eventually synchronizing language of the PA, or if the almost-sure weakly synchronizing language is universal. Since for both function sumT and max T , the universality problem of almost-sure weakly synchronizing languages is in PSPACE, by Lemma 5.11 on page 111, the PSPACE membership follows for the universality problem of almost-sure eventually synchronizing languages in PAs. Lemma 4.12 provides a matching lower bound.

Lemma 4.12. The universality problem of almost-sure eventually synchronizing language according to sumT in PAs is PSPACE-hard, even ifT is a singleton.

Proof. We present a proof using a reduction from a PSPACE-complete problem so called initial state problem. Given an NFAN=⟨L,A,∆⟩ and a set of statesF, the initial state ω problem is to decide whether there exists an initial stateℓ 0 ∈L and an inﬁnite wordw∈A such that all runsρ=r 0r1r2 ··· ofN starting inℓ 0, overw avoidF, i.e.r i ̸∈F for all i∈N. From the results of [CSV08, TBG09], it follows that the initial state problem is

87 PSPACE-complete. We present a polynomial-time reduction from the initial state problem to the universality problem, establishing the PSPACE-hardness result. LetN=⟨L,A,∆⟩ be an NFA with the setF≠ ∅ of states. We assume, without loss of generality, thatN is complete and all states inN are all reachable (from some stateℓ 0). FromN , we construct a PAP=⟨Q,A,δ⟩ where the set of states is augmented with a new absorbing state synch, formallyQ=L∪{synch}. The transition functionδ is deﬁned as follows, for allq∈L anda∈A, – ifq ̸∈F, thenδ(q, a) is the uniform distribution over∆(q, a), ′ 1 ′ 1 – and ifq∈F,δ(q, a)(q ) = 2|∆(q,a)| for allq ∈∆(q, a) andδ(q, a)(synch) = 2 . Letq init =ℓ 0 be the state from which all states are reachable. We show that the answer to the initial state problem forN is Yes if and only if the almost-sure eventually synchronizing language ofP in{synch} is not universal. First, assume that the answer to the initial state problem forN is Yes. Thus, there exists some stateˆq and a wordw∈A ω satisfying the initial state problem. We construct a word that is not almost-sure eventually synchronizing forP. First, consider the|Q|-times repetition of the uniform distributiond u overA. Then, with positive probability the state synch is reached, and also with positive probability the stateˆq is reached, say afterk steps. Letw ′ ∈A ω be such thatw=v·w ′ and|v|=|Q|−k. Note that from stateˆq the ﬁnite wordv is played with positive probability by the repetition of uniform distribution |Q| ′ du. Therefore, on the word(d u) ·w , with some positive probability the state synch is never reached, and thusP is not almost-sure eventually synchronizing in{synch}, and the almost-sure eventually synchronizing language ofP is not universal. Second, assume that the almost-sure eventually synchronizing language ofP is not universal. Since synch is an absorbing state the eventually and weakly synchronization coincide, then by the construction in Lemma 5.10 on page 108, there exists a pure word w∈A ω such that from the initial state, all runs overw avoid the terminal end component{synch}, and therefore also avoidF. Hence, the answer to the initial state problem forN is Yes. !

From the above arguments and Lemma 4.12, it appears that for all functionsf∈ {max T , sumT }, the universality problem of almost-sure eventually synchronizing language in PAs is PSPACE-complete.

88 Weakly Synchronizing5 Condition

First sight. In this chapter, we establish the complexity and memory requirement for weakly synchronizing conditions. We show that the membership problem for MDPs is PSPACE-complete for sure and almost-sure winning, that exponential memory is necessary and suﬃcient for sure winning while inﬁnite memory is necessary for almost- sure winning, and we show that limit-sure and almost-sure winning coincide. Moreover, we show that while the emptiness problem for sure weakly synchronizing languages in PAs is PSPACE-complete, the emptiness decision problems for almost-sure and limit-sure languages are undecidable. The results presented in this chapter are summarized in Table 5.1 and Table 5.2 on pages 90 and 103, respectively.

Contents 5.1 Weak synchronization in MDPs ...... 90 5.1.1 Sure weak synchronization ...... 90 5.1.2 Almost-sure weak synchronization ...... 93 5.1.3 Limit-sure weak synchronization ...... 96 5.2 Weak synchronization in PAs ...... 103 5.2.1 Sure weak synchronization ...... 103 5.2.2 Almost-sure weak synchronization ...... 105 5.2.3 Limit-sure weak synchronization ...... 105 5.3 Discussion ...... 106

89 Weakly Complexity Memory requirement Sure PSPACE-complete exponential Almost-sure PSPACE-complete inﬁnite Limit-sure PSPACE-complete inﬁnite

Table 5.1: Computational complexity of the membership problem of weak synchronization in MDPs, and memory requirement for the winning strategies. 5.1 Weak synchronization in MDPs

By Remarks 5 and 6 (on pages 50 and 50), we consider the membership problem according to function sum and from Dirac initial distributions (i.e., single initial stateq init). The results presented in this section are summarized in Table 5.1.

5.1.1 Sure weak synchronization In this subsection, we study the membership problem for sure weakly synchronizing condition and show that it is PSPACE-complete . The PSPACE upper bound of the membership problem for sure weak synchronization is obtained by the following characterization.

Lemma 5.1. LetM be an MDP andT be a target set. For all statesq init, we have weakly sum m qinit ∈ ⟨⟨1⟩⟩sure ( T ) if and only if there exists a setS⊆T such thatq init ∈ Pre (S) for somem≥0 andS⊆ Pre n(S) for somen≥1.

Proof. We recall that for all initial distributionX 0 with supportS, we haveX 0 ∈ event sum k ⟨⟨1⟩⟩sure ( T ) if and only if there existsk≥0 such thatS⊆ Pre M(T) (See Lemma 4.2). weakly sum First, ifq init ∈ ⟨⟨1⟩⟩sure ( T ), then letα be a sure winning weakly synchronizing α strategy. Then there are infinitely many positionsn such thatM n(T)=1, and since the state space isfinite, there is a setS of states such that for infinitely many positionsn α α we have Supp(Mn) =S andM n(T)=1, and thusS⊆T . By Lemma 4.2, it follows m thatq init ∈ Pre (S) for somem≥0, and by considering two positionsn 1 < n2 where α α n Supp(Mn1 ) = Supp(Mn2 ) =S, it follows thatS⊆ Pre (S) forn=n 2 −n 1 ≥1. The reverse direction is straightforward by considering a strategyα ensuring that α Mm(S)=1 for somem≥0, and then ensuring that the probability mass from all states inS remains inS after every multiple ofn steps wheren>0 is such thatS⊆ Pre n(S), showing thatα is a sure winning weakly synchronizing strategy inS (and thus inT ) from weakly sum qinit, thusq init ∈ ⟨⟨1⟩⟩sure ( T ). !

The PSPACE upper bound follows from the characterization in Lemma 5.1. A (N)PSPACE algorithm is to guess the setS⊆T , and the numbers m, n (with m, n≤

90 MDPN MDPM MDPM A

qinit q ˆq ⇒ qinit q ˆq ˆp ♯ ♯ ♯

A′ sink ♯

Figure 5.1: The reduction sketch to show PSPACE-hardness of the membership problem for sure weak synchronization in MDPs.

2|Q| since the sequence Pren(S) of predecessors is ultimately periodic), and check that m n qinit ∈ Pre (S) andS⊆ Pre (S). The PSPACE lower bound follows from the PSPACE- completeness of the membership problem for sure eventually synchronization. Lemma 5.2 gives the PSPACE lower bound. weakly sum Lemma 5.2. The membership problem for ⟨⟨1⟩⟩sure ( T ) is PSPACE-hard even ifT is a singleton.

event sum Proof. The proof is by a reduction from the membership problem for ⟨⟨1⟩⟩sure ( T ) with a singletonT , which is PSPACE-complete (See Theorem 4.3). From an MDPM=⟨Q,A,δ⟩ ′ ′ ′ with initial stateq init and target stateˆq, we construct another MDPN=⟨Q ,A ,δ ⟩ and a event weakly target stateˆp such thatq init ∈ ⟨⟨1⟩⟩sure (ˆq) inM if and only ifq init ∈ ⟨⟨1⟩⟩sure (ˆp) inN. The MDPN is a copy ofM with two new statesˆp and sink that are reachable only by a new action♯ (see Figure 5.1). Formally,Q ′ =Q∪{ˆp, sink} andA ′ =A∪{♯}. The transition functionδ ′ is deﬁned as follows:δ ′(q, a) =δ(q, a) for all statesq∈Q anda∈A, δ(q,♯)(sink) = 1 for allq∈Q ′ \{ˆq} andδ(ˆq,♯)(ˆp) = 1. The state sink is absorbing and from stateˆp all other transitions lead to the initial state, i.e.,δ(sink, a)(sink) = 1 and δ(ˆp,a)(initq ) = 1 for alla∈A. event We establish the correctness of the reduction as follows. First, ifq init ∈ ⟨⟨1⟩⟩sure (ˆq) inM, then letα be a sure winning strategy inM for eventually synchronization in{ˆq}. A sure winning strategy inN for weak synchronization in{ˆp} is to play according toα until the whole probability mass is inˆq, then play♯ followed by somea∈A to visitˆp and get back to the initial stateq init, and then repeat the same strategy fromq init. Hence weakly qinit ∈ ⟨⟨1⟩⟩sure (ˆp) inN. weakly α Second, ifq init ∈ ⟨⟨1⟩⟩sure (ˆp) inN , then consider a strategyα such thatN n (ˆp) = 1 for α somen≥0. By construction ofN , it follows thatN n−1(ˆq)=1, that is all path-outcomes ofα of lengthn−1 reachˆq, andα plays♯ in the next step. Ifα never plays♯ before position event n−1, thenα is a valid strategy inM up to stepn−1 and it shows thatq init ∈ ⟨⟨1⟩⟩sure (ˆq) is sure winning inM for eventually synchronization in{ˆq}. Otherwise letm be the largest number such that there is aﬁnite path-outcomeρ ofα of lengthm

91 a, b

a H1 1 1 q1 q2 a, b: 1 a 2 b

qinit qT H2 a 2 q2 1 a, b: 2 2 q1 a b

2 a q3

Figure 5.2: The MDPM 2.

♯∈ Supp(α(ρ)). Note that the action♯ can be played byα only in the stateˆq, and thus the initial state is reached again after one more step. It follows that in some path-outcome ′ ′ ρ ofα of lengthm+2, we have Last(ρ ) =q init, and by the choice ofm, the action♯ is not played byα until positionn−1 where all the probability mass is inˆq. Hence the strategy ′ that plays likeα fromρ inN is a valid strategy fromq init inM, and is a witness that event qinit ∈ ⟨⟨1⟩⟩sure (ˆq). !

The proof of Lemma 5.1 suggests an exponential-memory strategy for sure weakly synchronization that inq∈ Pre n(S) plays an actiona such that post(q,a)⊆ Pre n−1(S), which can be realized with exponential memory sincen≤2 |Q|. It can be shown that exponential memory is necessary in general. The argument is very similar to the proof of exponential memory lower bound for sure eventually synchronization (See Section 4.2). For the sake of completeness, we present a family of MDPsM n (n∈N) over alphabet{a, b} that are sure winning for weak synchronization, and where the sure winning strategies require exponential memory. The MDPM 2 is shown in Figure 5.2. The structure of Mn is an initial uniform probabilistic transition ton componentsH 1,...,Hn whereH i is a cycle of lengthp i thei-th prime number. On actiona, the next state in the cycle is reached, and on actionb the target stateq T is reached, only from the last state in the cycles. From other states, the actionb leads to an absorbing sink state (transitions not depicted). A sure winning2 strategy fromq init for weak synchronization in{q T } is to play # n a in theﬁrstp n = i=1 pi steps, and then play bb to reachq init again, throughq T . This # n 2 requires memory of sizep n >2 while the size ofM n is inO(n logn) [BS96]. It can be proved that all winning strategies for weak synchronization need to be, fromq init, sure

92 eventually synchronizing in{q T } (consider the last occurrence ofq init along a play before # all the probability mass is inq T ) and this requires memory of size at leastp n by standard pumping arguments as in the sure eventually case.

Theorem 5.1. For sure weak synchronization in MDPs and for all functionsf∈ {sumT , max T }: 1. (Complexity). The membership problem is PSPACE-complete. 2. (Memory). Exponential memory is necessary and suﬃcient for both pure and randomized strategies, and pure strategies are suﬃcient.

5.1.2 Almost-sure weak synchronization

Letx = sup n∈N xn for a bounded sequence(x n)n∈N; thenx is equal or greater than allx i and either there exists a certainn such thatx n =x orx = lim sup n→∞ xn. Since all entries in a probability distribution are bounded by1, we can see that for an MDPM and a target setT, event event weak – ⟨⟨1⟩⟩almost (f)= ⟨⟨1⟩⟩ sure (f) ∪ ⟨⟨1⟩⟩ almost (f) wheref∈{max T , sumT }. Despite this mathematical connection, in the sequel, we give the intuitive characterization to decide the membership problem of almost-sure weak synchronization. We present a characterization of almost-sure weak synchronization that gives a PSPACE upper bound for the membership problem. Our characterization uses the limit-sure eventually synchronizing conditions with exact support. Recall that this condition requires that the probability mass tends to1 in a target setT , and moreover that after the same number of steps the support of the probability distribution is contained in a given setU. Formally, event sum given an MDPM, let ⟨⟨1⟩⟩ limit ( T ,U) forT⊆U be the set of all initial distributions α such that for allϵ>0 there exists a strategyα andn∈N such thatM n(T)≥1−ϵ and α Mn(U)=1. We show that an MDP is almost-sure weakly synchronizing in targetT if (and only if), for some setU, there is a sure eventually synchronizing strategy in targetU, and from the probability distributions with supportU there is a limit-sure winning strategy for eventually synchronizing in Pre(T) with support in Pre(U). This ensures that from the initial state we can have the whole probability mass inU, and fromU have probability 1−ϵ in Pre(T) (and inT in the next step), while the whole probability mass is back in Pre(U) (and inU in the next step), allowing to repeat the strategy forϵ→0, thus ensuring inﬁnitely often probability at least1−ϵ inT (for allϵ>0).

Lemma 5.3. LetM be an MDP andT be a target set. For all statesq init, we have weakly sum qinit ∈ ⟨⟨1⟩⟩almost ( T ) if and only if there exists a setU such that event sum –q init ∈ ⟨⟨1⟩⟩sure ( U ), and

93 event sum –X U ∈ ⟨⟨1⟩⟩limit ( Pre(T) , Pre(U)) whereX U is the uniform distribution overU. weakly sum Proof. First, ifq init ∈ ⟨⟨1⟩⟩almost ( T ), then there exists a strategyα such that for all α −i i≥0 there existsn i ∈N such thatM ni (T)≥1−2 , and moreovern i+1 > ni for all α α i≥0. Lets i = Supp(Mni ) be the support ofM ni . Since the state space isﬁnite, there is a setU that occurs inﬁnitely often in the sequences 0s1 ... , thus for allk>0 there exists α −k α mk ∈N such thatM mk (T)≥1−2 andM mk (U)=1. It follows thatα is sure eventually event sum synchronizing inU fromq init, i.e.,q init ∈ ⟨⟨1⟩⟩sure ( U ). Moreover, we can assume that mk+1 > mk for allk>0 and thusM is also limit-sure eventually synchronizing in Pre(T) α with exact support in Pre(U) from the initial distributionX 1 =M m1 . By Corollary 4.3, since Supp(X1) =U= Supp(X U ) and since only the support of the initial probability distributions is relevant for the limit-sure eventually synchronizing condition, it follows event sum thatX U ∈ ⟨⟨1⟩⟩limit ( Pre(T) , Pre(U)). event sum To establish the converse, note that sinceX U ∈ ⟨⟨1⟩⟩limit ( Pre(T) , Pre(U)), it follows from Corollary 4.3 that from all initial distributions with support inU, for allϵ>0 αϵ αϵ there exists a strategyα ϵ and a positionn ϵ such thatM nϵ (T)≥1−ϵ andM nϵ (U)=1. We construct an almost-sure weakly synchronizing strategyα as follows. Sinceq init ∈ event sum ⟨⟨1⟩⟩sure ( U ), play according to a sure eventually synchronizing strategy fromq init until −i all the probability mass is inU. Then fori=1,2,... andϵ i = 2 , repeat the following procedure: given the current probability distribution, select the corresponding strategy −i αϵi and play according toα ϵi forn ϵi steps, ensuring probability mass at least1−2 in Pre(T) and support of the probability mass in Pre(U). Then from states in Pre(T), play an action to ensure reachingT in the next step, and from states in Pre(U) ensure reaching

U. Continue playing according toα ϵi+1 forn ϵi+1 steps, etc. Sincen ϵi + 1>0 for alli≥0, α weak sum this strategy ensures that lim supn→∞ Mn(T)=1 fromq init, henceq init ∈ ⟨⟨1⟩⟩almost ( T ). !

Note that from Lemma 5.3, it follows that counting strategies are suﬃcient to win almost-sure weakly synchronizing condition (a strategy is counting ifα(ρ) =α(ρ ′) for all preﬁxesρ,ρ ′ with the same length and Last(ρ) = Last(ρ′)). Since the membership problems for sure eventually synchronizing and for limit-sure eventually synchronizing with exact support are PSPACE-complete (See Theorem 4.4 and Theorem 4.5), the membership problem for almost-sure weak synchronization is in PSPACE event sum by guessing the setU, and checking thatq init ∈ ⟨⟨1⟩⟩sure ( U ), and checking thatX U ∈ event sum ⟨⟨1⟩⟩limit ( Pre(T) , Pre(U)). We establish a matching PSPACE lower bound. weakly sum Lemma 5.4. The membership problem for ⟨⟨1⟩⟩almost ( T ) is PSPACE-hard even ifT is a singleton.

Proof. The proof is by the same reduction from the universalﬁniteness problem as presented in Lemma 4.9 where from an MDPM and a singletonT , we constructedN and n qinit, and showed that PreM(T)≠ ∅ for alln≥0 if and only ifq init is limit-sure eventually synchronizing inT (with support trivially in the state spaceQ ′ ofN ). We show that

94 n PreM(T)≠ ∅ for alln≥0 if and only ifq init is almost-sure weakly synchronizing inT, proving that the membership problem for almost-sure weakly synchronization is PSPACE- hard. First, ifq init is almost-sure weakly synchronizing inT , thenq init is also limit-sure even- n tually synchronizing inT , and therefore Pre M(T)≠ ∅ for alln≥0 by the arguments in the proof of Lemma 4.9. n Second, if PreM(T)≠ ∅ for alln≥0, thenq init is limit-sure eventually synchronizing ′ inT (with support in the state spaceQ =Q∪{q init} ofN ), and since the stateq init is reached from all states inQ by playing the action♯ and fromq init by playing any action inA, it follows that the uniform distribution overQ ′ is also limit-sure eventually synchronizing ′ inT (with support inQ ) byﬁrst going toq init and then playing a limit-sure eventually synchronizing strategy. By the characterization of Lemma 5.3 and sinceq init is trivially ′ weakly sum sure eventually synchronizing inQ , it follows thatq init ∈ ⟨⟨1⟩⟩almost ( T ). !

We provide an example of an MDP such that all strategies to win almost-sure weakly synchronizing conditions require infinite memory. The example and argument are analogous to the proof that infinite memory is necessary for almost-sure eventually synchronizing, see Lemma 4.3. Lemma 5.5. There exists an almost-sure weakly synchronizing MDP for which all almost- sure weakly synchronizing strategies require infinite memory.

Proof. Consider the MDPM shown in Figure 4.4 with three statesq init, q1, q2 and two actions a, b. The only probabilistic transition is inq init on actiona that has successors 1 qinit andq 1 with probability 2 . The other transitions are deterministic. Letq init be the initial state. We construct a strategy that is almost-sure weakly synchronizing in{q 2}, weakly showing thatq init ∈ ⟨⟨1⟩⟩almost (q2). First, observe that for allϵ>0 we can have probability at least1−ϵ inq 2 afterfinitely many steps fromq init: playingn timesa and thenb leads 1 to probability1− 2n inq 2. Note that after that, the current probability distribution has support{q init, q2} and that from such a distribution, we can as well ensure probability at least1−ϵ inq 2. Thus for afixedϵ, the MDP is (1−ϵ)-synchronizing in{q 2} (afterfinitely many steps), and by taking a smaller value ofϵ, we can continue to play a strategy to have probability at least1−ϵ inq 2, and repeat this forϵ→0. This strategy ensures almost- sure weak synchronization in{q 2}. Below, we show that infinite memory is necessary for almost-sure winning in this MDP. Assume towards contradiction that there exists afinite-memory strategyα that is almost-sure weakly synchronizing in{q 2}. Consider the Markov chainM(α) (the product of the MDPM with thefinite-state transducer definingα). A state(q, m) inM(α) is called aq-state. Sinceα is almost-sure weakly synchronizing in{q 2}, there is aq 2-state in the recurrent states ofM(α). Since on all actionsq init is a successor ofq 2, andq init is a successor of itself, it follows that there is a recurrentq init-state inM(α), and that all periodic supports of recurrent states inM(α) contain aq init-state. Hence, in each

95 stationary distribution there is aq init-state with a positive probability, and therefore the probability mass inq init is bounded away from zero. It follows that the probability mass inq 2 is bounded away from1 thusα is not almost-sure weakly synchronizing in{q 2}, a contradiction. !

From previous lemmas, we obtain Theorem 5.2 stating that the membership problem for almost-sure weakly synchronizing, is PSPACE-complete.

Theorem 5.2. For almost-sure weak synchronization in MDPs and all functionsf∈ {max T , sumT }: 1. (Complexity). The membership problem is PSPACE-complete. 2. (Memory). Inﬁnite memory is necessary in general for both pure and randomized strategies, and pure strategies are suﬃcient.

5.1.3 Limit-sure weak synchronization We show that the winning regions for almost-sure and limit-sure weak synchronization coincide. The result is not intuitively obvious (recall that it does not hold for eventually synchronizing) and requires a careful analysis of the structure of limit-sure winning strategies to show that they always induce the existence of an almost-sure winning strategy. The construction of an almost-sure winning strategy from a family of limit-sure winning strategies is illustrated in the following example.

Example 5.1. Consider the MDPM in Figure 5.3 with initial stateq init and target set T={q 4}. Note that there is a relevant strategic choice only inq 3, and thatq init is limit- sure winning for eventually synchronization in{q 4} since we can inject a probability mass arbitrarily close to1 inq 3 (by always playinga inq 3), and then switching to playingb inq 3 gets probability1−ϵ inT (for arbitrarily smallϵ). Moreover, the same holds from stateq 4. These two facts are sufficient to show thatq init is limit-sure winning for weak synchronization in{q 4}: givenϵ>0, play fromq init a strategy to ensure probability at ϵ leastp 1 = 1− 2 inq 4 (infinitely many steps), and then play according to a strategy that ϵ ensures fromq 4 probabilityp 2 =p 1 − 4 inq 4 (infinitely many, and at least one step), and repeat this process using strategies that ensure, if the probability mass inq 4 is at leastp i, ϵ that the probability inq 4 is at leastp i+1 =p i − 2i+1 (in at least one step). It follows that ϵ ϵ ϵ pi = 1− 2 − 4 −···− 2i >1−ϵ for alli≥1, and thus lim sup i→∞ pi ≥1−ϵ showing that qinit is limit-sure weakly synchronizing in target{q 4}. It follows from the result that we establish in this section (Theorem 5.3) thatq init is actually almost-sure weakly synchronizing in target{q 4}. To see this, consider the sequence i Pre (T) fori≥0:{q 4},{q 3},{q 2},{q 3},... is ultimately periodic with periodr=2 and

96 1 a, b a, b: 2 b a, b qinit q2 q3 q4 q5

a 1 1 a, b: a, b: 2 2 a, b 1 a, b a, b: 2

q1 q6

weakly weakly Figure 5.3: An example to showq init ∈ ⟨⟨1⟩⟩limit (q4) impliesq init ∈ ⟨⟨1⟩⟩almost (q4).

2 R={q 3}= Pre(T) is such thatR= Pre (R). The period corresponds to the loopq 2q3 in the MDP. It turns out that limit-sure eventually synchronizing inT implies almost-sure eventually synchronizing inR, see Corollary 4.2, thus fromq init a single strategy ensures that the probability mass inR is1, either in the limit or afterfinitely many steps. Note that in both cases sinceR= Pre r(R) this even implies almost-sure weakly synchronizing in R. The same holds from stateq 4. Moreover, note that all distributions produced by an almost-sure weakly synchronizing strategy are themselves almost-sure weakly synchronizing. An almost-sure winning strategy for weak synchronization in{q 4} consists in playing fromq init an almost-sure eventually synchronizing strategy in targetR={q 3}, and considering a decreasing sequenceϵ i such that limi→∞ ϵi = 0, when the probability mass inR is at least1−ϵ i, inject it inT={q 4}. Then the remaining probability mass defines a distribution (with support{q 1, q2} in the example) that is still almost-sure eventually synchronizing inR, as well as the states in T . Note that in the example, (almost all) the probability mass inT={q 4} can move toq 3 in an even number of steps, while from{q 1, q2} an odd number of steps is required, resulting in a shift of the probability mass. However, by repeating the strategy two times fromq 4 (injecting large probability mass inq 3, moving toq 4, and injecting inq 3 again), we can make up for the shift and reachq 3 fromq 4 in an even number of steps, thus in synchronization with the probability mass from{q 1, q2}. ▹ This idea behind Example 5.1, which is constructing an almost-sure winning strategy from a family of limit-sure winning strategies by annihilating the shifts at proper times, is formalized in the rest of this subsection, and we prove that we can always make up for the shifts, which requires a carefully analysis of the allowed amounts of shifting. The result is easier to prove when the targetT is a singleton, as in Example 5.1. For an arbitrary target setT , we need to get rid of the states inT that do not contribute a significant (i.e., bounded away from0) probability mass in the limit, that we call the vanishing states. We show that they can be removed fromT without changing the winning region for limit-sure winning. When the target set has no vanishing state, we can construct an almost-sure winning strategy as in the case of a singleton target set.

97 1 a, b: 2 a 1 a: 2 qinit q1

1 a, b b: 2 b a, b

q2 q3

Figure 5.4: The stateq 2 is a vanishing state for the target setT={q 2, q3} and the family of stagiest(α) i∈N described in Example 5.2.

weakly sum Deﬁnition 5.1. Given an MDPM with initial stateq init ∈ ⟨⟨1⟩⟩limit ( T ) that is limit-sure winning for the weakly synchronizing condition in target setT , let(α i)i∈N be αi a family of limit-sure winning strategies such that lim supn→∞ Mn (T)≥1−ϵ i where limi→∞ ϵi = 0. Hence by deﬁnition of lim sup, for alli≥0 there exists a strictly αi increasing sequencek i,0 < ki,1 <··· of positions such thatM ki,j (T)≥1−2ϵ i for all αi j≥0. A stateq∈T is vanishing if lim inf i→∞ lim infj→∞ Mki,j (q)=0 for some family of limit-sure weakly synchronizing strategies(α i)i∈N.

Intuitively, the contribution of a vanishing stateq to the probability inT tends to0 and thereforeM is also limit-sure winning for the weakly synchronizing condition in target setT\{q}. Example 5.2. Consider the MDP in Figure 5.4 with four states and two actions a, b. All transitions are deterministic except inq init: thea-transitions have two successorsq init 1 andq 1 each with probability 2 , as well as theb-transitions with two successorsq init andq 2. Thea-transition inq 1 is a self-loop whereas theb-transition deterministically goes toq 3. All transitions inq 2 are directed toq init, and all inq 3 are directed toq 1. LetT={q 2, q3} be the target set and for alli∈N, letα i be the strategy that repeats the following template forever: playingi timesa and then playing two timesb. The family of strategies(α i)i∈N is weakly sum a witness to show thatq init ∈ ⟨⟨1⟩⟩limit ( T ) where the stateq 2 is a vanishing state. We see that the contribution ofq 2 in accumulating the probability mass in{q 2, q3} tends to0 weakly wheni→∞. Thus,q init ∈ ⟨⟨1⟩⟩limit (q3) too. ▹

Lemma 5.6. If an MDPM is limit-sure weakly synchronizing in target setT , then there exists a setT ′ ⊆T such thatM is limit-sure weakly synchronizing inT ′ without vanishing states.

′ Proof. If there is no vanishing state for(α i)i∈N, then takeT =T and the proof is complete. Otherwise, let(α i)i∈N be a family of limit-sure winning strategies such that

98 αi lim supn→∞ Mn (T)≥1−ϵ i where limi→∞ ϵi = 0 and letq be a vanishing state for(α i)i∈N. We show that(α i)i∈N is limit-sure weakly synchronizing inT\{q}. For everyi≥0 let αi ki,0 < ki,1 <··· be a strictly increasing sequence such that(a)M ki,j (T)≥1−2ϵ i for all αi i, j≥0, and(b) lim inf i→∞ lim infj→∞ Mki,j (q)=0. It follows from(b) that for allϵ>0 and allx>0 there existsi>x such that for αi ally>0 there existsj>y such thatM ki,j (q)<ϵ, and thus

αi Mki,j (T\{q})≥1−2ϵ i −ϵ by(a). Since this holds for inﬁnitely manyi’s, we can choosei such thatϵ i <ϵ and we have αi lim sup Mki,j (T\{q})≥1−3ϵ j→∞ and thus αi lim sup Mn (T\{q})≥1−3ϵ n→∞

since the sequence(k i,j)j∈N is strictly increasing. This shows that(α i)i∈N is limit-sure weakly synchronizing inT\{q}. By repeating this argument as long as there is a vanishing state (thus at most|T|−1 times), we can construct the desired setT ′ ⊆T without vanishing state. !

For a limit-sure weakly synchronizing MDP in target setT (without vanishing states), we show that from a probability distribution with supportT , a probability mass arbitrarily close to1 can be injected synchronously back inT (in at least one step), that isX T ∈ event sum ⟨⟨1⟩⟩limit ( Pre(T) ). The same holds from the initial stateq init of the MDP. This property is the key to construct an almost-sure weakly synchronizing strategy.

Lemma 5.7. If an MDPM with initial stateq init is limit-sure weakly synchronizing event sum in a target setT without vanishing states, thenq init ∈ ⟨⟨1⟩⟩limit ( Pre(T) ) andX T ∈ event sum ⟨⟨1⟩⟩limit ( Pre(T) ) whereX T is the uniform distribution overT.

weakly sum weakly sum event sum Proof. Sinceq init ∈ ⟨⟨1⟩⟩limit ( T ) and ⟨⟨1⟩⟩limit ( T ) ⊆ ⟨⟨1⟩⟩limit ( T ), we have event sum event sum qinit ∈ ⟨⟨1⟩⟩limit ( T ) and thus it suﬃces to prove thatX T ∈ ⟨⟨1⟩⟩limit ( Pre(T) ). This is because then fromq init, probability arbitrarily close to1 can be injected in Pre(T) through a distribution with support inT (since by Corollary 4.3 only the support of the initial probability distribution is important for limit-sure eventually synchronizing). αi Let(α i)i∈N be a family of limit-sure winning strategies such that lim supn→∞ Mn (T)≥ 1−ϵ i where limi→∞ ϵi = 0, and such that there is no vanishing state. For everyi≥0 αi letk i,0 < ki,1 <··· be a strictly increasing sequence such thatM ki,j (T)≥1−2ϵ i for αi all i, j≥0, and letB = min q∈T lim infi→∞ lim infj→∞ Mki,j (q). Note thatB>0 since there is no vanishing state. It follows that there existsx>0 such that for alli>x there αi B existsy i >0 such that for allj>y i and allq∈T we haveM ki,j (q)≥ 2 .

99 νB Givenν>0, leti>x such thatϵ i < 4 , and forj>y i, consider the positionsn 1 =k i,j αi αi andn 2 =k i,j+1. We haven 1 < n2 andM n1 (T)≥1−2ϵ i andM n2 (T)≥1−2ϵ i, and αi B Mn1 (q)≥ 2 for allq∈T . Consider the strategyβ that plays likeα i plays from position αi αi n1 and thus transforms the distributionM n1 intoM n2 . For all statesq∈T , from the Dirac distribution onq under strategyβ, the probability to reachQ\T inn 2 −n 1 steps is thus at most αi Mn2 (Q\T) 2ϵi αi ≤ <ν. Mn1 (q) B/2 Therefore, from an arbitrary probability distribution with supportT (over setT ) we β haveM n2−n1 (T)>1−ν, showing thatX T is limit-sure eventually synchronizing inT and thus in Pre(T) sincen 2 −n 1 >0 (it is easy to show that if the mass of probability inT ν is at least1−ν, then the mass of probability in Pre(T) one step before is at least1− η whereη is the smallest positive probability inM). !

To show that limit-sure and almost-sure winning coincide for weakly synchronizing conditions, from a family of limit-sure winning strategies we construct an almost-sure winning strategy that uses the eventually synchronizing strategies of Lemma 5.7. The construction consists in using successively strategies that ensure probability mass1−ϵ i in the targetT , for a decreasing sequenceϵ i →0. Such strategies exist by Lemma 5.7, both from the initial state and from the setT . However, the mass of probability that can be guaranteed to be synchronized inT by the successive strategies is always smaller than1, and therefore we need to argue that the remaining masses of probability (of sizeϵ i) can also get synchronized inT , and despite their possible shift with the main mass of probability. Two main key arguments are needed to establish the correctness of the construction: (1) eventually synchronizing implies that afinite number of steps is sufficient to obtain a probability mass of1−ϵ i inT , and thus the construction of the strategy is well defined, and (2) by thefiniteness of the periodr (such thatR= Pre r(R) whereR= Pre k(T) for some k) we can ensure to eventually make up for the shifts, and every piece of the probability mass can contribute (synchronously) to the target infinitely often.

weakly sum weakly sum Theorem 5.3. ⟨⟨1⟩⟩limit ( T ) = ⟨⟨1⟩⟩almost ( T ) for all MDPs and target setsT.

weakly sum weakly sum Proof. Since ⟨⟨1⟩⟩almost ( T ) ⊆ ⟨⟨1⟩⟩limit ( T ) holds by the definition, it is sufficient to weakly sum weakly sum prove that ⟨⟨1⟩⟩limit ( T ) ⊆ ⟨⟨1⟩⟩almost ( T ) and by Lemma 5.6 it is sufficient to prove weakly sum that ifq init ∈ ⟨⟨1⟩⟩limit ( T ) is limit-sure weakly synchronizing inT without vanishing state, thenq init is almost-sure weakly synchronizing inT . IfT has vanishing states, then ′ considerT ⊆T as in Lemma 5.6 and it will follows thatq init is almost-sure weakly syn- ′ weakly sum chronizing inT and thus also inT . We proceed with the proof thatq init ∈ ⟨⟨1⟩⟩limit ( T ) weakly sum impliesq init ∈ ⟨⟨1⟩⟩almost ( T ).

100 Fori=1,2,... consider the sequence of predecessors Pre i(T), which is ultimately periodic: let1≤ k,r≤2 |Q| such that Prek(T)= Pre k+r(T), and letR= Pre k(T). Thus R= Pre k+r(T)= Pre r(R).

event sum event sum Claim 1. We haveq init ∈ ⟨⟨1⟩⟩almost ( R) andX T ∈ ⟨⟨1⟩⟩almost ( R).

Proof of Claim 1. By Lemma 5.7, since there is no vanishing state inT we haveq init ∈ event sum event sum ⟨⟨1⟩⟩limit ( Pre(T) ) andX T ∈ ⟨⟨1⟩⟩limit ( Pre(T) ), and it follows from the characterization of Lemma 4.6 and Corollary 4.2 that: event sum event sum either (1)q init ∈ ⟨⟨1⟩⟩sure ( Pre(T) ) or (2)q init ∈ ⟨⟨1⟩⟩almost ( R), and event sum event sum either(a)X T ∈ ⟨⟨1⟩⟩sure ( Pre(T) ) or(b)X T ∈ ⟨⟨1⟩⟩almost ( R). Note that(a) implies(b) (and thus(b) holds) since(a) impliesT⊆ Pre i(T) for some i≥1 (by Lemma 4.2) and thusT⊆ Pre n·i(T) for alln≥0 by monotonicity of Pre i(·), m which entails forn·i≥k thatT⊆ Pre (R) wherem=(n·i−k) modr and thusX T is sure (and almost-sure) winning for the eventually synchronizing condition in targetR. Note also that (1) implies (2) since by (1) we can play a sure-winning strategy fromq init to ensure inﬁnitely many steps probability1 in Pre(T) and in the next step probability1 inT , and by(b) play an almost-sure winning strategy for eventually synchronizing inR. event sum Henceq init ∈ ⟨⟨1⟩⟩almost ( R) and thus (2b) holds, which concludes the proof of Claim 1.

We now show that there exists an almost-sure winning strategy for the weakly synchronizing condition in targetT. Recall that Prer(R)=R and thus once some probability massp is inR, it is possible to ensure that the probability mass inR afterr steps is at leastp, and thus that (with periodr) the probability inR does not decrease. By the result of Lemma 4.7, almost-sure winning for eventually synchronizing inR implies that there exists a strategyα such that the probability inR tends to1 at periodic positions: for some0≤h0·∃N·∀n≥N:n≡h α α modr=⇒M n(R)≥1−ϵ. We also say that the initial distributionX 0 =M 0 is almost-sure eventually synchronizing inR with shifth.

Claim 2. α α (⋆) IfM 0 is almost-sure eventually synchronizing inR with some shifth, thenM i is almost-sure eventually synchronizing inR with shifth−i modr. (⋆⋆) Lett such thatX T is almost-sure eventually synchronizing inR with shiftt. If a distribution is almost-sure eventually synchronizing inR with some shifth, then it is also almost-sure eventually synchronizing inR with shifth+k+t modr (where we chosek such thatR= Pre k(T)).

101 Proof of Claim 2. The result (⋆) immediately follows from the definition of shift, and we prove (⋆⋆) as follows. We show that almost-sure eventually synchronizing inR with shifth implies almost-sure eventually synchronizing inR with shifth+k+t modr. Intuitively, the probability mass that is inR with shifth can be injected inT ink steps, and then fromT we can play an almost-sure eventually synchronizing strategy in target R with shiftt, thus a total shift ofh+k+t modr. Precisely, an almost-sure winning strategyα is constructed as follows: given afinite prefix of playρ, if there is no stateq∈R that occurs inρ at a positionn≡h modr, then play inρ according to the almost-sure winning strategyα h for eventually synchronizing inR with shifth. Otherwise, if there is noq∈T that occurs inρ at a positionn≡h+k modr, then we play according to a sure winning strategyα sure for eventually synchronizing inT , and otherwise we play according to an almost-sure winning strategyα t fromT for eventually synchronizing inR with shift t. To show thatα is almost-sure eventually synchronizing inR with shifth+k+t, note thatα h ensures with probability1 thatR is reached at positionsn≡h modr, and thus T is reached at positionsh+k modr byα sure, and from the states inT the strategyα t ensures with probability1 thatR is reached at positionsh+k+t modr. This concludes the proof of Claim 2.

Construction of an almost-sure winning strategy. We construct strategiesα ϵ for ϵ>0 that ensure, from a distribution that is almost-sure eventually synchronizing inR (with some shifth), that afterfinitely many steps, a distributiond ′ is reached such that d′(T)≥1−ϵ andd ′ is almost-sure eventually synchronizing inR (with some shifth ′). Sinceq init is almost-sure eventually synchronizing inR (with some shifth), it follows that the strategyα as that plays successively the strategies (each forfinitely many steps)α 1 ,α 1 , 2 4 α 1 ,... is almost-sure winning fromq init for the weakly synchronizing condition in targetT. 8 We define the strategiesα ϵ as follows. Given an initial distribution that is almost-sure eventually synchronizing inR with a shifth and givenϵ>0, letα ϵ be the strategy that plays according to the almost-sure winning strategyα h for eventually synchronizing inR with shifth for a number of stepsn≡h modr until a distributiond is reached such thatd(R)≥1−ϵ, and then fromd it plays according to a sure winning strategyα sure for eventually synchronizing inT from the states inR (fork steps), and keeps playing ′ according toα h from the states inQ\R (fork steps). The distributiond reached from d afterk steps is such thatd ′(T)≥1−ϵ and we claim that it is almost-sure eventually synchronizing inR with shiftt. This holds by definition from the states in Supp(d ′)∩T, and by (⋆) the states in Supp(d′)\T are almost-sure eventually synchronizing inR with shifth−(h+k) modr, and by (⋆⋆) with shifth−(h+k)+k+t=t. It follows that the strategyα as is well-defined and ensures, for allϵ>0, that the probability mass inT is infinitely often at least1−ϵ, thus is almost-sure weakly synchronizing inT . This concludes the proof of Theorem 5.3. !

From previous lemmas and Theorem 5.3, we obtain the following result.

102 Weakly Sure PSPACE-complete Almost-sure undecidable Limit-sure undecidable

Table 5.2: Computational complexity of the emptiness problem of weakly synchronizing languages in PAs.

Theorem 5.4. For limit-sure weak synchronization in MDPs and all functionsf∈ {max T , sumT }: 1. (Complexity). The membership problem is PSPACE-complete. 2. (Memory). Inﬁnite memory is necessary in general for both pure and randomized strategies, and pure strategies are suﬃcient.

5.2 Weak synchronization in PAs

Again by Remarks 5 and 6 (on pages 50 and 50), we consider the emptiness problem for weakly synchronizing languages in PAs only according to function sum and initialized from Dirac distributions. The results presented in this section are summarized in Table 5.2.

5.2.1 Sure weak synchronization We show that the emptiness problem for sure weakly synchronizing languages in PAs, is PSPACE-complete. The PSPACE upper bound is an immediate result from the following intuitive Lemma.

Lemma 5.8. LetP be a PA andT be a target set. For all initial stateq init, we have weakly sum ω Lsure ( T )≠ ∅ if and only if there are two words w,v∈A and a setS⊆T such that –w is sure eventually synchronizing inS fromq 0, –v is non-trivially sure eventually synchronizing inS from an arbitrary distribution with supportS.

weakly sum Proof. First, ifL sure ( T )≠ ∅ fromq init, then letw be a sure weakly synchronizing w word. Then, there are infinitely many positionsn such thatP n (T)=1, and since the state space isfinite, there is a setS of states such that for infinitely many positionsn w w we have Supp(Pn ) =S andP n (T)=1, and thusS⊆T . It thus follows thatw is sure eventually synchronizing inS fromq 0 too. Moreover by considering two positionsn 1 < n2 w w ′ where Supp(Pn1 ) = Supp(Pn2 ) =S, it follows that there is a non-empty wordv such

103 that all wordsv∈v ′ ·A ω are non-trivially sure eventually synchronizing inS from the w distributionP n1 with supportS. The reverse direction is straightforward. Letw n =a 0 ···a n−1 be the prefix ofw, which w consists of thefirstn-th letters, wheren is such thatP n (S)=1 fromq 0. Letv m = b0 ···b m−1 be the non-empty prefix ofv, which consists of thefirstm-th letters, wherem is v such thatP m(S)=1 from an arbitrary distribution with supportS. By characterization presented in Lemma 4.10, we know that only the support of an initial distribution matters ω weak sum for sure eventually synchronizing languages in PAs, thenw n ·(v m) ∈L sure ( T ) fromq 0. !

The PSPACE upper bound follows from the characterization in Lemma 5.8 and the fact that emptiness problem for sure eventually synchronizing languages in PAs, is PSPACE- complete (See Theorem 4.6). The PSPACE-hardness is by a reduction fromﬁnite automata intersection that is the same reduction to obtain hardness result for the emptiness problem of sure eventually synchronizing languages in PAs (See Lemma 4.11). weakly sum Lemma 5.9. The emptiness problem forL sure ( T ) in PAs, is PSPACE-hard even ifT is a singleton.

Proof. The proof is by the same reduction fromfinite automata intersection problem that is presented in Lemma 4.11 where fromn DFAsN 1,N 2,···,N n we constructed a PAP, an initial stateq init and a target stateq T and showed that alln DFAs accept a common event word if, and only if,L sure (qT )≠ ∅ fromq init. event weakly Now we show that, in the constructed PAP, we haveL sure (qT ) =L sure (qT ) fromq init, proving that the emptiness problem for sure weakly synchronizing languages is PSPACE- hard. weakly event First direction is straightforward since by definition,L sure (qT )⊆L sure (qT ) from all initial states. event w Second, ifw∈L sure (qT ), then there isi∈N such thatP i (qT ) = 1. Letw i = a0a1 ···a i−1 be the prefix ofw that consists of thefirsti-th letter. Since all transitions ′ ω inq T are self-loops, then allω-wordsw i ·(A ) are sure weakly synchronizing fromq init (See weakly Lemma 5.8) As a result,w∈L sure (qT ) fromq init that completes the proof. !

From the previous lemmas, we obtain the following Theorem.

Theorem 5.5. For all functionsf∈{sum T , max T } the emptiness problem for sure weakly synchronizing languages in PAs is PSPACE-complete.

104 5.2.2 Almost-sure weak synchronization In this subsection, we provide undecidability result for the emptiness problem of almost- sure weakly synchronizing languages in PAs.

Theorem 5.6. For all functionsf∈{max T , sumT }, the emptiness problem of almost- sure weakly synchronizing language in PAs is undecidable.

Proof. The proof is by the same reduction from value1 problem in PAs that is presented in Theorem 4.7 where from a PAA equipped with an initial stateq init and a setF of accepting states, we constructed another PAP and a target stateq T ; and we showed that event the PAA has value1 if and only ifL almost (qT )≠ ∅ fromq init in the PAP. weak Now we prove that the PAA has value1 if and only ifL almost (qT )≠ ∅ fromq init in the PAP too. First, assume thatA has value1. In the proof of Theorem 4.7, we have argued that the inﬁnite wordv=w 1 ·#·#·w 1 ·#·#···w 1 ·#·#···· is almost-sure 2 3 n eventually synchronizing in{q T } fromq init. The same argument showing that for alln≥2, 1 the mass of probability1− n is accumulated inq T after reading the preﬁx ofv (from beginning) up to the subwordw 1 ·♯, is valid here and implies thatv is also almost-sure n weakly synchronizing inq T . weak event The reverse direction is straightforward since we knowL almost (qT )⊆L almost (qT ) from event all initial states, and since we have proven that ifL almost (qT )≠ ∅ fromq init, thenA has value1. !

5.2.3 Limit-sure weak synchronization In this subsection, we provide undecidability result for the emptiness problem of limit- sure weakly synchronizing languages in PAs.

Theorem 5.7. For all functionsf∈{max T , sumT }, the emptiness problem of limit- sure weakly synchronizing language in PAs is undecidable.

Proof. The proof is by the same reduction from value1 problem in PAs that is presented in Theorem 4.7 and Theorem 5.6 where from a PAA equipped with an initial stateq init and a setF of accepting states, we constructed another PAP and a target stateq T ; and event we showed that the PAA has value1 if and only ifL almost (qT )≠ ∅ fromq init if and only if weak Lalmost (qT )≠ ∅ fromq init in the PAP. weak Now we prove that the PAA has value1 if and only ifL limit (qT )≠ ∅ fromq init in the PAP too. First, assume thatA has value1. In the proof of Theorem 5.6, we weak have shown that there exists some almost-sure weakly synchronizing wordv∈L almost (qT ) fromq init. By deﬁnition, sincev is almost-sure weakly synchronizing, so is limit-sure weakly weak synchronizing. Thus,L limit (qT )≠ ∅ fromq init.

105 weak Second, assume thatL limit (qT )≠ ∅ fromq init. Therefore, for allϵ>0 there exists an w inﬁnite wordw such that lim sup n→∞ Pn (qT )≥1−ϵ. Forϵ>0, let the wordw and the w numbern be such thatP n (qT )≥1−2ϵ. Sinceq T is only reachable via#-transitions then then-th letter ofw must be#. We pick the subwordv ofw as we have done in the proof of Theorem 4.7: – if there is no occurrence of# before the(n− 1)-th letter, letv be the subword from the beginning up to(n− 1)-th letter. – otherwise, if the last occurrence of# before the(n−1)-th letter happens as them-th letter, check whether post(qinit, wm) ={q init}: 1. if yes, letv be the subword afterm-th letter up to(n− 1)-th letter 2. otherwise, letv be the subword after(m + 1)-th letter up to(n− 1)-th letter

wherew m =a 0 ·a 1 ···a m is the preﬁx ofw consisting in theﬁrstm+1 letters. By a similar argument (as one in the proof of Theorem 4.7), we have Pr(v)≥1−2ϵ implying that there is a family of words whose accepting probability inA is arbitrarily close to1. This meansA has value1 and the proof is complete. !

5.3 Discussion

In this section, we brieﬂy discuss the universality problems for sure and almost-sure weakly synchronizing languages in PAs.

Universality problems of sure weakly synchronizing language. Consider a PAP with the universal sure weakly synchronizing language according to max T . By definition, w w w for all wordsw the symbolic runP 0 P1 P2 ··· is infinitely often1-synchronized in a singleton{q} whereq∈T . We claim that there exist n, m≤2 |Q| such that for all wordsw, w and alli∈N we have∥P k ∥=1 wherek=n+i·m. Toward contradiction, assume |Q| that for all n, m≤2 there exists some wordw n,m such that for somei∈N, we have w ∥Pk ∥<1 wherek=n+i·m. Then, the symbolic run of the automaton over the ran- 1 1 |Q| domized wordv= k2 w1,1 +···+ k2 wk,k wherek=2 is never weakly1-synchronized implying thatv is not sure weakly synchronizing according to max T , a contradiction with the fact thatP has a universal sure weakly synchronizing language according to max T . A similar argument proves that the stateq where the probability mass is accumulated at steps of the sequence n, n+ m, n+2m,··· is unique for all words too. In order to decide the universality problem of sure weakly synchronizing language according to max T , one can compute the sequences 0s1s2 ··· wheres 0 ={q init} ands i = post(si−1,A) for alli∈N. |Q| The sequences 0s1s2 ··· is ultimately periodic meaning that there are n, m≤2 such thats n =s n+m; thus it is sufficient to check if there is somes i in the periodic part of the

106 sequences 0s1s2 ··· such thats i ={q} is a singleton withq∈T , which can be decided in PSPACE. A similar argument for the function sum T shows that to decide the universality problem of sure weakly synchronizing language, the sequences 0s1s2 ··· wheres 0 ={q init} and si = post(si−1,A) for alli∈N, can be computed and then one can check whether there exists somes i in the periodic part of the sequences 0s1s2 ··· such thats i ⊆T , which can be done in PSPACE. We do not provide a matching lower bound for the universality problem of sure weakly synchronizing languages. However, there is a reduction from tautology problem proving that deciding whether all pure words are sure weakly synchronizing according to the function sumT , is co-NP-hard.

Universality problems of almost-sure weakly synchronizing language. Recall that for a PAP, an end componentU⊆Q is terminal if post(U,A)⊆U; and an inﬁnite randomized word is the uniformly randomized word over the alphabetA denoted byw u = d0d1d2 ··· ifd i is the uniform distribution overA for alli∈N. For the function max T , a PAP with a universal almost-sure weakly synchronizing language, must only have a unique terminal end component. Otherwise, the uniformly randomized wordw u would reach all terminal end components with some positive probability, and thusP would not be almost-sure weakly synchronizing byw u. Example 5.3 provides an instance of a PA to show that even though having only one terminal end component is necessary for universality of almost-sure weakly synchronizing language, it is not suﬃcient.

Example 5.3. The PAP depicted in Figure 5.5 has four statesq init, q1, q2 andq 3 and two 1 actions a, b. All transitions inq init have two successorsq 1 andq 2, each with probability 2 . Two statesq 1 andq 2 are somewhat symmetric: both have one transition directed to another, and one transition directed to the absorbing stateq 3; however, thea-transition of one acts like theb-transition of the other. The PAP has only one terminal end component{q 3}, but the almost-sure weakly synchronizing language is not universal. As a counterexample word, considerw=a·(a·b) ω 1 max whereP is weakly 2 -synchronizing in two sets{q 1, q2} and{q 3} (according to ). It implies that having one terminal end component is not suﬃcient to have a universal almost- sure weakly synchronizing language. ▹ In order to have a universal almost-sure weakly synchronizing language, a PA must ensure that for all randomized words, the probability mass tends to accumulate in a unique terminal end component; we express this property for a terminal end component as being w absorbing. A terminal end componentU is absorbing if lim n→∞ Pn (U)=1 for all inﬁnite randomized wordsw∈D(A) ω. Example 5.4 shows an instance of a PA where the unique terminal end component is absorbing and the almost-sure weakly synchronizing language is universal.

107 qinit

qinit 1 1 1 1 a, b: 2 a, b: 2 a, b: 2 a, b: 2 a q q 1 1 1 2 a, b: 2 a, b: 2

b q1 q3 q2 a, b a b b a a, b b a q3 q4

Figure 5.5: Non-absorbing end compo- Figure 5.6: Absorbing end component. nent.

Example 5.4. The PAP shown in Figure 5.6 hasfive statesq init, q1, q2, q3 andq 4 and two actions a, b. All transitions inq init andq 3 have two successorsq 1 andq 2, each with 1 probability 2 . Two statesq 1 andq 2 are somewhat symmetric: both have one transition directed toq 3, and one transition directed to the absorbing stateq 4; however, thea-transition of one acts like theb-transition of the other. The PAP has only one terminal end component{q 4}, which is absorbing: w limn→∞ Pn (q4) = 1 for all infinite randomized wordsw. The end component{q 4} is absorbing because for the other end components{q 1, q2, q3}, even though there are strategies to ensure that once the PA enters in the end component, it stays there (the requirement to be an end component), there is no blind strategy (word) to ensure this property with non-zero probability. We see that after thefirst input,P arrives either inq 1 orq 2 each with 1 probability 2 . Next, after every two consecutive inputs some positive probability leaks down to the absorbing end component{q 4} while the remaining probability is in the end components{q 1, q2, q3}. Repeating this forever, the probability mass is gradually accumulated in{q 4} implying that the almost-sure weakly synchronizing language ofP is universal. ▹

Decision problem(s). The absorbing decision problem asks, given a PA and an end component, whether the end component is absorbing for the PA.

Lemma 5.10 provides the PSPACE membership of the absorbing problem. Lemma 5.10. The absorbing problem for PAs is decidable in PSPACE.

Proof. Given a terminal end componentU of a PAP, we construct an NFAN equipped with an initial state and a coBüchi acceptance condition such thatU is absorbing if and

108 only if the language ofN is empty. The automatonN is exponential in the size ofP, and since the emptiness problem of NFAs with coBüchi conditions is NLOGSPACE-complete, the PSPACE membership of absorbing problem follows. Intuitively, the NFAN guesses a word such that the symbolic-outcome ofP over that word, from some point on, keeps assigning a (strictly) positive probabilityp>0 to sets of states which are disjoint withU. The alphabet ofN is2 A whereA is the alphabet ofP. An infinite word over2 A is a sequence of sets of letters which can be viewed as the sequence of supports of a randomized word. The states ofN are pairs(s, b) consisting of a sets⊆Q of states ofP and a Booleanflagb∈{0,1}. A pair(s, b) is0-flagged ifb=0; and it is 1-flagged otherwise. Theflagb is set to1 whenN guesses that from now on,P is always outside ofU with some positive probabilityp. The NFAN begins in the pair({q init}, 0) ′ whereq init is the initial state ofP. On an inputA ⊆A,N not-deterministically moves from a0-flagged state(s, 0) into – either another0-flagged state(s ′, 0) wheres ′ = post(s,A ′) is the set of successors of s and the input, – or a1-flagged state(c, 1) wherec is a non-empty subset of post(s,A ′)\U; note that c is disjoint withU, i.e.,c∩U=∅. After reaching thefirst1-flagged pair(c, 1),N only checks whether the successors ofc (where the probabilityp was assigned) always has an empty intersection withU. The coBüchi acceptance condition requires thatN visits only1-flagged pairs, infinitely often. From the PAP=⟨Q,A,δ⟩ with the initial stateq init and the end componentU, we constructN=⟨L,2 A \{∅},∆⟩ whereL=2 Q × {0,1} is the set of pairs and2 A \{∅} is the alphabet. The transition function∆:L×2 Σ →2 L is defined as follows. For all subsetss⊆Q andA ′ ⊆A, lets ′ = post(s,A ′), and – define∆((s, 0),A ′) ={(s ′, 0)}∪{(c, 1)|c≠ ∅ andc⊆s ′ \U}, – define∆((s, 1),A ′) ={(s ′, 1)} ifs ′ ∩U=∅, and∆((s, 1)) =∅ otherwise. We equipN with the initial pair({q init}, 0) and the coBüchi condition♦!F whereF= 2Q × {1}.

To establish the correctness of the reduction, we show that the coBüchi language ofN is empty if and only if the terminal end componentU is absorbing. First, assume that the terminal end componentU is absorbing for the PAP. Toward contradiction, assume that the coBüchi language ofN is not empty. Therefore, there exists some infinite wordv and n∈N such that for the runρ=r 0r1 ... ofN overv and for alli>n, we haver i ∈F. From the accepting wordv=A 0A1 ... for coBüchi condition♦!F, we construct a randomized wordw and claim that the symbolic-run ofP overw tends to accumulate some positive probability outsideU. Definew=d 0d1 ... such thatd i is the uniform distribution overA i, w w Q thus Supp(di) =A i. LetP 0 P1 ··· be the symbolic-run ofP overw. SinceF=2 × {1} and by the construction ofN , we know that for alli>n, the pairr n = (si, 1) is1-flagged w ands i+1 = post(si,A i). By construction ofw, we see thats n ⊆ Supp(Pn ); therefore, w eachs i is a subset of the support Supp(Pi ) of the symbolic-outcome at stepi. Since w w ∅= ̸ s n ⊆ Supp(Pn ), we haveP n (sn) =p>0. Sinces i+1 = post(si,A i) for alli>n, the w probabilityP i (si) is always at leastp. On the other hand, we haves i ⊆Q\U which gives

109 w w Pi (Q\U)≥p>0. Therefore lim i→∞ Pi (U)<1−p<1, a contradiction with the fact thatU is an absorbing end component. Second, assume that the coBüchi language ofN is empty. We prove thatU is absorbing w by showing that limn→∞ Pn (U)=1 for all randomized wordsw. We claim that for all ran- w w w domized wordsw=d 0d1 ··· , for alln∈N and for all statesq∈ Supp(P n ) whereP 0 P1 ··· is the symbolic-run ofP overw, there exists aw-path fromq toU with lengthk≤2 |Q|; w aw-path fromq∈ Supp(P n ) toU is a pathℓ 0ℓ1 ···ℓ k such thatℓ 0 =q,ℓ k ∈U and for all0≤i

w k k P2·k(Q\U)≤ (1−η )· (1−η ) k w k where (1−η ) is an upper bound ofP 2·k(Q\U) and (1−η ) is an upper bound for the probability mass which does not move toU withink steps. Afterj repetitions, we have

w k j Pj·k (Q\U)≤ (1−η ) . Since0≤1−η k <1, we have

w lim Pj·k (Q\U)=0. j→∞

110 w The above arguments immediately prove that limn→∞ Pn (U)=1 for all wordsw, meaning thatU is an absorbing end component. The proof is complete. !

Another necessary condition for a PAP to have a universal almost-sure weakly synchronizing language is thatP is almost-sure weakly synchronizing by the uniformly randomized word. For instance, the automaton presented in Lemma 3.4 has the absorbing end component{q 2, q3}, but since theP is not almost-sure weakly synchronizing in{q 3} by the uniformly randomized word, the almost-sure weakly synchronizing language is not universal.

Lemma 5.11. The universality problem of almost-sure weakly synchronizing languages according to max T in PAs is in PSPACE.

Proof. Given a PAP=⟨Q,A,δ⟩ with the initial stateq init and the function max T , the almost-sure weakly synchronizing language is universal if and only if (i) there is a (then necessarily unique) absorbing end componentU, and

(ii)P is almost-sure weakly synchronizing by the uniformly randomized wordw u. One direction is straightforward. For the reverse direction, assume that both of the conditions (i) and (ii) are fulﬁlled in the PAP. We show that the almost-sure weakly synchronizing language (according to max T ) ofP is universal. SinceP is almost-sure weakly synchronizing according to max T by the uniformly randomized wordw u, there exists wu ˆq∈T∩U such that for allϵ>0 and allm∈N, there existsn>m whereP n (ˆq)>1−ϵ. Letc 0 ={ˆq} and deﬁne the sequence of setsc 1c2 ··· as follows;c i+1 = post(ci,A) for all i∈N. Letn be the smallest number such that the absorbing end componentU=∪ 0≤i0 be such thatP n0 (ˆq)>1−ϵ and wu wu n wu n ck ⊆ Supp(Pn0 ). Thus,P n0+n(q)≥η (1−ϵ) andP n0+n(ˆq)<1−η (1−ϵ). We can repeat wu n the arguments for allj∈N and show thatP n0+jn(ˆq)<1−η (1−ϵ)<1−ϵ which is a contradiction with the fact thatw u is almost-sure weakly synchronizing according to max T .

111 We have shown that the setsc 0, c1,···,c n−1 are disjoints andc n =c 0 ={ˆq} where ˆq∈T . Towards contradiction with the universality of the almost-sure weakly synchronizing language ofP, assume that there exists an inﬁnite wordw that is not almost-sure weakly max w w synchronizing according to T . LetP 0 P1 ··· be the symbolic-run ofP overw; and wu wu P0 P1 ··· be the symbolic-run over the uniformly randomized wordw u. SinceU is absorbing and it consists ofn disjoint setsc 0, c1,···,c n−1 wherec 0 ⊆T is a singleton, there existsm∈N such that for alln>m there are two subsetsc andc ′ (from the sets ′ w c0, c1,···,c n−1 coveringU) wherec∪c ⊆ Supp(Pn ); otherwisew would be almost-sure weakly synchronizing. By assumption, the uniformly randomized wordw u is almost-sure w weakly synchronizing. All states reachable byw are also reachable byw u: Supp(Pn )⊆ wu ′ Supp(Pn ). Hence, for alln>m there are two subsetsc andc (from the setsc 0, c1,···,c n−1 ′ wu coveringU) such thatc∪c ⊆ Supp(Pn ) too. Since the probabilities assigned to these ′ disjoint setsc andc always loop through the sequencec 0c1 ···c n, the probability mass could never accumulate in the singletonc 0 (or any of singletonsc⊆T in the sequencec 0c1 ···c n), a contradiction with condition (ii) stating thatw u is almost-sure weakly synchronizing.

Condition (i) can be checked in PSPACE by Lemma 5.10, and condition (ii) can be checked in PTIME by steady state analysis of the Markov chain induced by the PA under the uniformly randomized word. The PSPACE bound follows. !

We have proved that, given a PAP with the initial stateq init and the function max T , the almost-sure weakly synchronizing language (according to max T ) ofP is universal if and only if (i) there is an absorbing end componentU, and (ii)P is almost-sure weakly synchronizing by the uniformly randomized wordw u. As we have shown in the proof of Lemma 5.11, these two conditions enforce a special shape to the absorbing end componentU; the end componentU is partitioned inton disjoints setsc 0, c1 ···,c n where one of them is singleton, and for the symbolic-run of the PA over the uniformly randomized word, all supports can only contain one of these disjoint setsc i (0≤i

112 The above arguments provided the PSPACE membership of the the universality problem of almost-sure weakly synchronizing languages for both function sum T and max T , a matching lower bound can be proved by the same reduction presented in Lemma 4.12 for eventually synchronizing languages. In the presented reduction, the constructed PA is almost-sure eventually synchronizing in the singleton{synch} if and only if it is almost-sure weakly synchronizing. From the above arguments and Lemma 5.11, it turns out that for all functionsf∈ {max T , sumT }, the universality problem of almost-sure weakly synchronizing language in PAs is PSPACE-complete.

113 114 Strongly Synchronizing Condition6

First sight. In this chapter, we show that the membership problem for strongly synchronizing conditions in MDPs is PTIME-complete, for all winning modes, and both with function max T and function sumT . We show that linear-size memory is necessary in general for max T , and memoryless strategies are suﬃcient for sum T . It follows from our results that the limit-sure and almost-sure winning modes coincide for strong synchronization. Moreover, we study the emptiness problem for strongly synchronizing languages in PAs. We prove that for sure and almost-sure languages according to both functions max T and sumT , the emptiness problem is PSPACE-complete whereas the problems for limit-sure languages are undecidable. The results presented in this chapter are summarized in Table 6.1 and Table 6.2 on pages 116 and 124, respectively.

Contents 6.1 Strong synchronization in MDPs ...... 116 6.1.1 Strong synchronization with function max ...... 116 6.1.2 Strong synchronization with function sum ...... 122 6.2 Strong synchronization in PAs ...... 124 6.2.1 Sure strong synchronization with function max ...... 124 6.2.2 Almost-sure strong synchronization with function max . . . . 126 6.2.3 Sure strong synchronization with function sum ...... 130 6.2.4 Almost-sure strong synchronization with function sum . . . . 131 6.2.5 Limit-sure strong synchronization ...... 134 6.3 Discussion ...... 138

115 Strongly max sum Complexity Required memory Complexity Required memory Sure PTIME-complete linear PTIME-complete memoryless Almost-sure PTIME-complete linear PTIME-complete memoryless Limit-sure

Table 6.1: Computational complexity of the membership problem of strongly synchronization in MDPs, and memory requirement for the winning strategies (Two winning modes almost-sure and limit-sure coincide).

6.1 Strong synchronization in MDPs

For strong synchronization the membership problem with function max T reduces to the membership problem with function max Q whereQ is the entire state space, by a construction similar to the proof of Remark 5: states inQ\T are duplicated, in such a way that a stateq∈Q\T contains some probabilityp if and only if the duplication of that stateq contain the exact probabilityp too. This construction ensures that only states inT are used to accumulate the probability mass tending to1. The results presented in this section are summarized in Table 6.1.

6.1.1 Strong synchronization with function max In this subsection, to prove the PTIME-completeness for the membership problem of strong synchronization with function max T , we solve the problem with function max Q that establishes the upper bound. We provide the matching lower bound by proving PTIME- hardness for the membership problem of strong synchronization with function max T where T is a singleton. The strongly synchronizing condition with function max requires that from some point on, almost all the probability mass is at every step in a single state. The sequence of states that contain almost all the probability corresponds to a sequence of deterministic transitions in the MDP, and thus eventually to a cycle of deterministic transitions. The graph of deterministic transitions of an MDPM=⟨Q,A,δ⟩ is the directed graph G=⟨Q, E⟩ whereE={⟨q 1, q2⟩|∃a∈A:δ(q 1, a)(q2) = 1}. Forℓ≥1, a deterministic cycle inM of lengthℓ is aﬁnite pathˆq ℓ ˆℓq−1 ···ˆq0 inG (that is,⟨ˆq i,ˆqi−1⟩ ∈E for all 1≤i≤ℓ) such thatˆq 0 = ˆℓq. The cycle is simple ifˆqi ≠ ˆjq for all1≤i

116 Deﬁne the MDPM×[ℓ] =⟨Q ′,A,δ ′⟩ whereQ ′ =Q×{0,1,···,ℓ−1} and δ′(⟨q, i⟩, a)(⟨q′, i−1⟩) =δ(q, a)(q ′) (wherei−1 isℓ−1 fori=0) for all states q,q ′ ∈Q, actionsa∈A, and0≤i≤ℓ−1. Lemma 6.1. Given an MDPM, letη be the smallest positive probability in the transitions 1 α ofM and let 1+η < p≤1. There exists a strategyα such that lim inf n→∞∥Mn∥ ≥p from an initial stateq init if and only if there exists a simple deterministic cycleˆqℓ ˆℓq−1 ···ˆq0 in β M and a strategyβ inM×[ℓ] such that Pr (♦{⟨ˆ0q,0⟩})≥p from⟨q init,0⟩.

Proof. First, assume that there exists a simple deterministic cycleˆqℓ ˆℓq−1 ···ˆq0 with lengthℓ and a strategyβ inM×[ℓ] that ensures the target set♦{⟨ˆq 0,0⟩} is reached with probability at leastp from the state⟨q init,0⟩. Since randomization is not necessary for reachability conditions, we can assume thatβ is a pure strategy. We show that there exists a strategyα α such that lim infn→∞∥Mn∥ ≥p fromq init. Fromβ, we construct a pure strategyα inM. Given aﬁnite pathρ=q 0a0q1a1 . . . qn inM (withq 0 =q init), there is a corresponding path ′ ρ =⟨q 0, k0⟩a0⟨q1, k1⟩a1 ...⟨q n, kn⟩ inM×[ℓ] wherek i =−i modℓ. Since the sequence k0k1 ... is uniquely determined fromρ, there is a clear bijection between the paths inM and the paths inM×[ℓ] that we often omit to apply and mention. Forρ, we deﬁneα as

follows: ifq n = ˆkqn , then there exists an actiona such that post(ˆqkn , a) ={ˆqkn+1 }={ˆqn+1} and we defineα(ρ) =a, otherwise letα(ρ) =β(ρ ′). Thusα mimicsβ unless a stateq is reached at stepn such thatq=ˆq k wherek=−n modℓ, and thenα switches to always playing actions that keepsM in the simple deterministic cycleˆq ℓ ˆℓq−1 ···ˆq0. Below, we α prove that givenϵ>0 there existsk such that for alln≥k, we have∥M n∥ ≥p−ϵ. It α β follows that lim infn→∞∥Mn∥ ≥p fromq init. Since Pr (♦{⟨ˆ0q,0⟩})≥p, there existsk such β ≤k that Pr (♦ {⟨ˆ0q,0⟩})≥p−ϵ. We assume w.l.o.g.+ that k modℓ=0. Fori=0,1,...,ℓ−1, β ≤k ℓ−1 letR i ={⟨ˆqi, i⟩}. Then trivially Pr+(♦ i=0 Ri)≥p−ϵ and sinceα agrees+ withβ on all ℓ−1 ℓ−1 finite paths that do not (yet) visit i=0 Ri, given a pathρ that visits i=0 Ri (for thefirst time), only actions that keepM in the simple cycleˆq ℓ ˆℓq−1 ···ˆq0 are played byα and thus all continuations ofρ in the outcome ofα will visitˆq 0 afterk steps (in total). It follows β k α α that Pr (♦ {⟨ˆ0q,0⟩})≥p−ϵ, that isM k (ˆ0q)≥p−ϵ. Thus,∥M k ∥ ≥p−ϵ. Since next, α will always play actions that keepsM looping through the cycleˆq ℓ ˆℓq−1 ···ˆq0, we have α ∥Mn∥ ≥p−ϵ for alln≥k. α Second, assume that there exists a strategyα such that lim inf n→∞∥Mn∥ ≥p fromq init. α Thus, for allϵ>0 there existsk∈N such that for alln≥k we have∥M n∥ ≥p−ϵ. 1 Fixϵk such that for alla there existsq≠ ˆp j+1 such that{q,ˆp j+1}⊆ post(ˆpj, a). Therefore, α α Mj+1(q)≥M j (ˆjp)·η≥(p−ϵ)·η. Hence,

α α Mj+1(ˆjp+1)≤1−M j+1(q)≤1−(p−ϵ)·η.

α 1 Thus,p−ϵ≤∥M j+1∥ ≤1−(p−ϵ)·η that givesϵ≥p− 1+η , a contradiction. This argument proves that for alln≥k, there exists an actiona∈A such that post(ˆp n, a) ={ˆpn+1}. The

117 finiteness of the state spaceQ entails that in the sequenceˆp k ˆkp+1 ··· , some state and thus some simple deterministic cycle occur infinitely often. Letˆqℓ ˆℓq−1 ···ˆq0 be a cycle that occurs infinitely often in the sequenceˆpk ˆkp+1 ··· (in the right order). For allj, leti j be the position ofˆq0 in all occurrences of the cycleˆqℓ ˆℓq−1 ···ˆq0 in the sequenceˆpk ˆkp+1 ··· ; and let tj =i j modℓ. In the sequencet 0t1 ··· , there exists0≤t<ℓ that appears infinitely often. Let the cycler ℓrℓ−1 ···r 0 be such thatr (i+t) modℓ = îq. Then, the cycler ℓrℓ−1 ···r 0 happens infinitely often in the sequenceˆpk ˆkp+1 ··· such that the positions ofr 0 are infinitely often 0 (moduloℓ). Therefore, the probability ofM to be inr 0 in positions (moduloℓ) equals to0, is infinitely often equal or greater thanp. Hence, for a strategyβ inM×[ℓ] that β copies all the plays of the strategyα, we have Pr (♦{⟨r0,0⟩})≥p from⟨q init,0⟩. !

It follows directly from Lemma 6.1 withp=1 that almost-sure strong synchronization is equivalent to almost-sure reachability to a deterministic cycle inM×[ℓ]. The same equivalence holds for the sure and limit-sure winning modes.

Lemma 6.2. Given an MDPM, a stateq init is sure (resp., almost-sure or limit-sure) winning for the strongly synchronizing condition (according to max Q) if and only if there exists a simple deterministic cycleˆqℓ ˆℓq−1 ···ˆq0 such that⟨q init,0⟩ is sure (resp., almost-sure or limit-sure) winning for the reachability condition♦{⟨ˆq0,0⟩} inM×[ℓ].

Proof. The proof is organized in three sections: (1) sure winning mode: First, assume that there exists a simple deterministic cycleˆqℓ ˆℓq−1 ···ˆq0 with lengthℓ such that⟨q init,0⟩ is sure winning for the reachability condition♦{⟨ˆq0,0⟩}. Thus, there exists a pure memoryless strategyβ such that Outcomes(⟨qinit,0⟩,β)⊆♦{⟨ˆq 0,0⟩}. Sinceβ is memoryless, there must bek≤|Q| ×ℓ ≤k such that Outcomes(⟨q init,0⟩,β)⊆♦ {⟨ˆ0q,0⟩} meaning that all infinite paths starting in⟨q init,0⟩ and followingβ reach⟨ˆq 0,0⟩ withink steps. Fromβ, we construct a pure finite-memory strategyα inM that is represented byT=⟨Memory, i,α u,α n⟩ where Memory={0,···,ℓ−1} is the set of modes. The idea is thatα simulates whatβ plays in the state⟨q, i⟩, in the stateq ofM and modei ofT (there is only one exception). Thus, the initial mode is0. The update function only decreases modes by1 (αu(i, a, q)=(i− 1) modℓ for all statesq and actionsa) since by taking any transition the mode is decreased by1. The next-move functionα n(i, q) is defined as follows: αn(i, q)=β(⟨q, i⟩) for all statesq and modes0≤i<ℓ, except whenq=ˆq i, in this case let αn(i, q)=a where post(ˆqi, a) ={q i−1}. Thusβ mimicsα unless a stateq is reached at step n such thatq=ˆq−n modℓ , and thenα switches to always playing actions that keepsM in the simple deterministic cycleˆqℓ ˆℓq−1 ···ˆq0. Now we prove thatq init is sure winning for the strongly synchronizing condition according to max Q. Letj≥k be such that j modℓ=0. LetR={⟨ˆq i, i⟩|0≤i<ℓ}. Thus obviously Outcomes(⟨q init,0⟩,β)⊆♦R. and sinceα agrees withβ on allfinite paths that do not (yet) visitR, given a pathρ that visitsR (for thefirst time), only actions that keepM in the simple cycleˆq ℓ ˆℓq−1 ···ˆq0 are played byα and thus all continuations ofρ in the outcome ofα will visitˆq 0 afterj steps. It follows

118 β j α α that Pr (♦ {⟨ˆ0q,0⟩}) = 1, that isM j (q0) = 1. Thus,∥M j ∥=1. Since next,α will always α play actions that keepsM looping through the cycleˆq ℓ ˆℓq−1 ···ˆq0, we have∥M n∥=1 for alln≥j. Second, assume that there exists a strategyα andk such that for alln≥k we have α α ∥Mn∥=1 from the initial stateq init. For alln≥k, letˆp n be a state such thatM n(ˆnp) = 1. Thefiniteness of the state spaceQ entails that in the sequenceˆp k ˆkp+1 ··· , some state and thus some simple deterministic cycle occur infinitely often. Letˆqℓ ˆℓq−1 ···ˆq0 be a cycle that occurs infinitely often in the sequenceˆpk ˆkp+1 ··· (in the right order). For allj, leti j be the position ofˆq0 in all occurrences of the cycleˆqℓ ˆℓq−1 ···ˆq0 in the sequenceˆpk ˆkp+1 ··· ; and let tj =i j modℓ. In the sequencet 0t1 ··· , there exists0≤t<ℓ that appears infinitely often. Let the cycler ℓrℓ−1 ···r 0 be such thatr (i+t) modℓ = îq. Then, the cycler ℓrℓ−1 ···r 0 happens infinitely often in the sequenceˆpk ˆkp+1 ··· such that the positions ofr 0 are infinitely often 0 (moduloℓ). Hence, for a strategyβ inM×[ℓ] that copies all the plays of the strategy α, we have Outcomes(⟨q init,0⟩,β)⊆♦{⟨r 0,0⟩} from the initial state⟨q init,0⟩. (2) almost-sure winning mode: This case is an immediate result from Lemma 6.1, by takingp=1. (3) limit-sure winning mode: First, assume that there exists a simple deterministic cycleˆqℓ ˆℓq−1 ···ˆq0 with lengthℓ such that⟨q init,0⟩ is limit-sure (and thus almost-sure) winning for the reachability condition♦{⟨ˆq0,0⟩}). Since⟨ˆqinit,0⟩ is almost-sure for reachability condition, thenq init is almost-sure (and thus limit-sure) for strongly synchronizing condition. Second, assume thatq init is limit-sure winning for the strongly synchronizing condition (according to max Q). It means that for alli there exists a strategyα i such that αi −i −k 1 lim infn→∞∥Mn ∥ ≥1−2 . Letk be such that1−2 ≥ 1+η . By Lemma 6.1, for alli≥k

there exists a simple deterministic cyclec i = ˆℓpi ˆℓpi−1 ···ˆp0 with lengthℓ i and a strategyβ i βi −i inM×[ℓ i] such that Pr (♦{⟨ˆ0q,0⟩})≥1−2 from⟨q init,0⟩. Since the number of simple deterministic cycle isﬁnite, there exists some simple cyclec that occurs inﬁnitely often in the sequencec kck+1ck+2 ··· . We see that for the cyclec=ˆq ℓ ˆℓq−1 ···ˆq0, the states⟨ˆqinit,0⟩ is limit-sure winning for the reachability condition♦{⟨ˆq0,0⟩}). !

Since the winning regions of almost-sure and limit-sure winning coincide for reachability conditions in MDPs [dAHK07], the next corollary follows from Lemma 6.2. strongly max strongly max Corollary 6.1. ⟨⟨1⟩⟩limit ( T ) = ⟨⟨1⟩⟩almost ( T ) for all MDPsM and target setsT. If there exists a cyclec satisfying the condition in Lemma 6.2, then all cycles reachable fromc in the graphG of deterministic transitions also satisﬁes the condition. Hence it is suﬃcient to check the condition for an arbitrary simple cycle in each strongly connected component (SCC) ofG. It follows that strong synchronization can be decided in PTIME (SCC decomposition can be computed in PTIME, as well as sure, limit-sure, and almost- sure reachability in MDPs). The length of the cycle gives a linear bound on the memory needed to win, and the bound is tight.

119 Lemma 6.3. For all sure, almost-sure and limit-sure winning modes, the membership problem for strongly synchronizing condition according to max T in MDPs is in PTIME.

Proof. Given an MDPM=⟨Q,A,δ⟩ and a stateq init, we say a simple deterministic cyclec=ˆq ℓ ˆℓq−1 ···ˆq0 is sure winning (resp., almost-sure and limit-sure) for strong synchronization fromq init if⟨q init,0⟩ is sure winning (resp., almost-sure and limit-sure) for the reachability condition♦{⟨ˆq0,0⟩} inM×[ℓ]. We show that ifc is sure winning (resp., almost-sure and limit-sure) for strong synchronization fromq init, then so are all simple ′ cyclesc = ˆℓp′ ˆℓp′−1 ···ˆp0 reachable fromc in the deterministic digraph induced byM. (1) sure winning: Sincec is sure winning for strong synchronization fromq init,M is1-synchronized inˆq 0. Since there is a path via deterministic transitions fromˆq0 toˆp0, ′ M is1-synchronized inˆp 0 too. So the cyclec is sure winning for strong synchronization fromq init, too. (2) limit-sure winning: Assume thatc is limit-sure winning for strong synchronization fromq init. By deﬁnition, for alli∈N, there existsn such that for allj>n we have −i−j −2i M is1−2 inˆq0. It implies that for alli there existsn such thatM is1−2 - synchronized inˆq0. Since there is a path via deterministic transitions fromˆq0 toˆp0, then −2i ′ M is1−2 -synchronized inˆp0 for alli. So the cyclec is limit-sure winning for strong synchronization fromq init, too. (3) almost-sure winning: By corollary 6.1, since a cycle is almost-sure winning for strong synchronization fromq init if and only if it is limit-sure winning, the results follows. The above arguments prove that if a simple deterministic cyclec is sure winning (resp., almost-sure and limit-sure) for strongly synchronizing condition fromq init, then all simple cycles reachable fromc in the graph of deterministic transitionsG induced byM, are sure winning (resp., almost-sure and limit-sure). In particular, it holds for all simple cycles in the bottoms SCCs reachable fromc inG. Therefore, to decide membership problem for strongly synchronizing condition, it suﬃces to only check whether one cycle in each bottom SCC ofG is sure winning (resp., almost-sure and limit-sure). Since the SCC decomposition for a digraph is in PTIME, and since the number of bottom SCCs in a connected digraph is at most the size of the digraph (the number of states|Q|), the PTIME upper bound follows. !

Since memoryless strategies are sufficient for reachability conditions in MDPs, it follows from the proof of Lemma 6.1 and Lemma 6.2 that the (memoryless) winning strategies inM×[ℓ] can be transferred to winning strategies with memory{0,1,···,ℓ−1} inM. Sinceℓ≤|Q|, linear-size memory is sufficient to win strongly synchronizing conditions. We present a family of MDPsM n (n∈N) that are sure winning for strongly synchronization (according to maxQ), and where the sure winning strategies require linear memory. The MDPM 2 is shown in Figure 6.1, and the MDPM n is obtained by replacing the cycleq 2q3 of deterministic transitions by a simple cycle of lengthn. Note that only inq 1 there is a real strategic choice. Since after thefirst action the two statesq 1 andq 2 always contain some probability, we need to wait inq 1 (by playingb) until we playa when the probability

120 b

1 q1 a, b: 2

qinit a a, b 1 a, b: 2 q2 q3 a, b target setT

Figure 6.1: An MDP that all strategies to win sure strongly synchronizing with function max {q2,q3} require memory.

inq 2 comes back inq 2 through the cycle. We need to playn−1 timesb, and thena, thus linear memory is suﬃcient and it is easy to show that it is necessary to ensure strongly synchronization. strongly max Lemma 6.4. Forµ∈{sure, almost, limit}, the membership problem for ⟨⟨1⟩⟩ µ ( T ) in MDPs is PTIME-hard even for a singletonT.

Proof. For allµ∈{sure, almost, limit} the proof is by a reduction from monotone Boolean circuit problem (MBC). Given an MBC with an underlying binary tree, the value of leaves are either0 or1 (called0 or1-leaves), and the value of other vertices, labeled with∧ or ∨, are computed inductively. It is known that deciding whether the value of the root is1 for a given MBC, is PTIME-complete [GR88]. From an MBC, we construct an MDPM where the states are the vertices of the tree with three new absorbing states synch,q 1 and q2, and two actionsL,R. On both actionsL andR, the next successor of the1-leaves is 1 only synch, and the next successor of the0-leaves isq 1 orq 2 with probability 2 . The next successor of a∧-state is each of their children with equal probability, on all actions. The next successor of a∨-state is its left (resp., right) child by actionL (resp., actionR). We can see thatM can be synchronized only in synch. We call a subtree complete if (1) root is in the subtree, (2) at least one child of all ∨-vertices is in the subtree, (3) both children of all∧-vertices are in the subtree. There is a bijection between a complete subtree and a strategy inM. The value of root is 1 if and only if there is a complete subtree such that it has no0-leaves (all leaves are1-leaves). For such subtrees, all plays under the corresponding strategy reach some1-leaf and thus are strongly synchronized in synch. It means that root ∈ ⟨⟨1⟩⟩µ (synch) if and only if the value of root is1. The proof is complete and the PTIME-hardness result is established. !

From previous lemmas and arguments, Theorem 6.1 follows.

121 1 a: 2 a 1 a: 2 a qinit q1 q2

Figure 6.2: An MDP such thatq init is sure-winning for coBüchi condition inT={q init, q2} but not for sure strong synchronization according to sum T .

Theorem 6.1. For the three winning modes of strong synchronization according to max T in MDPs: 1. (Complexity). The membership problem is PTIME-complete. 2. (Memory). Linear memory is necessary and suﬃcient for both pure and randomized strategies, and pure strategies are suﬃcient.

6.1.2 Strong synchronization with function sum

The strongly synchronizing condition with function sum T requires that eventually all the probability mass remains inT . We show that this is equivalent to a traditional reachability condition with target deﬁned by the set of sure winning initial distributions for the safety condition!T. It follows that almost-sure (and limit-sure) winning for strong synchronization is equivalent to almost-sure (or equivalently limit-sure) winning for the coBüchi condition♦!T. However, sure strong synchronization is not equivalent to sure winning for the coBüchi condition. Example 6.1 provides an MDP and a target setT such that the MDP is sure winning for coBüchi condition♦!T , but not for sure strongly synchronizing with sum T .

Example 6.1. Consider the MDPM depicted in Figure 6.2 that has three statesq init, q1, q2 and one lettera. Alla-transitions are deterministic except inq init where thea-transition 1 has two successorsq init andq 1 each with probability 2 . The MDPM is sure winning for the coBüchi condition♦!{q init, q2} fromq init, but not sure winning for the reachability condition ♦S whereS={q 2} is the winning region for the safety condition!{q init, q2} and thus not sure strongly synchronizing. The MDPM in Example 6.1 is almost-sure strongly synchronizing in targetT= {qinit, q2} fromq init, and almost-sure winning for the coBüchi condition♦!T , as well as almost-sure winning for the reachability condition♦S whereS={q 2}. Lemma 6.5. Given a target setT , an MDPM is sure (resp., almost-sure or limit-sure) winning for the strongly synchronizing condition according to sum T if and only ifM is sure (resp., almost-sure or limit-sure) winning for the reachability condition♦S whereS is the sure winning region for the safety condition!T.

122 Proof. First, assume that a stateq init ofM is sure (resp., almost-sure or limit-sure) winning for the strongly synchronizing condition according to sum T , and show thatq init is sure (resp., almost-sure or limit-sure) winning for the reachability condition♦S. ′ ϵ |Q| (i) Limit-sure winning. For allϵ>0, letϵ = |Q| ·η whereη is the smallest positive probability in the transitions ofM. By the assumption, fromq init there exists a strategyα α ′ andN∈N such that for alln≥N, we haveM n(T)≥1−ϵ . We claim that at stepN, all ϵ α ϵ non-safe states have probability at most |Q| , that isM N (q)≤ |Q| for allq∈Q\S. Towards α ϵ contradiction, assume thatM N (q)> |Q| for some non-safe stateq∈Q\S. Sinceq ̸∈S is not safe, there is a path of lengthℓ≤|Q| fromq to a state inQ\T , thus with probability |Q| α ϵ |Q| ′ at leastη . It follows that afterN+ℓ steps we haveM N+ℓ (Q\T)> |Q| ·η =ϵ , in α ′ contradiction with the factM n(T)≥1−ϵ for alln≥N. Now, since all non-safe states ϵ α ϵ have probability at most |Q| at stepN, it follows thatM N (Q\S)≤ |Q| · |Q|=ϵ and thus Prα(♦S)≥1−ϵ. ThereforeM is limit-sure winning for the reachability condition♦S fromq init. (ii) Almost-sure winning. Since almost-sure strongly synchronizing implies limit-sure strongly synchronizing, it follows from(i) thatM is limit-sure (and thus also almost- sure) winning for the reachability condition♦S, as limit-sure and almost-sure reachability coincide for MDPs [dAHK07]. (iii) Sure winning. Fromq init there exists a strategyα andN∈N such that for all α n≥N, we haveM n(T)=1. Henceα is sure winning for the reachability condition α α ♦Supp(MN ), and from all states in Supp(MN ) the strategyα ensures that only states in α T are visited. It follows that Supp(MN )⊆S is sure winning for the safety condition!T, and thusα is sure winning for the reachability condition♦S fromq init. For the converse direction of the lemma, assume that a stateq init is sure (resp., almost- sure or limit-sure) winning for the reachability condition♦S. We construct a winning strategy for strong synchronization inT as follows: play according to a sure (resp., almost- sure or limit-sure) winning strategy for the reachability condition♦S, and whenever a state ofS is reached along the play, then switch to a winning strategy for the safety condition!T . The constructed strategy is sure (resp., almost-sure or limit-sure) winning for strong synchronization according to sum T because for sure winning, afterfinitely many steps all paths fromq init end up inS⊆T and stay inS forever, and for almost-sure (or equivalently limit-sure) winning, for allϵ>0, after sufficiently many steps, the setS is reached with probability at least1−ϵ, showing that the outcome is strongly (1−ϵ)- synchronizing inS⊆T , thus the strategy is almost-sure (and also limit-sure) strongly synchronizing. ! strongly sum Corollary 6.2. For an MDPM and all target setsT , we have ⟨⟨1⟩⟩ limit ( T ) = strongly sum ⟨⟨1⟩⟩almost ( T ). The following result follows from Lemma 6.5 and the fact that the winning region for sure safety, sure reachability and almost-sure reachability can be computed in PTIME for MDPs [dAHK07]. Moreover, memoryless strategies are sufficient for these conditions.

123 Strongly max sum Sure PSPACE-complete PSPACE-complete Almost-sure PSPACE-complete PSPACE-complete Limit-sure undecidable

Table 6.2: Computational complexity of the emptiness problem of strongly synchronizing languages in PAs.

Theorem 6.2. For the three winning modes of strong synchronization according to sumT in MDPs: 1. (Complexity). The membership problem is PTIME-complete. 2. (Memory). Pure memoryless strategies are suﬃcient.

6.2 Strong synchronization in PAs

The same reduction mentioned to reduce the membership problem of strong synchronization with function max T to membership problem with function max Q in MDPs (see the beginning of Section 6.1), is valid for the emptiness problem of strongly synchronizing languages according to max in PAs. In the sequel, we thus study the emptiness problem of strongly synchronizing languages only according to max Q. The results presented in this section are summarized in Table 6.2.

6.2.1 Sure strong synchronization with function max In this subsection, we prove that the emptiness problem of sure strongly synchronizing languages with function max Q is PSPACE-complete. The PSPACE upper bound of the problem is obtained by the following intuitive characterization. strongly max Lemma 6.6. For a PAP, we haveL sure ( Q)≠ ∅ from an initial stateq init if and only if there exists some stateq such that event –L sure (q)≠ ∅ fromq init, always max – andL sure ( Q)≠ ∅ fromq.

strongly max Proof. First, we assume that the sure strongly synchronizing languageL sure ( Q) for the PAP is not empty. By deﬁnition, there exists a wordw andn 0 ∈N such that for w alln≥n 0, the probability mass∥P n ∥=1 is1 at every stepn. For all stepsn≥n 0, w letˆqn be the state in which the probability mass is accumulated:P n (ˆnq) = 1. So we see

124 PAP A NFAN a: 1 # a 2 sink qinit ⇒ qinit A a # 1 qT a: 2 #

Figure 6.3: The reduction sketch to show PSPACE-hardness of the emptiness problem for sure strongly synchronizing languages in PAs.

event thatL sure (qn0 )≠ ∅ fromq init. On the other hand, letv=a n0 an0+1 ··· be the subword

ofw=a 0a1 ··· obtained by cutting thefirstn 0 letters. Starting inq n0 , we see that v for alli∈N there is always a unique stateˆq n0+i such thatP i (ˆnq0+i) = 1, and thus always max Lsure ( Q)≠ ∅ fromq n0 . event Second, assume that there exists some stateq such thatL sure (q)≠ ∅ fromq init and always max Lsure ( Q)≠ ∅ fromq. Letw be the word that is sure eventually synchronizing in{q} w fromq init, andn be such thatP n (q)=1. Take the wordv=a 0a1 ···a n−1 that is the prefix ′ ofw=a 0a1 ··· consisting of thefirstn letters. Letv be the word that is sure always synchronizing according to max Q fromq. By definition, for alli∈N there is always a v′ ′ unique stateˆqi ∈Q such thatP i (îq) = 1. Concatenating the infinite wordv at the end v·v′ strongly max ofv, we haveP n+i(îq) = 1 for alli≥n meaning thatL sure ( Q)≠ ∅ from the initial stateq init. !

As we show in Theorem 4.6, the emptiness problem for sure eventually synchronizing languages in PAs is in PSPACE and as we show in Theorem 4.2, the emptiness problem for sure always synchronizing languages with function max is in PTIME, it follows from the characterization in Lemma 6.6 that the membership problem for sure strongly synchronizing language is in PSPACE, using the following (N)PSPACE algorithm: guess the stateq, event always max and check thatL sure (q)≠ ∅ fromq init andL sure ( Q)≠ ∅ fromq. Lemma 6.7 presents a matching lower bound by a reduction from the non-universality problem in NFAs. As an alternative to the reduction from the non-universality problem in NFAs, we could use the same reduction from theﬁnite automata intersection problem established to prove PSPACE-hardness of the emptiness problem of sure eventually synchronizing languages in Lemma 4.11. strongly max Lemma 6.7. The emptiness problem ofL sure ( T ) in PAs is PSPACE-hard even ifT is a singleton.

125 Proof. We present a proof using a reduction from the universality problem for NFAs that is known to be PSPACE-complete [IRS76]. Given a NFAN=⟨Q,A,∆⟩ equipped with an initial stateq init and a setF of accepting states, we construct a PAP and a singleton strongly max target setT such that the language ofN is universal if and only ifL sure ( T ) =∅ fromq init. The reduction is illustrated in Figure 6.3. The non-deterministic transitions ofN become probabilistic inP with a uniform distribution over successors. The target set is the absorbing stateT={q T }, a singleton. Therefore, a sure strongly synchronizing word needs to inject all that probability intoq T . This can be done with the new letter# from the non-accepting states of the NFA. From the accepting states, the#-transitions lead to a sink state sink from which there is no way to synchronize intoq T The construction of the PAP=⟨Q ′,A ′,δ⟩ is such that the state spaceQ ′ =Q∪ ′ {qT , sink} has two new states which are only accessible with a new letter#, andA = ′ 1 A∪{#}. For allq∈Q anda∈A, the probabilistic transition functionδ(q, a)(q ) = |∆(q,a)| ifq ′ ∈∆(q, a), andδ(q, a)(q ′) = 0 otherwise. For alla∈A∪{#}, thea-transition in both statesq T and sink are self-loops meaning that both these states are absorbing. The #-transitions in all non-accepting statesq∈Q\F bring the PA deterministically intoq T , while the#-transitions in all accepting statesq∈F is redirected to sink. To establish the correctness of the reduction, wefirst assume that the language ofN is not universal. Thus, there exists somefinite wordw such that all runs ofN overw end in a non-accepting state, and inputtingw·# toP the stateq T is reached with probability1. ω Therefore,w· (#) is a sure strongly synchronizing word in{q T } forP fromq init. strongly Second, assume that there exists some infinite wordv such thatv∈L sure (qT ) v v v fromq init inP. LetP 0 P1 P2 ··· be the symbolic run ofP overv and letn 0 be such v thatP n(qT ) = 1 for alln>n 0. The wordv must contain#, as this is the only transition ∗ to the target stateq T . Letw∈A be the prefix ofv before thefirst occurrence of#. We claim thatw is not accepted by the NFAN . By contradiction, if there is an accepting run ρ ofN overw, then positive probability is injected in sink by thefinite wordw·# and v stays there forever, in contradiction with the fact thatP n(qT ) = 1 for alln>n 0. Therefore w is not accepted by the NFAN , and the language of the NFA is not universal. !

From previous Lemmas and argument, Theorem 6.3 follows.

Theorem 6.3. The emptiness problem for sure strongly synchronizing languages with function max T in PAs is PSPACE-complete.

6.2.2 Almost-sure strong synchronization with function max We present a construction to reduce the emptiness problem for almost-sure strongly synchronizing languages according to max Q in PAs to the emptiness problem forω-automata

126 with Büchi condition. The construction is exponential; however thanks to NLOGSPACE- completeness of the emptiness problem for Büchi automata, PSPACE upper bound follows. We also prove a matching lower bound.

Lemma 6.8. The emptiness problem of almost-sure strongly synchronization with function max Q in PAs, is in PSPACE.

Proof. Given a PAP=⟨Q,A,δ⟩ with an initial stateq init, we construct an automatonN strongly max equipped with an initial stateℓ 0 and a Büchi condition!♦F such thatL almost ( Q) =∅ fromq init if and only ifL !♦F (N)=∅. The automatonN is exponential in the size ofP, and thus the PSPACE bound follows from the NLOGSPACE-completeness of the emptiness problem for Büchi automata. The construction ofN relies on the following intuition. The symbolic run of the PAP w w w under an almost-sure strongly synchronizing wordw is the sequenceP 0 P1 P2 ··· of proba- w bility distributionsP i (i∈N) in which the probability mass tends to accumulate in a single stateˆqi at stepi. We prove that for all sufficiently largei, there exists a deterministic transition fromˆqi toˆqi+1 meaning that there exists somea∈A such that post(ˆq i, a) ={ˆqi+1} in the PAP. The Büchi automatonN will guess the witness sequenceˆq i îq+1 ... and check that the probability mass is injected into this sequence. The automatonN keeps track w of the supports i = Supp(Pi ) of the symbolic run, and at some point guesses that the witness sequenceˆqi îq+1 ... starts. Then, using an obligation seto i ⊆s i, it checks that every state ins i eventually injects some probability mass in the witness sequence. When the obligation set gets empty, it is recharged with the current supports i. We construct the Büchi automatonN=⟨L,A,∆⟩ as follows. The setL=2 Q ∪ (2Q × 2Q ×Q) is the set of states. A states⊆Q is the support of the current probability distribution. A state(s, o,ˆq)∈2 Q ×2 Q ×Q consists of the supports, the obligation set o⊆s, and a stateˆq∈s of the witness sequence. The alphabetA is in common with the PAP. The transition function∆:L×Σ→2 L is defined as follows. – For alls∈2 Q anda∈A, lets ′ = post(s, a), and define

∆(s, a) ={s ′}∪{(s ′, s′,ˆq)|ˆq∈s ′}.

A transition in the states which leads to a state(s ′, s′,ˆq), guessesˆq as the initial state of the witness sequence. – For all(s, o,ˆq)∈2 Q ×2 Q ×Q anda∈A, lets ′ = post(s, a). If post(ˆq, a) is not a singleton, then∆((s, o,ˆq), a) =∅, otherwise let{ˆq ′}= post(ˆq, a), and: – Ifo≠ ∅, then

∆((s, o,ˆq), a) ={(s′, o′ \{ˆ′q},ˆq′)|∀q∈o:o ′ ∩ post(q,a)≠ ∅}.

These transitions deterministically choose the next stateˆq′ of the witness sequence, and in addition, non-deterministically take care of paths from the obligation seto to the witness sequence. For this sake, the constrainto ′ ∩ post(q, a)≠ ∅ is required for allq∈o.

127 – Ifo=∅, then∆((s, o,ˆq), a) ={(s ′, s′,ˆq′)}. This transition is to recharge the obligation set with the current supports ′ when it gets empty. The automatonN is equipped with the initial stateℓ 0 ={q init} and with the Büchi condition!♦F whereF={(s, o,ˆq)∈2 Q ×2 Q ×Q|o=∅}. is the set of states with an empty obligation seto. To establish the correctness of the reduction, weﬁrst assume thatL !♦F (N)≠ ∅ and strongly max show thatL almost ( Q)≠ ∅ fromq init. Since the Büchi language ofN is not empty, there exists some inﬁnite wordw=a 0a1a2 ··· inducing an ultimately periodic runρ= ω r0r1 . . . rn−1(rn . . . rm−1) overN such thatr n ∈F. Letr m =r n andκ=m−n be the size of the cycle fromr n to itself. From the construction of Büchi automaton, we know w that allr i (wheren≤i≤m) are states of the form(s i, oi,ˆqi) wheres i = Supp(Pi ) is the support of the symbolic run at stepi (and also at stepsi+(j·κ) for allj∈N). As w a result,P i (q)>0 for all statesq∈s i. Sinceˆqn ∈s n, the stateˆqn is reached at stepn with some positive probabilityp>0. Since for alln≤i

w κ Pm(Q\{ˆqm})≤ (1−p)· (1−η ).

Visiting the states in the periodic part of the runρ forj times leads to

w κ j Pm+j·κ(Q\{ˆqm+j·κ})≤ (1−p)(1−η ) .

The probability outside the witness sequence then tends to0 whenj→∞, i.e., w w limj→∞ Pm+j·κ(Q\{ˆqm+j·κ}) = 0 implying that limi→∞∥Pi ∥=1. strongly max Second, assume thatw∈L almost ( Q) fromq init. We assume thatw=a 0a1a2 ... is 2 w w pure . LetP 0 P1 ... be the symbolic run ofw overP. Sincew is almost-sure strongly synchronizing, then∀ϵ>0 there existsn 0 ∈N such that for alln≥n 0 there exists some w 1 stateˆq such thatP n (ˆq)>1−ϵ. By assumingϵ< 2 , the stateˆq is unique in each step

′ 1. Formally,η = min q,q′∈Q,a∈A(δ(q, a)(q )). 2. To generalize the results to consider randomized words, the alphabet of the Büchi automatonN could be deﬁned2 A whereA is the alphabet ofP. An inﬁnite word over2 A is a sequence of sets of letters which can be viewed as the sequence of supports of a randomized word. For the sake of simplicity, we prove the results for pure words.

128 n≥n 0; we therefore denote this state byˆqn. Moreover, it is easy to see that the stateˆqn is independent ofϵ. We prove that for alln≥n 0, the stateˆqn+1 is the unique successor ofq n anda n: η post(ˆnq, an) ={ˆqn+1}. Letϵ< 1+η whereη is the smallest probability in the transitions ofP. Towards contradiction, assume that there exists somen>n 0 such that post(ˆnq, an)≠ {ˆq n+1}; in other words, there exists another stateq≠ ˆq n+1 such that {q,ˆqn+1}⊆ post(ˆqn, an). By next transition, the probability inq then is

w w Pn+1(q)≥δ(ˆqn,σ n)(q)·P n (ˆnq)≥η· (1−ϵ).

w The probabilityP n+1(ˆnq+1) in the witness stateˆqn+1 is thus at most1−η· (1−ϵ). The computed upper bound for the probability in the witness stateˆqn+1 and the lower bound coming from the deﬁnition of almost-sure strongly synchronizing gives

w 1−ϵ≤∥P n+1∥ ≤1−η· (1−ϵ)

η that results inϵ≥ 1+η , a contradiction. Therefore for alln≥n 0, the stateˆqn+1 is the unique successor ofˆqn anda n. For a given stateq and wordv∈A +, we deﬁne post(q,v) as the natural extension of post(q,a) over words: post(q,v) = post(post(q,v ′), a) wherev=v ′ ·a andv ′ ∈A +. To complete the proof, we construct an accepting runρ ofN overw. For theﬁrstn 0 w transitions, the runρ visitss i wheres i = Supp(Pi ) and0≤i0. Since the probability mass accumulated in the witness sequence would be bounded by1−p for all next steps aftern, a contradiction arises with the fact thatw is almost-sure strongly synchronizing. As a result of above argument, for alln≥n 0 and all states inq∈s n, there exists some pathp 0p1 ···p k that is fromq to the witness sequence; letk(q)=k denote the length of this path and letp(q, i) =p i denote thei-th state visited along this path. Let

K = maxq∈sn k(q) be the length of the longest paths starting from some stateq∈s n to the

witness sequence. The accepting runr after the state(s n0 , on0 ,ˆqn0 ) visits the followingK

states(s n0+1, on0+1,ˆqn0+1)···(s n0+K , on0+K ,ˆqn0+K ) consecutively, where for all0

– the obligation seto n0+i =O\{ˆqn0+i} withO= {p(q, i)}, and q∈sn0 :k(q)≥i

–ˆqn0+i is the witness state at stepn 0 +i.

By the choice of obligation sets, we see thato n0+K =∅ andr n0+K ∈F. The automatonN,

by next transition, deterministically, reset the obligation set withs n0+K+1 . With a similar construction, the runr visits the accepting states inﬁnitely often andw is accepting. !

129 strongly max Lemma 6.9. The emptiness problem ofL almost ( T ) in PAs is PSPACE-hard even ifT is a singleton.

Proof. The proof is by the same reduction from the non-universality problem for NFAs that is presented in Lemma 6.7 where from a NFAN=⟨Q,A,∆⟩ equipped with an initial stateq init and a setF of accepting states, we construct a PAP and a singleton target strongly stateq T such that the language ofN is universal if and only ifL sure (qT ) =∅ fromq init. We now argue that in the constructed PAP, the sure and almost-sure strongly synchro- max strongly strongly nizing languages with function qT are equal:L sure (qT ) =L almost (qT ). By deﬁnition, we know that all sure strongly synchronizing words are almost-sure strongly synchronizing too. To complete the proof, we thus assume thatw is an almost-sure strongly synchroniz- strongly ing word in{q T }, and prove thatw∈L sure (qT ) too. Letϵ>0. Sincew is almost-sure w strongly synchronizing, there exists somen∈N such thatP n (qT )≥1−ϵ. All#-transitions ′ inP are redirected to either sink orq T : post(Q , #) ={sink, q T }. This and the fact thatq T w is only reachable by#-transitions give that Supp(P n )⊆{q T , sink}. Hence, the remaining probability mass at stepn is accumulated in either of those statesq T , sink. On the other w w hand since sink is a self-loop, ifP n (sink)>0 thenP i (sink)>0 for all next stepsi>n. It means that in the case sink contains some positive probabilityp at stepn, the probability w inq T is bounded away from1 for all next stepsi>n:P i (qT )<1−p, a contradiction with the fact thatw is almost-sure strongly synchronizing inq T . Hence, the probability inq T at stepn is1, and we see that it remains1 forever irrespective to the next inputs. It proves thatw is a sure strongly synchronizing word inq T too. !

From the Lemma 6.8 and Lemma 6.9, we conclude Theorem 6.4.

Theorem 6.4. The emptiness problem for almost-sure strongly synchronizing languages with function max T in PAs is PSPACE-complete.

In subsection 6.2.5, we provide undecidability result for the emptiness problem of limit- sure strong synchronization with both functions max and sum.

6.2.3 Sure strong synchronization with function sum In the sequel, we prove that the emptiness problem of sure strongly synchronizing languages with function sumT is PSPACE-complete. The PSPACE upper bound of the problem is obtained by Lemma 6.10. Since the hardness result in Lemma 6.7 is proved for singleton target setT , the result holds for both functions max T and sumT . strongly sum Lemma 6.10. LetP be a PA andT be a target set. We haveL sure ( T )≠ ∅ from an initial stateq init if and only if there exists some setS⊆T such that event sum –L sure ( S)≠ ∅ fromq init,

130 always sum – andL sure ( T )≠ ∅ from an arbitrary distribution overS. strongly sum Proof. First, we assume that the sure strongly synchronizing languageL sure ( T ) for the PAP is not empty. By deﬁnition, there exists a wordw andn 0 ∈N such that for w alln≥n 0, the probability massP n (T) is1 at every stepn. For all stepsn≥n 0, let w Sn ⊆T be the set of statesq∈T with strictly positive probability:S n = Supp(Pn ). Let w event sum S=S n0 . ThenP n0 (S)=1 implying thatL sure ( S)≠ ∅ fromq init. On the other hand,

letv=a n0 an0+1 ··· be the subword ofw=a 0a1 ··· obtained by cutting thefirstn 0 letters. w Starting in the distributiond whered=P n0 with supportS, we see that for alli∈N there v always sum is always a subsetS n0+i ⊆T such thatP i (Sn0+i) = 1, and thusL sure ( T )≠ ∅ fromd. event sum Second, assume that there exists some setS⊆T such thatL sure ( S)≠ ∅ fromq init always sum andL sure ( T )≠ ∅ from some distributiond overS. Letw be the word that is sure w ′ w eventually synchronizing inS fromq init, andn be such thatP n (S)=1. Letd =P n with always sum ′ always sum supportS. By Remark 8, we know thatd∈L sure ( T ) if and only ifd ∈L sure ( T ) Take the wordv=a 0a1 ···a n−1 that is the prefix ofw=a 0a1 ··· consisting of thefirstn 0 letters. Letv ′ be the word that is sure always synchronizing inS fromd ′. By definition, v′ for alli∈N there is always a setS i ⊆T such thatP i (Si) = 1. Concatenating the infinite ′ v·v′ strongly sum wordv at the end ofv, we haveP n+i(T)=1 for alli≥n meaning thatL sure ( T )≠ ∅ from the initial stateq init. !

As we show in Theorem 4.6 and Theorem 4.2 that the emptiness problems for both sure eventually synchronizing languages and sure always synchronizing languages with function sum are in PSPACE, it follows from the characterization in Lemma 6.10 that the membership problem for sure strongly synchronizing language is in PSPACE, using the event sum following (N)PSPACE algorithm: guess the subsetS⊆T , and check thatL sure ( S)≠ ∅ always sum fromq init andL sure ( S)≠ ∅ from the uniform distribution onS. From previous Lemma and discussion, Theorem 6.5 follows.

Theorem 6.5. The emptiness problem for sure strongly synchronizing languages with function sumT in PAs is PSPACE-complete.

6.2.4 Almost-sure strong synchronization with function sum We present a reduction from the emptiness problem for almost-sure strongly synchronizing languages with function sum T to the emptiness problem for Büchi automata. The construction is similar to Proof of Lemma 6.8 and it is exponential in the size of the PA. The (N)PSPACE membership thus follows from the NLOGSPACE-completeness of the emptiness problem for Büchi automata. Lemma 6.11. The emptiness problem of almost-sure strongly synchronization with function sumT in PAs, is in PSPACE.

131 Proof. Given a PAP=⟨Q,A,δ⟩ with an initial stateq init, we construct an automatonN strongly sum equipped with an initial stateℓ 0 and a Büchi condition!♦F such thatL almost ( T ) =∅ fromq init if and only ifL !♦F (N)=∅. The automatonN is exponential in the size ofP; however thanks to the NLOGSPACE-completeness of the emptiness problem, the PSPACE membership follows. The construction ofN relies on the following intuition (similarly to Proof of Lemma 6.8 where the Büchi automaton guesses the single stateˆq in which the mass of probability is accumulated, here the automaton guesses the setc of target states which contribute in accumulating the probability mass). The symbolic run of the PAP under an almost-sure w w w strongly synchronizing wordw is the sequenceP 0 P1 P2 ··· of probability distributions w Pi (i∈N) in which the probability mass tends to accumulate in some subsetc i ⊆T of the target setT at stepi. We prove that for all sufficiently largei, there exists somea-transition fromc i toc i+1 such thatc i+1 = post(ci, a) in the PAP. The Büchi automatonN will guess the witness sequencec ici+1 ... and check that the probability mass is injected into w this sequence. The automatonN keeps track of the supports i = Supp(Pi ) of the symbolic run, and at some point guesses that the witness sequencec ici+1 ... starts. Then, using an obligation seto i ⊆s i, it checks that every state ins i eventually injects some probability mass in the witness sequence. When the obligation set gets empty, it is recharged with the current supports i. We construct the Büchi automatonN=⟨L,A,∆⟩ as follows. The setL=2 Q ∪ (2Q × 2Q ×2 Q) is the set of states. A states⊆Q is the support of the current probability distribution. A state(s, o, c)∈2 Q ×2 Q ×2 Q consists of the supports, the obligation set o⊆s, and the subsetc⊆s∩T of the witness sequence. The alphabetA is in common with the PAP. The transition function∆:L×Σ→2 L is defined as follows. – For alls∈2 Q anda∈A, lets ′ = post(s, a), and define ∆(s, a) ={s ′}∪{(s ′, s′, c)|c⊆T∩s ′}. A transition in the states which leads to a state(s ′, s′, c), guessesc as the initial state of the witness sequence. – For all(s, o, c)∈2 Q ×2 Q ×2 Q anda∈A, lets ′ = post(s, a). If post(c, a) ̸⊆T is not included inT , then∆((s, o, c), a) =∅, otherwise letc ′ = post(c, a), and: – Ifo≠ ∅, then ∆((s, o, c), a) ={(s ′, o′ \c ′, c′)|∀q∈o:o ′ ∩ post(q,a)≠ ∅}. These transitions deterministically choose the next subsetc ′ of the witness sequence, and in addition, non-deterministically take care of paths from the obligation seto to the witness sequence. For this sake, the constrainto ′ ∩ post(q,a)≠ ∅ is required for allq∈o. – Ifo=∅, then∆((s, o, c), a) ={(s ′, s′, c′)}. This transition is to recharge the obligation set with the current supports ′ when it gets empty. The automatonN is equipped with the initial stateℓ 0 ={q init} and with the Büchi condition!♦F whereF={(s, o, c)∈2 Q ×2 Q ×2 Q |o=∅} is the set of states with an empty obligation seto.

132 To establish the correctness of the reduction, weﬁrst assume thatL !♦F (N)≠ ∅ and strongly sum show thatL almost ( T )≠ ∅ fromq init. Since the Büchi language ofN is not empty, there exists some inﬁnite wordw=a 0a1a2 ··· inducing an ultimately periodic runρ= ω r0r1 . . . rn−1(rn . . . rm−1) overN such thatr n ∈F. Letr m =r n andκ=m−n be the size of the cycle fromr n to itself. From the construction of Büchi automaton, we know w that allr i (wheren≤i≤m) are states of the form(s i, oi, ci) wheres i = Supp(Pi ) is the support of the symbolic run at stepi (and also at stepsi+(j·κ) for allj∈N). w As a result,P i (q)>0 for all statesq∈s i. Sincec n ⊆s n, all the statesq∈c n are reached at stepn with some positive probabilityp>0. Sincec i+1 = post(ci, ai) , we have w w w Pi+1(ci+1)≥P i (ci)≥p, and we will prove that lim inf i→∞ Pi (ci) = 1. Sincer n ∈F, this state is of the form(s n,∅, c n) where the obligation set is empty. Consequentlyr n+1 = (sn+1, sn+1, cn+1) where the obligation seto n+1 =s n+1 is recharged. ′ Leto i =o i ∪c i for alln≤i≤m. By a backward induction, we show that there are paths going through each state ofo i and ending in some stateq∈c m. The base of induction ′ ′ ′ ′ holds becauseo m =c m and post(q , am−1)∩o m ≠ ∅ for allq ∈o m−1. The induction holds ′ ′ ′ for alln≤i≤m too, thanks to the fact that post(q , ai−1)∩o i ≠ ∅ for allq ∈o i−1. ′ Therefore, from all statesq ∈s n+1 some stateq∈c m is reached withinm−n steps, with some positive probability that is at leastη κ whereκ=m−n andη is the smallest w w probability of taking a transition inP. Recall thatP n (cn) =p andP n (Q\c n)≤1−p. We thus have w κ Pm(Q\c m)≤ (1−p)· (1−η ). Visiting the states in the periodic part of the runρ forj times leads to

w κ j Pm+j·κ(Q\c m+j·κ)≤ (1−p)(1−η ) . The probability outside the witness sequence then tends to0 whenj→∞. That means w w limj→∞ Pm+j·κ(Q\c m+j·κ) = 0 implying that limi→∞ Pi (T)=1. Second, assume that the almost-sure strongly synchronizing language with func- sum always sum tion T ofP is not empty. LetS be the set of initial statesq such thatL sure ( T )≠ ∅. We know that the setS is equivalently the set of all initial states from which theω-language strongly sum for the safety condition!T inP is non-empty. Letw∈L almost ( T ) fromq init. We as- 3 w w sume thatw=a 0a1a2 ... is pure . LetP 0 P1 ... be the symbolic run ofw overP. 1 ′ ϵ |Q| Considerϵ> 2 and letϵ = |Q| ·η whereη is the smallest positive probability in the transitions ofP. By deﬁnition, there existsN∈N such that for alln≥N, we have w ′ Pn (T)≥1−ϵ . We claim that at stepN, all non-safe statesq∈Q\S have probability ϵ w ϵ at most |Q| , that isP N (q)≤ |Q| for allq∈Q\S. Towards contradiction, assume that w ϵ PN (q)> |Q| for some non-safe stateq∈Q\S. Sinceq ̸∈S is not safe, there is a path of lengthℓ≤|Q| fromq to a state inQ\T , thus with probability at leastη |Q|. It follows w ϵ |Q| ′ that afterN+ℓ steps we haveP N+ℓ (Q\T)> |Q| ·η =ϵ , in contradiction with the fact

3. To generalize the results to consider randomized words, the alphabet of the Büchi automatonN could be deﬁned2 A whereA is the alphabet ofP. An inﬁnite word over2 A is a sequence of sets of letters which can be viewed as the sequence of supports of a randomized word. For the sake of simplicity, we prove the results for pure words.

133 w ′ ϵ Pn (T)≥1−ϵ for alln≥N. Now, since all non-safe states have probability at most |Q| at w ϵ w w stepN, it follows thatP N (Q\S)≤ |Q| · |Q|=ϵ. ThusP N (S)≥1−ϵ, andP n (S)≥1−ϵ for alln≥N. For a given stateq and wordv∈A +, we deﬁne post(q,v) as the natural extension of post(q,a) over words: post(q,v) = post(post(q,v ′), a) wherev=v ′ ·a andv ′ ∈A +. To complete the proof, we construct an accepting runρ ofN overw. For theﬁrstN w transitions, the runρ visitss i wheres i = Supp(Pi ) and0≤i0. Since the probability mass accumulated in the witness sequence would be bounded by1−p for all next steps aftern, a contradiction arises with the fact thatw is almost-sure strongly synchronizing. As a result of above argument, for alln≥n 0 and all states inq∈s n, there exists some pathp 0p1 ···p k that is fromq to the witness sequence; letk(q)=k denote the length of this path and letp(q, i) =p i denote thei-th state visited along this path. Let

K = maxq∈sn k(q) be the length of the longest paths starting from some stateq∈s n to the witness sequence. The accepting runr after the state(s N , oN , cN ) visits the followingK states(s N+1 , oN+1 , cN+1 )···(s N+K , oN+K , cN+K ) consecutively, where for all0

Theorem 6.6. The emptiness problem for almost-sure strongly synchronizing languages with function sum T in PAs is PSPACE-complete.

6.2.5 Limit-sure strong synchronization We prove that the emptiness problems for limit-sure strongly synchronizing languages according to both functions max T and sumT are undecidable, by a similar construction used in [GO10] to establish the undecidability result for the value1 problem in PAs.

134 1 qinit 1 a, b a, b: 2 a, b: 2 a, b

b b q6 q4 q1 q3 a:1−x a:x b b a:x a:1−x q5 q2

a a

1 Figure 6.4: An automaton presented in [GO10], that has value1 if and only ifx> 2 .

Recall that, given a PAN and a threshold0<λ<1, the strict emptiness problem is undecidable: this problem asks whether the languageL λ(N)={w∈A ∗ | Pr(w)>λ} is empty. To prove undecidability of value1 problem, a PAP is presented where the probability of some transitions are parametrized withx; and it is proved thatP has value1 1 if and only ifx> 2 ; see Example 6.2. The undecidability of value1 problem follows from 1 λ 1 combiningP and another PAN such thatx> 2 if and only ifL (N)=∅ withλ= 2 . We refer the reader to [GO10] for details.

Example 6.2. LetP=⟨Q,A,δ⟩ be the PA depicted in Figure 6.4 whereA={a, b} andQ={q init, q1,···,q 6}. On all letters, the successors ofq init areq 1 andq 4 each with 1 probability 2 . Two statesq 1 andq 4 are in some sense symmetric. Thea-transitions have two successors one with probabilityx and another with probability1−x: inq 1, the next successor isq 1 with probabilityx and isq 2 with probability1−x whereas inq 4, the next successor is itself with probability1−x and isq 5 with probabilityx. Theb-transitions are redirected toq 3 andq 6, respectively. Two statesq 2 andq 5 are also somehow symmetric: thea-transitions are self-loops while theb-transitions are deterministically redirected toq 1 andq 4, accordingly. Two statesq 3 andq 6 are symmetrically absorbing states. n n Starting inq 1 the probability of the worda ·b to be accepted in{q 3} isx , and similarly n n starting inq 4 the probability of the worda ·b to be accepted in{q 6} is (1−x) . By construction of the automaton, all accepting words are in the shapew=(a+b)·(a+b)· ∗ ∗ ∗ n0 n1 ni (a ·b) ·a . For a sequencen 0n1n2 ··· and for alli∈N, letw i =a ·b·a ·b···a be a ﬁnite word. Starting inq 1 the probability of the wordw i to be accepted in{q 3} is ( (1− (1−x nj )). 0≤j

Starting inq 4 the probability of the wordw i to be accepted in{q 6} is ( ) (1− (1− (1−x) nj )< (1−x nj ). 0≤j≤i 0≤j≤i

135 LetF={q 3, q5} be the accepting states andq init be the initial state. We see that after 1 theﬁrst input letter, two statesq 1 andq 4 both contains probability 2 . Starting fromq init, the probability of the wordsv i ∈(a+b)·w i to be accepted inF thus is ( ( 1 1 Pr(v ) = (1− (1−x nj )) + ( (1− (1−x) nj )). i 2 2 0≤j

There are two cases: 1 (i.) Letx≤ 2 . By the above computation, considering the accepting set{q 3}, after inputting a word the probability inq 3 is less than the probability inq 6. As a result, the 1 probability of accepting all words is less than 2 . 1 (ii.) Letx> 2 . It is proved that for allϵ>0 there exists some increasing sequencen 0 < n1 < n2 <··· such that ( ) (1−x nj )→0 and (1−x nj )<ϵ 0≤j

wheni→∞. It means that there is a careful way of choosing the sequencen 1n2n3 ··· ∗ where after reading each subworda ·b ofv i (for suﬃciently largei)q 3 contains larger probability than what is lost inq 6. Hence, the probability of the wordsv i to be accepted inF is at least1−ϵ. 1 It proves thatP has value1 if and only ifx> 2 . This example is presented in [GO10] to establish the undecidability result of the value1 problem. ▹ We brieﬂy recall the reduction from the strict emptiness problem to the value1 problem [GO10]. Remark9. The value1 problem in PAs is undecidable. Proof (sketch). LetP=⟨Q,A,δ⟩ be the PA described in Example 6.2 and letN bea PA whose alphabetA ′ has no intersection withA. The automatonN is equipped with an initial state and the setF of accepting states. ′ ′ LetN 1 =⟨Q 1,A ,δ 1⟩ andN 4 =⟨Q 4,A ,δ 4⟩ be two copies ofN . FromN 1,N 4 andP we 1 construct another PAB=⟨Q B,A B,δ B⟩ such thatL 2 (N)=∅ if and only ifB has value1. The sketch of constriction is as follows. ′ – Let the alphabet ofB beA B =A ∪{b,⋆}. – Copy all states ofP intoB. Add allb-transitions inP toB. – Copy all states ofN 1 andN 4 intoB such that the initial state ofN 1 is the (copy of) ′ stateq 1 ofP and the initial state ofN 4 is the (copy of) stateq 4 ofP. For alla∈A , add alla-transitions inN 1 andN 4 toB. ′ – For alla∈A , add a self-loopa-transition in all statesq∈{q init, q2, q3, q5, q6} toB.

136 – From all copies of statesq∈Q 1 ∪Q 4 ofN 1 andN 4 except initial statesq 1 andq 4, add ab-transition directed to the absorbing stateq 6. – For all copies of statesq∈Q 1 ofN 1, the⋆-transition goes to the (initial) stateq 1 if q is an accepting state forN 1, and goes to the stateq 2 otherwise. – For all copies of statesq∈Q 4 ofN 4, the⋆-transition goes to the stateq 5 ifq is an accepting state forN 4, and goes to the (initial) stateq 4 otherwise. To establish the correctness of reduction. Assume that there exists some wordw forN 1 n0 n2 ni such that Pr(w)> 2 . To prove thatB has value1, letϵ>0 and letv=b·a ·b·a ···a ·b be a word that is accepted byP with probability1−ϵ, see Example 6.2. By construction ofB, one can verify that the wordb·(w·⋆) n0 ·b·(w·⋆) n2 ···(w·⋆) ni ·b is accepted byB with probability1−ϵ. Thus,B has value1. Second, assume thatB has value1. Towards contradiction, assume that for all wordsw 1 the probability to be accepted byN is equal or less than one half: Pr(w)≤ 2 . Let w′ ∈A ′ ∪{b,⋆} be an arbitrarilyﬁnite accepting word forB. By construction ofB, the ′ ∗ ′∗ wordw can be written asu 0 ·v 0 ·⋆·u 1 ·v 1 ·⋆··· whereu i ∈b andv i ∈A . By construction ofB, the automatonB acceptsw with the probability less than the acceptance probability 1 1 ofw=u 0 ·a·u 1 ·a··· byP wherex≤ 2 , that is less than 2 . It gives a contradiction with the fact thatB has value1. !

Theorem 6.7 provides the undecidability result of the emptiness problems for limit-sure strongly synchronizing languages with both functions max T and sumT .

Theorem 6.7. For all functionsf∈{max T , sumT }, the emptiness problem of limit- sure strongly synchronizing language in PAs is undecidable.

Proof. The proof is by a similar construction used in [GO10] to establish the undecidability result for the value1 problem in PAs. Consider the PAP described in Example 6.2. From ′ the PAP=⟨Q,A,δ⟩, we construct another PAP and a target setq T such thatP has ′ value1 if and only ifP is limit-sure strongly synchronizing in{q T }. The construction ofP ′ =⟨Q ′,A ′,δ ′⟩ is as follows. The PAP ′ is a copy ofP where two new absorbing statesq T and sink are added that are only accessible by a new letter#. ′ ′ ′ Formally,Q =Q∪{q T , sink} andA =A∪{#}. All transitions inP are copied toP , soδ ′(q, a) =δ(q, a) for allq∈Q anda∈A. The following transitions are added: all #-transitions in accepting statesq 3 andq 5 are directed toq T ; all#-transitions in non- accepting statesq∈Q\{q 3, q5} are directed to sink. Both new statesq T and sink are absorbing. Below, we prove thatP has value1 if and only ifP ′ is limit-sure strongly synchronizing in{q T }. First, assume thatP has value1. By deﬁnition, for allϵ>0 there exists a wordw such that Pr(w)≥1−ϵ. Taking the#-transition after such wordsw, we see that

137 ω ′ v∈w· (#) is (1−ϵ)-synchronized inq T . Thus,P is limit-sure strongly synchronizing in{q T }. strongly Now, assume thatL sure (qT )≠ ∅. Letϵ>0, and letw be such thatw is (1−ϵ)- synchronized inq T . Since the only in-going transitions toq T are#-transitions, thenw must have some#. Letv be the prefix ofw from the beginning up to thefirst occurrence of#, so thatv does not contain any# and is a valid word forP. ′ Since post(Q , #) ={q T , sink}, after reading the prefixv·# ofw the only states assigned with some possibly positive probability are sink andq T . Since both of these states are absorbing, then the assigned probabilities would never changed, no matter what is the next input letters. Thus, ifw is (1−ϵ)-synchronized inq T , thenv is (1−ϵ)-synchronized in{q 3, q5} implying that Pr(v)≥1−ϵ. The above argument proves thatP has value1 if and only ifP ′ is limit-sure strongly synchronizing in{q T }. The same reduction, presented in Remark 9, from the strict emptiness problem to the value1 problem is used to complete the proof. !

6.3 Discussion

In this section, we brieﬂy discuss the universality problems for sure and almost-sure strongly synchronizing languages in PAs.

Universality problems of sure strongly synchronizing language. Consider a PAP with the universal sure strongly synchronizing language according to max T . By deﬁnition, w w w for all wordsw the symbolic runP 0 P1 P2 ··· is ultimately always1-synchronized in a |Q| singleton{q} whereq∈T , i.e., there exists somen 0 ≤2 (whereQ is the state space w ofP) such that for alln>n 0 there exists some stateq∈T whereP n (q)=1. We claim |Q| that for alln≥2 there exists a unique stateˆqn such that for all wordsw we have w ′ Pn (ˆnq) = 1. Toward contradiction, assume that there existsn, two statesq andq and ′ w w ′ two wordsw andw such thatP n (q)=1 andP n (q ) = 1. Thus, the symbolic run of the 1 1 ′ automaton over the randomized wordv= 2 w+ 2 w is not1-synchronized in any state; v max i.e.,∥P n∥T ≠ 1 implying thatv is not sure strongly synchronizing according to T , a contradiction with the fact thatP has a universal sure strongly synchronizing language according to max T . By above arguments, we see that to decide the universality problem of sure strongly synchronizing language according to max T , one can check whether the graph underlying the PA has a unique bottom SCC consisting of a simple cycle, and whether the periodic supports of recurrent states in the Markov chain induced by the PA under the uniformly randomized word, are all singletons sets. This can be done in PTIME. A similar argument for the function sum T shows that to decide the universality problem of sure strongly synchronizing language, the sequences 0s1s2 ··· wheres 0 ={q init} and si = post(si−1,A) for alli∈N, can be computed and then one can check whethers i ⊆T for alln≤i≤n+m, which can be done in PSPACE.

138 We do not provide a matching lower bound for the universality problem of sure strongly synchronizing languages according to the function sum T .

Universality problems of almost-sure strongly synchronizing language. Given a PAP with the initial stateq init and the function max T , Lemma 6.12 proves that the almost-sure strongly synchronizing language ofP is universal if and only if (i) there is an absorbing end componentU, and (ii)P is almost-sure strongly synchronizing by the uniformly randomized wordw u, which provides the PSPACE membership of this problem. Lemma 6.12. The universality problem of almost-sure strongly synchronizing languages according to max T in PAs is in PSPACE.

Proof. Given a PAP=⟨Q,A,δ⟩ with the initial stateq init and the function max T , the almost-sure strongly synchronizing language is universal if and only if (i) there is a (then necessarily unique) absorbing end componentU, and

(ii)P is almost-sure strongly synchronizing by the uniformly randomized wordw u. One direction is straightforward. For the reverse direction, assume that both of the conditions (i) and (ii) are fulﬁlled in the PAP. We show that the almost-sure strongly synchronizing language (according to max T ) ofP is universal. LetU={q 0, q1, . . . , q|U|−1 }, for convenience also letq |U| =q 0. We claim that post(qi,A) ={q i+1} is singleton for all η 0≤i<|U|, which meansU consists of a simple cycle. Letϵ= (1+η) whereη is the smallest probability of taking a transition inP. Sincew u is almost-sure strongly synchronizing, there wu existsn 0 ∈N such that for alln≥n 0, there exists some stateq whereP n (q)>1−ϵ. Letn 1 ∈N be such that for alln>n 1, the total probability to be in statesQ\U is wu smaller thanϵ:P n (Q\U)<ϵ. Then, for allk> max(n 0, n1), there exists a unique state wu q∈U such thatP k (q)≥1−ϵ. Assume towards contradiction thatq has two successors q1, q2 ∈U withq 1 ≠ q 2, that is{q 1, q2}⊆ post(q,A). Then,

wu wu Pk+1(q1)≥P k (q)·δ(q, a)(q 1)> (1−ϵ)·η=ϵ

wu whereq 1 ∈ post(q,a). By a symmetric argument, we haveP k+1(q2)>ϵ, showing that wu ∥Pk+1∥<1−ϵ, a contradiction. This arguments proves that the absorbing end compo- nentU consists of a simple cycle. We have shown that the absorbing end componentU={q 0, q1, . . . , q|U|−1 } consists of a simple cycle. Towards contradiction with the universality of the almost-sure strongly synchronizing language ofP, assume that there exists an inﬁnite wordw that is not almost- max w w sure strongly synchronizing according to T . LetP 0 P1 ··· be the symbolic-run ofP wu wu overw; andP 0 P1 ··· be the symbolic-run over the uniformly randomized wordw u. SinceU is absorbing and consists of a simple cycle, there existsn∈N such that for w ′ alli>n the support Supp(P i ) contains more than one stateˆq∈U, say two states q,q ; otherwisew would be almost-sure strongly synchronizing. By assumption, the uniformly randomized wordw u is almost-sure strongly synchronizing. All states reachable byw are w wu also reachable byw u: Supp(Pi )⊆ Supp(P i ). Hence, for alli>n there are two statesq

139 ′ ′ wu andq such that{q, q }⊆ Supp(P i ) too. Since the probabilities assigned to these stateq ′ andq always loop through the sequenceq 0q1 ···q |U| , the probability mass could never accumulate in a singleton anymore, a contradiction with condition (ii) stating thatw u is almost-sure strongly synchronizing.

We have proved that, given a PAP with the initial stateq init and the function max T , the almost-sure strongly synchronizing language (according to max T ) ofP is universal if and only if (i) there is an absorbing end componentU, and (ii)P is almost-sure strongly synchronizing by the uniformly randomized wordw u. As we have shown in the proof of Lemma 6.12, these two conditions enforce a special shape to the absorbing end compo- nentU; the end componentU consists of a simple cycle, and for the symbolic-run of the PA over the uniformly randomized word, all supports can only contain one of the states ofU. One can observe that these conditions indeed imply that the periodic supports of recurrent states in the Markov chain induced by the PA under the uniformly randomized word, are all singletons sets. For the function sumT , a similar argument shows that the almost-sure strongly synchronizing language (according to sum T ) ofP is decidable in PSPACE. The requirement of having an absorbing end component is not necessary for the function sum T ; in fact, states in all end componentsU can contribute in accumulating the probability mass in the target setT . Each end componentU can be treated as an absorbing end component by removing all outgoing transitions, and the periodic supports of recurrent statesc 0, c1,···,c n can be computed in PSPACE. From those supports, a randomized wordw can be constructed such that it is a linear combinations from all pure words where the PA stays inU with some positive probability, once it has entered inU (by those pure words); and check whether P is almost-sure strongly synchronizing byw. After repeating these arguments for all end componentsU, the PA must be almost-sure strongly synchronizing by the uniformly randomized wordw u too. The above arguments provided the PSPACE membership of the universality problem of almost-sure strongly synchronizing languages for both function sum T and max T , a matching lower bound can be proved by the same reduction presented in Lemma 4.12 for eventually synchronizing languages. In the presented reduction, the constructed PA is almost-sure eventually synchronizing in the singleton{synch} if and only if it is almost-sure strongly synchronizing. From the above arguments and Lemma 6.12, it turns out that for all functionsf∈ {max T , sumT }, the universality problem of almost-sure strongly synchronizing language in PAs is PSPACE-complete.

140 Synchronization in Timed7 Automata

First sight. In this chapter, we introduce variants of synchronizing words for timed automata. Timed automata arefinite automata enriched with some clocks and the transitions arefired with respect to some time constraints. The behavior of a timed automaton thus depends on such quantitative constraints, and this is one of the challenges when considering synchronization. Moreover, the automaton must be brought into the same state no matter what is the initial state as it does when synchronizing afinite automaton, however the initial state here ranges among possibly infinite set of states. We focus on deciding the existence of different variants of synchronizing word for timed automata, proving PSPACE-completeness of the problems for deterministic timed automata, and proving undecidability for non-deterministic timed automata. The results presented in this chapter are summarized in Table 7.1.

Contents 7.1 Preliminaries ...... 142 7.1.1 Timed automata and timed words ...... 142 7.1.2 Problems in TAs ...... 144 7.2 Synchronization in TA ...... 145 7.3 Synchronization in deterministic TAs ...... 146 7.4 Synchronization in non-deterministic TAs ...... 153

141 Synchronization Location-synchronization Deterministic TAs PSPACE-complete PSPACE-complete Non-deterministic TAs undecidable

Table 7.1: Computational complexity of the synchronizing and location-synchronizing problem in TAs. 7.1 Preliminaries

Weﬁrst introduce labeled transitions systems with possibly inﬁnitely many states:

Definition 7.1 (Labeled transition systems).A labeled transition system over a (possibly infinite) alphabetΓ is a pair⟨Q, R⟩ whereQ is a set of states andR⊆Q×Γ×Q is a transition relation. The state spaceQ=L×X consists of afinite setL of locations and a possibly infinite setX of quantitative values.

Given a stateq=(ℓ, x), let loc(q) =ℓ be the location ofq, and fora∈Γ, let post+ (q,a) ={q ′ |(q, a, q ′)∈R}. ForP⊆Q, let loc(P)={loc(q)|q∈P} and post(P,a) = ′ + ′ q∈P post(q,a). For nonempty words w,w ∈Γ wherew=a·w , deﬁne inductively post(q,w) = post(post(q,a), w′).A run (or path) in a labeled transition system⟨Q, R⟩ ∗ overΓ is aﬁnite sequenceq 0q1 ···q n such that there exists a worda 0a1 ···a n−1 ∈Γ for which(q i, ai, qi+1)∈R for all0≤i

7.1.1 Timed automata and timed words

LetC={x 1, . . . , x|C| }beafinite set of clocks. A (clock) valuation is a mapping v:C→R ≥0 that assigns to each clock a non-negative real number. We denote by0 C (or0 when the set of clocks is clear from the context) the valuation that assigns0 to every clock. LetI be the set of intervals with integer or infinite endpoints. A guardg=(I 1,...,I|C| ) overC is a tuple of|C| intervalsI i ∈I. A valuationv satisfiesg, denotedv|=g, if v(xi)∈I i for all1≤i≤|C|. Fort∈R ≥0, we denote byv+t the valuation defined by (v+t)(x) =v(x) +t for allx∈C, and for a setr⊆C of clocks, we denote byv[r] the valuation such thatv[r](x) = 0 for allx∈r, andv[r](x) =v(x) for allx∈C\r.

Definition 7.2 (Timed automata).A timed automaton (TA) over afinite alphabetA is a tuple⟨L, C, E⟩ consisting of afinite setL of locations, afinite setC of clocks, and a setE⊆L×I |C| ×A×2 C ×L of edges.

g,a,r WhenE is clear from the context, we denote byℓ −−→ℓ ′ the edge(ℓ, g, a, r,ℓ ′)∈E, which represents a transition on the lettera from locationℓ toℓ ′ with the guardg and setr of clocks to reset.

142 x <1, a x≤2, a 1≤x≤2, a x≥1, a, x := 0 ℓ0 ℓ1 ℓ2 ℓ3 a, x := 0 x >2, a, x := 0 x <1∨x>2, a

Figure 7.1: A 1-letter TA with synchronizing wordd(3)·a 3 ·d(1)·a 3 whered is a special letter indicating quantitative delays.

The semantics of a TAA=⟨L, C, E⟩ is the labeled transition system!A"=⟨Q, R⟩ 1 ′ ′ over the alphabetΓ=R ≥0 ∪A whereQ=L×(C→R ≥0), and((ℓ,v),γ,(ℓ , v ))∈R if ′ ′ – eitherγ∈R ≥0, andℓ=ℓ andv =v+γ; – orγ∈A, and there is an edge(ℓ, g,γ, r,ℓ ′)∈E such thatv|=g andv ′ =v[r].

A timed word is a sequencew=a 0a1 ···a n witha i ∈A∪R ≥0 for all0≤i≤n. For a timed word, the length is the number of letters inA it contains, and the granularity is inﬁnite if the word involves non-rational delays, and it is the largest denominator if the timed word only involves rational delays. As an example consider the timed wordw= 3 d(3)·a ·d(1)·a whered(t) witht∈R ≥0 denotes the quantitative delays, and the length ofw is4.

Example 7.1. The TA drawn in Figure 7.1 has four locationsℓ 0,ℓ 1,ℓ 2,ℓ 3, one lettera and one clockx. In the locationℓ 0, if the lettera is read when the clock valuationx<1 is still less than1, the automaton moves toℓ 1; otherwise, the automaton stays inℓ 0 but resets the clockx. In the locationℓ 1, if the lettera is input when the clock valuationx≤2 is less than or equal to2, the automaton moves toℓ 2; otherwise it goes toℓ 0 and resets the clockx too. In the locationℓ 2, reading the lettera leads the automaton to end up inℓ 3 or move 1≤x≤2,a x<1∨x>2,a toℓ 1 depending on the guards:ℓ 2 −−−−−→ℓ 3 andℓ 2 −−−−−−→ℓ 1 where the disjunction in the guard of a transition is obtained by several interval guarded transitions. The locationℓ 3 is a sink state since no transitions leave it. Consider the timed wordw=d(3)·a 3 ·d(1)·a whered(t) witht∈R ≥0 denotes the quantitative delays. A valid run of the automaton from the state(ℓ 0, 0) to the state(ℓ 3, 1) overw is

d(3) a a a d(1) a (ℓ0, 0) −−→(ℓ 0, 3) −→(ℓ 0, 0) −→(ℓ 1, 0) −→(ℓ 2, 0) −−→(ℓ 2, 1) −→(ℓ 3, 1). ▹

The TAA is deterministic if for all states(ℓ, v)∈Q, for all edges(ℓ, g 1, a, r1,ℓ 1) and (ℓ, g2, b, r2,ℓ 2) inE, ifa=b, andv|=g 1 andv|=g 2, thenr 1 =r 2 andℓ 1 =ℓ 2; it is complete if for all(ℓ, v)∈Q and alla∈A, there exists an edge(ℓ, g, a, r,ℓ ′)∈E such thatv|=g. For instance, consider the TA in Example 7.1. Since for all locationsℓ and all valuations for the only clockx, there is only onea-transition the TA is deterministic.

1. We assume thatA∩R ≥0 =∅.

143 7.1.2 Problems in TAs The decidability of several classical problems in TAs are proved by the classical notion of region equivalence, see [AD94]. The region equivalence partitions the set of clock valuations intofinitely many classes called regions where two states in the same location and same region are time-abstract bisimilar. For a TA⟨L, C, E⟩ over thefinite alphabetA, letM be the maximal constant that appears in the guards in absolute value. The region equivalence ∼= on valuations is defined as follows. For all pairs of clock valuationsv andv ′, we definev ∼= v′ if and only if the three following conditions hold for all pairs of clocks x, y∈C: 1. the clockx in valuationv is greater thanM if and only if it is inv ′ v(x)>M⇔v ′(x)>M,

2. if the clockx in one of the valuations is less thanM, then the integer part of the clockx in both valuations is equal, and moreover the fractional part is zero only if it is zero in both valuationsv andv ′ ifv(x)≤M then(⌊v(x)⌋=⌊v ′(x)⌋) and(frac(v(x)) = 0⇔ frac(v ′(x)) = 0),

3. if both clocks x, y in one of the valuations are less thanM, then the value of clockx is more than clocky inv if and only if the same holds in the valuationv ′ ifv(x), v(y)≤M then(frac(v(x))> frac(v(y))⇔ frac(v ′(x))> frac(v ′(y))), where frac(v(x)) =v(x)−⌊v(x)⌋ denotes the fractional part of the clock valuation. The equivalence class of a valuationv for the relation ∼= is a region ofA. ByR we denote the set of regions where its size |R| isO(2 |C| ×|C|!×(2M +1) |C| ), i.e. exponential in the size of the TAA. The region automaton is then aﬁnite-state automaton obtained by quotienting the original TA with the region equivalence (where delay transitions are abstracted by a d-transition). We refer to [AD94] for more details on the region abstraction for TAs. Example 7.2. The TA depicted in Figure 7.2 has two locations p, q, one letter and one clockx. Thea-transitions in the locationp either stay there and reset the clockx; or if the clockx is less than1, thea-transition brings the automaton fromp toq. The stateq is a self-loop that the only transition is always available. The region automaton of the TA has eight states as it is shown in Figure 7.2. Thed-transitions abstract away the exact values of delays. For instance, from the state(p,0

144 a x≥1, a, x := 0 a d d d p p, x=0 p,0

x <1, a a a

q q, x=0 q,0

Figure 7.2: A TA and its region automaton (d is a special letter indicating delay transitions). The region automaton is synchronized by the worda·a·d·d·d, but the TA cannot be synchronized (because there is no way to reset the clockx when starting in the locationq).

Decision problem(s). The reachability problem in TAs asks, given a TA and two locationsℓ i andℓ f , whether there exists a run from(ℓ i,0) to some state(ℓ f , v) (with arbitrary valuationv).

The reachability problem is PSPACE-complete where the PSPACE membership follows from reachability in the region automata which has exponentially many states in the size of the input TA. The PSPACE-hardness is obtained by a reduction from halting problem for linear bounded automata (Turing machines that use only a polynomial number of cells on their tape).

Given a TA equipped with an initial locationℓ i and a setF of accepting locations, the timed language of the automaton is the set of timed words for which there exists some run from the initial state(ℓ i,0) reaching some accepting state(ℓ f , v) whereℓ f ∈F andv is an arbitrary clock valuation.

Decision problem(s). The universality problem in TAs asks, given a TA equipped with an initial state and a set of accepting states, whether the timed language contains all timed words.

The universality problem is known to be undecidable in (non-deterministic) TAs by encoding the computation of a Minsky Machine [AD94].

7.2 Synchronization in TA

We generalize theﬁnite synchronizing words in NFAs, presented in Deﬁnition 3.1 on page 38, to TAs.

145 Deﬁnition 7.3 (Synchronizing and location-synchronizing words for TAs). Given a complete TAA=⟨L, C, E⟩ over the alphabetA and with the semantic (the labeled transition system)!A"=⟨Q, R⟩, a timed wordw is synchronizing if post(Q, w) is a singleton, and it is location-synchronizing if loc(post(Q, w)) is a singleton.

Thus a synchronizing word can be read from every state and bring the TA to a single state, and a location-synchronizing word brings the TA to a (set of states where all agree on a) single location with possibly diﬀerent clock valuations. A TA is (location-)synchronizing if there exists some (location-)synchronizing word for it.

Decision problem(s). The (location-)synchronizing problem asks, given a TAA, whether it is (location-)synchronizing.

For instance, consider the TA described in Example 7.1. We have inﬁnitely many states to synchronize using the lettera and quantitative delaysd(t)(t∈R ≥0). One can 3 3 verify that the timed wordd(3)·a ·d(1)·a synchronizes the TA into the state(ℓ 3, 0). Note that a synchronizing word, in general, is location-synchronizing too. However, in this example, since thea-transition in the locationℓ 3 is always available and always resets the a,x:=0 clockx(ℓ 3 −−−→ℓ 3), concatenating one morea at the end of each location-synchronizing word gives a synchronizing word, and thus synchronization and location-synchronization coincide. In following, Example 7.3 illustrates a TA where the region equivalence is not sound toﬁnd a synchronizing word: the region automata is synchronizing while the TA has no synchronizing word. This unsoundness is due to the fact that region equivalence abstracts away the exact value of the clocks (as explained in Example 7.2), while synchronizing needs to keep track of them.

Example 7.3. Consider the TA described in Example 7.2. Since thea-transition in the locationq never reset the clockx, so two runs starting from two states(q, v 1) and(q, v 2) wherev 1(x)≠ v 2(x) are never able to end up in the same state. The automaton thus has no synchronizing word whereas the region automata is synchronizing by the worda·a·d·d·d. ▹ There are simple examples showing that corner point abstraction, another known abstraction in TAs [BBL08], is not sound toﬁnd synchronizing words in TAs too.

7.3 Synchronization in deterministic TAs

In this section, we prove that deciding whether there exists a synchronizing word for a given complete deterministic TA, is PSPACE-complete. To establish the membership in

146 PSPACE, wefirst prove the existence of a short witness (in the sequel, a timed word is short when its length and granularity are inO(2 |C| × |L| × |R|)). The built short witness starts with afinitely-synchronizing word, a word that brings the infinite set of states of the automaton to afinite set, and continues by synchronizing the states of thisfinite set pairwise. Example 7.4. Consider the TA described in Example 7.1 that is a complete automaton. To find a synchronizing word for the TA, our proposed algorithm consists in two phases:first reducing the (uncountably) infinite set of configurations into afinite set (with at most the number of locations in the TA) byfinding afinitely-synchronizing word, and then pairwise synchronizing the obtainedfinite set of states. The wordd(3)·a·a is afinitely synchronizing word that synchronizes the infinite set of states into afinite set: whatever the initial state, inputting the wordd(3)·a·a the automaton ends up in one of the states(ℓ 0, 0),(ℓ 1, 0) or (ℓ3, 0). On the other hand, sinceℓ 3 is an absorbing state (all transitions are self-loop onℓ 3), all synchronizing words must synchronize the automaton in a state(ℓ 3, x) with locationℓ 3 andx∈R ≥0. It then suffices to inputa·d(1)·a·a·a to end up in(ℓ 3, 0), whatever the initial state. Our algorithm thus returns the synchronizing wordd(3)·a 3 ·d(1)·a 3 for the TA, which always leads to the state(ℓ 3, 0). ▹ Wefix a complete deterministic TAA=⟨L, C, E⟩ with the maximal constantM appearing in the guards in absolute value. We begin with two folklore remarks on TAs. For all locationsℓ, we denote byL ℓ ={(ℓ, v)|v(x)>M for all clocksx∈C} the set of states with locationℓ and where all clocks are unbounded. Then the clock valuations inL ℓ form a region (andL ℓ is one of the states in the region automata ofA).

Remark 10. For all locationsℓ and for all timed wordsw, the set loc(post(L ℓ, w)) is a singleton and post(Lℓ, w) is included in a single region.

Proof. The proof is by an induction on the length of the timed wordsw. We reinforce the statement of the remark as follows. For all locationsℓ∈L, for all pairs of statesq 1, q2 ∈L ℓ and all timed wordsw, let us write post(q 1, w) ={(ℓ 1, u1)} and post(q2, w) ={(ℓ 2, u2)}. After inputting the timed wordw from both statesq 1 andq 2, the automaton – ends up in the same location:ℓ 1 =ℓ 2, and – for all clocksx∈C, either the clock in both valuations is unbounded

u1(x)>M⇔u 2(x)>M,

or the clocks have the same value:u 1(x) =u 2(x). The base of induction clearly holds: since after inputting the empty wordϵ we haveu 1(x)> M andu 2(x)>M for all clocksx∈C. Assuming that the statement holds for all timed wordsw, one can easily see that it still holds after a delay transition. In case of a letter- transition (such asa-transition wherea∈A), the clock values are preserved, except for the clocksx that are reset by taking the transition. Both those clocks then have value zero in both valuations (bothu 1(x) andu 2(x)).

147 !

Notice that Remark 10 is a special property of the stateL ℓ, and in general: elapsing the same delay from two region-equivalent valuations may lead to non-equivalent valuations. Remark 11 is technical and provides the length and granularity of timed words that are considered when solving reachability problem in TAs.

Remark 11. For all locationsℓ∈L, letr∈R whereL ℓ = (ℓ, r) in the region automaton ofA. For all locationsℓ ′ ∈L and regionsr ′ ∈R such that(ℓ ′, r′) is reachable from the state(ℓ, r) in the region automaton, there exists a short timed wordw of length at most |L| × |R| and two valuationsv∈r andv ′ ∈r ′ such that post((ℓ, v), w)={(ℓ ′, v′)}.

Proof. Letρ=(ℓ 0, r0)(ℓ1, r1)···(ℓ, n, r n) be a simple path in the region automaton ′ ′ ′ ′ from(ℓ, r) to(ℓ , r ) where(ℓ 0, r0) = (ℓ, r) and(ℓ n, rn) = (ℓ , r ). Since the pathρ is simple (there is no cycles), it has size less than the number of states in the region automaton, namelyn≤|L| × |R|. Letv∈r be a valuation in the regionr, then(ℓ, v)∈L ℓ. We denote the sequence of letters read along the runρ witha 0 ·a 1 ···a m wherea i ∈A for all 0≤i≤m, andm≤n. Considering the timestampst i (for all0≤i≤m) and letting t−1 = 0, we define the timed wordw=b 0 ·b 1 ···b 2m+1 as follows: letb 2i = (ti −t i−1) and letb 2i+1 =a i. The guards that have to be satisfied along the pathρ entail conditions of the formt n −t m ∼c, which defines a convex zone of possible timestamps. This zone has at most|ρ| integer vertices (which may not belong to the zone itself, but only to its closure). The center of these vertices belongs to the zone, and has the expected granularity, which proves the result. !

Lemma 7.1. All synchronizing deterministic TAs have a shortﬁnitely-synchronizing word.

Proof. LetA=⟨L, C, E⟩ be a complete deterministic TA with the maximal constantM appearing in the guards in absolute value. Assume that the TAA has a synchronizing word, we build a shortfinitely-synchronizing wordw f forA. The built word has a key property: for all clocksx∈C, irrespective of the starting state, the run overw f takes some transition resettingx. Wefirst argue that for all clocksx∈C, from all states with the clock valuationsv(x)≠ 0 not equal to zero, there exists a reachablex-resetting transition. Towards contradiction, assume that there exist some state(ℓ, v) and clockx such thatx will never be reset along any run from(ℓ, v). Runs starting from states with the same locationℓ but different clock valuations, say(ℓ, v ′) withv ′(x)≠ v(x), over a synchronizing wordw, may either (1) resetx, and thus thefinal values ofx on two runs from(ℓ, v) and(ℓ, v ′) are different, or (2) not resetx, so that the difference betweenv(x) andv ′(x) is preserved along the runs overw. Both cases give contradiction, and thus for all clocksx∈C, from all states withv(x)≠ 0, there exists a reachablex-resetting transition. Letℓ∈L be some location andx∈C be some clock. Let time wordw ℓ,x be a word built by applying the argument presented above to an arbitrary stateq ofL ℓ and the clockx.

148 By Remark 10, inputting the same timed word from any state ofL ℓ always leads to the same transition resettingx. Moreover, all such runs end up in the same region. Note that by Remark 11,w ℓ,x can be chosen to have length and granularity at most|L| × |R|. Below, we construct the shortfinitely synchronizing wordw f forA whereS is the infinite set of states to be (finitely) synchronized (meaning that post(S, wf ) must be a finite set). Repeat the following procedure untilS is synchronized to afinite set.

1. Choose a locationℓ∈ loc(S) such that there is an infinite setS ℓ ⊆S of states(ℓ, v) with that locationℓ and some clock valuationv, included inS. 2. For all clocksx∈C, iteratively, input a word that consists of a(M + 1)-time-unit delay followed by the wordw ℓ,x. The timed word ofM+1 delay brings the infinite setS ℓ to the unbounded regionL ℓ. Next, inputting the wordw ℓ,x makes the runs starting fromS ℓ end up in a single region where clockx has the same value for all runs (since it has been reset). The wordw ℓ = (d(M + 1)·w ℓ,x)x∈C synchronizes the infinite setS ℓ to a single state by resetting all clocks, one-by-one, and it also shrinksS.

3. Apply the wordw ℓ toS and update this set byS= post(S,w ℓ). Among all reached ′ locations, choose the next locationℓ ∈ loc(S) such that there is an inﬁnite setS ℓ ⊆S of states(ℓ, v) with that locationℓ and some clock valuationv, included inS. Repeat the second and third step untilS is synchronized to aﬁnite set.

Note that for all locationsℓ, the wordw ℓ has length at most|C|×|L| ×(|R| + 1) and granularity at most|L| × |R|. Thus theﬁnitely-synchronizing wordw f , obtained by 2 concatenating the successive wordsw ℓ, has length bounded by|C|×|L| ×(|R| + 1) and granularity at most|L| × |R|, so that it is short. By construction, the wordw f ﬁnitely- synchronizesA, which concludes our proof. !

From the proof of Lemma 7.1, we see that for all synchronizing TAs, there exists a ﬁnitely-synchronizing word which, in a sense, synchronizes the clock valuations. Precisely:

Corollary 7.1. For all synchronizing deterministic TAs, there exists a shortﬁnitely- synchronizing wordw f such that for all locationsℓ, this wordw f synchronizes the set {ℓ} ×(C→R ≥0) into a single state. Lemma 7.2 uses Corollary 7.1 to construct a short synchronizing word for a synchronizing TA. Short synchronizing words consist of aﬁnitely-synchronizing word followed by a pairwise synchronizing word (i.e., a word that iteratively synchronizes pairs of states).

Lemma 7.2. All synchronizing deterministic TAs have a short synchronizing word.

Proof. LetA=⟨L, C, E⟩ be a complete deterministic TA with the maximal constantM appearing in the guards in absolute value. By Corollary 7.1, we know that forA there exists a shortﬁnitely synchronizing wordw f that synchronizes the clock valuations. Thus, to synchronizeA it is suﬃcient (and necessary) to synchronize the setS= post(L×(C→

149 R≥0), wf ) obtained by shrinking the inﬁnite state spaceL×(C→R ≥0) after inputtingw f . We assume, w.l.o.g., that all clock valuations of states inS are in the unbounded region (since otherwise we can concatenate the timed word ofM+1 delay at the end ofw f ). SinceS hasﬁnite cardinality at mostn=|L|, we enumerate its elements and denote it by

Sn ={(ℓ 1, v1),(ℓ 2, v2),···,(ℓ n, vn)}. Consider the timed automatonA 2 obtained as the product of two copies ofA: this automaton has2|C| clocks and|L| 2 locations where we denote the clocks inA 2 by j (x )j∈{1,2},x∈C . To synchronizeS n to a single state, we repeat the following procedure for alli= n, n−1,···,1. Take a pair of states inS i, say(ℓ 1, v1) and(ℓ 2, v2). Consider 2 j the state(L, V) ofA deﬁned by these states whereL=(ℓ 1,ℓ 2) andV(x ) =v j(x) for j∈{1,2} andx∈C. SinceA has a synchronizing word, there is a run from(L, V) inA 2 to a symmetric state of the form(L ′,V ′) withL ′ = (ℓ′,ℓ ′) for someℓ ′ andV(x 1) =V(x 2) for allx∈C. Writew i for the corresponding word, and letS i−1 = post(Si, wi). Re- peat the same procedure fori−1. We see thatS i contains at mosti states, thus the wordw=w n ·w n−1 ···w 1 synchronizesS n to a single state. For alli, the wordw i is chosen 2 to have length at most2|C|×|L| ×(|R 2| + 1) and granularity at most|L| × |R 2|, where 2|C| |C|+1 |R2|≤ (4M + 1) × (2|C|)!≤ |R| , so thatw has length and granularity polynomial in|L| andM, and exponential in|C|. Thus,w f ·w is a synchronizing word forA, and it is short. !

A naive algorithm for deciding the existence of a synchronizing word would consist in non-deterministically picking a short timed word, and checking whether it is synchronizing. However, the latter cannot be done easily, because we have inﬁnitely many states to check, and the region automaton is not sound for this. Lemma 7.3. The synchronizing problem in deterministic TAs is in PSPACE.

Proof. To have a PSPACE algorithm to decide the synchronizing problem in TAs, we cannot simply guess a short timed word and check whether it is synchronizing since we have infinitely many states to check if the runs starting from those states end up in a single state after reading the short witness. LetA=⟨L, C, E⟩ be a complete deterministic TA with the maximal constantM appearing in the guards in absolute value. We thusfirst consider the setS 0 ={(ℓ,0)|ℓ∈ L} and compute the set of successors reached fromS 0 after reading afinitely-synchronizing wordw f (built in the proof of Lemma 7.1); that is, we simply compute the set post(S0, wf ). This can be achieved using polynomial space, sinceS 0 contains polynomially many states andw f can be guessed on-the-fly. The important point here is that sincew f begins with a delay ofM+1 time unit, the set post(S 0, wf ) is equal to the set post(Q, wf ) where C Q=L×R ≥0 is the state space of the semantic!A" of the TAA. The set post(S0, wf ) contains at most|L| states, and this set post(S 0, wf ) can now be synchronized pairwise. This pairwise-synchronization phase can be achieved by computing

150 ′ TAA ℓ0

A∪{#} # # ℓ ℓ ℓi ℓf ⇒ # ℓi ℓf A∪{#} ℓ′ ℓ′ TAA TAA

Figure 7.3: The reduction sketch to show PSPACE-hardness of the (location-)synchronizing problem in TAs. the product automatonA 2 and solving reachability problems in that automaton (similar to the proof of Lemma 7.2). This algorithm runs in polynomial space, and successfully terminates if, and only if,A has a synchronizing word. !

Using similar arguments, we obtain the following result stating the PSPACE membership of the location-synchronizing problem in TAs.

Lemma 7.4. The location-synchronizing problem in deterministic TAs is in PSPACE.

Proof. LetA=⟨L, C, E⟩ be a complete deterministic TA with the maximal constantM appearing in the guards in absolute value. To location-synchronize the TAA, we propose the following. – The location-synchronizing wordw starts with(M + 1)-time-unit delay, in such a way that after this delays all the clocksx∈C end up with values above the maximal constantM. – We see that for all locationsℓ, all timed wordsw and all pairs of region-equivalent valuationsv andv ′, the states in post((ℓ, v), w) and post((ℓ, v ′), w) still have the same location and region-equivalent valuations. Hence location-synchronization can be achieved by just considering the set{(ℓ, v M+1 )|ℓ∈L}, where the valuationv M+1 maps all clocks toM+1. These states can be location-synchronized pairwise. There is one important point to take into account: two statesq andq ′ that have been location-synchronized might later de-synchronize, since they do not end up in the same region. However, this problem is easily avoided by lettingM+1 time unit elapse after location-synchronizing each pair of states. Using the same arguments as for synchronizing word the above procedure runs in polynomial space. The membership of location- synchronizing problem in PSPACE follows. !

151 Lemma 7.5. The (location-)synchronizing problem in deterministic TAs is PSPACE-hard.

Proof. The proof is by a reduction from the reachability problem in TA, which is known to be PSPACE-complete [AD94]. Given a deterministic TAA=⟨L, C, E⟩ (without loss of generality, we assume thatA is complete) defined over afinite alphabetA and equipped with two locationsℓ i andℓ f , we construct another TAA ′ such that there is some clock valuationv such that there exists ′ a run inA starting from(ℓ i,0) reaching the state(ℓ f , v) if and only if the automatonA has a (location-)synchronizing word. ′ ′ ′ ′ The TAA =⟨L ,C,E ⟩ is a copy ofA with one new locationℓ 0:L =L∪{ℓ 0}. The automatonA ′ is defined on the alphabetA ′ =A∪{#} where# ̸∈A is a new letter. See ′ Figure 7.3. All transitions inA are copied toA except the transitions in the locationℓ f . The following transitions are added, that are always available and reset all the clocks, – ana-transition inℓ f for all lettersa∈A∪{#} which is a self-loop:

(ℓf , true, a, C,ℓ f ) for alla∈A.

– ana-transition from the new locationℓ 0 toℓ i for all lettersa∈A∪{#}:

(ℓ0, true, a, C,ℓ i) for alla∈A.

– a#-transition inℓ i which is a self-loop:(ℓ i, true,#,C,ℓ i). ′ – from all locationsℓ∈L \{ℓ0,ℓ i,ℓ f } exceptℓ 0,ℓ i andℓ f , a#-transition toℓ 0:

(ℓ, true,#,C,ℓ 0). The resulting automatonA ′ is deterministic and complete. We establish the correctness of the reduction as follows. First assume thatA ′ has a synchronizing wordw. We prove that there exists some clock valuationv such thatA has a ′ run from(ℓ i,0) to(ℓ f , v). On reading the synchronizing wordw by the automatonA , the ﬁnal location necessarily is(ℓ f , v) wherev is some clock valuation, asℓ f has no outgoing transition. Thus, the run starting in(ℓ i,0) over this wordw also reaches(ℓ f , v); however this run might possibly take some#-transitions, which are not valid transitions inA. Consider the shortest subrun going from(ℓ i,0) to(ℓ f , v). That run does not contain any #-transition, since any such transition either corresponds to the self-loop onℓ f orℓ i, or leads toℓ 0 and will be followed by another visit to(ℓ i,0), both options contradicting the fact that we have considered the shortest subrun from(ℓ i,0) to(ℓ f , v). Second, assume that there exists a clock valuationv such that there is a run inA from(ℓ i,0) to(ℓ f , v) over some timed wordw. Then the word#·#·w·# necessarily ′ synchronizes the whole state space of the TAA into the state(ℓ f ,0): indeed, ′ – since there are only self-loops in the locationℓ f , we have post((ℓf , v ), w)=(ℓf ,0) for all clock valuationsv ′. ′ ′ – moreover for all clock valuationsv , from(ℓ i, v ) reading two times# brings the automaton into(ℓ i,0), and then following the wordw the state(ℓ f , v) is reached. The last# resets all the clocks and the automaton end up in(ℓ f ,0).

152 ′ ′ – for all clock valuationsv , from(ℓ 0, v ) the automaton end up in(ℓ i,0) by taking the#-transitions. Next, inputting the wordw·# synchronizes the automaton into(ℓ f ,0) as discussed in the former case. ′ – for all clock valuationsv , from(ℓ, v) whereℓ/∈{ℓ 0,ℓ i,ℓ f }, theﬁrst#-transition leads to(ℓ 0,0), and the second one goes to(ℓ i,0). Inputting the wordw·# afterward synchronizes the automaton into(ℓ f ,0) as discussed in the former case. The proof is complete and the PSPACE-hardness follows. Note that the same reduction is correct to establish the result for the location-synchronizing problem: in fact since all transitions inℓ f (the only possible location to synchronize) always reset all clocks. Therefore,A ′ is synchronizing if and only if it is location-synchronizing. !

From previous Lemmas, the following Theorem 7.1 is obtained.

Theorem 7.1. The synchronizing and location-synchronizing problems in deterministic TAs are PSPACE-complete.

7.4 Synchronization in non-deterministic TAs

We now show the undecidability of the synchronizing problem for non- deterministic TAs. The proof is by a reduction from the non-universality problem of timed language for non-deterministic TAs, which is known to be undecidable [AD94].

Theorem 7.2. The (location-)synchronizing problem in non-deterministic TAs is undecidable.

Proof. LetA=⟨L, C, E⟩ be a non-deterministic TA over an alphabetA equipped with an initial locationℓ i and a setF of accepting locations (w.l.o.g. we assume thatA is complete). FromA, we construct another TAA ′ overA ′ such that the language ofA is not universal if and only ifA ′ has a location-synchronizing word. The construction of the TAA ′ =⟨L ′,C,E′⟩ deﬁned over a new alphabetA ′ is as follows (See Figure 7.4). There are two new statess andd and the alphabet is augmented with two new letters# and⋆. Formally,L ′ =L∪{d, s} (assuming d, s /∈L) andA ′ =A∪{#,⋆}. All transitions inA are copied toA ′ and the following new transitions are added. – Locations is a sink location, carrying a self-loop for all lettersa∈A∪{#,⋆}. Locationd is a departure location: it also carries a self-loop for all letters, except for⋆, which leads toℓ i. All those transitions reset all the clocks. – From all locationsℓ∈L, there is a⋆-transition toℓ i along which all the clocks are reset. From the states not inF, there is a#-transition tos along which all clocks are reset. From the states inF, the#-transition goes tod resetting all clocks.

153 ′ TAA s A∪{#,⋆} # ℓ ⋆ ℓ ℓi ℓf ⋆ ℓi ℓf ⇒ ⋆ # # TAA TAA ⋆ d A∪{#}

Figure 7.4: The reduction sketch to show undecidability of the (location-)synchronizing problem in TAs where in this example:{ℓ i,ℓ f }⊆F.

To establish the correctness of the reduction, weﬁrst assume that the timed language ofA is not universal. So there exists a wordw that is not accepted byA. Then all runs ′ ofA overw starting in(ℓ i,0) end in copies of non-accepting states. Hence inA , the word⋆·w·# reaches the state(s,0), whatever the starting state. It shows thatA ′ has a synchronizing word. Second, assume thatA ′ has a synchronizing word. Letw be one of the shortest synchronizing words (in terms of the number of transitions). Ass has no outgoing transitions, A′ can only be synchronized ins. Since enterings is only possible by reading the letter#, it must be the case thatw has at least one occurrence of#. Similarly, the states with the locationd are also able to synchronize intos, it must be the case thatw contains at least one occurrence of⋆ which is followed by one occurrence of#. Letw 1 be the subword of w between the last⋆ that is followed by an occurrence of#, up to theﬁrst subsequent occurrence of#. Sow 1 contains neither# nor⋆, andw can be written asw 0 ·⋆·w 1 ·#·w 2, wherew 2 contains no⋆ (otherwisew would not be one of the shortest synchronizing word). We show that the wordw 1 is not accepting byA (and thus, the timed language ofA is not universal). Towards a contradiction, assume thatw 1 is accepted byA. It cannot be the case thatw 0 is synchronizing, asw was chosen to be one of the shortest such words. Hence there must be two states(ℓ, v) and(ℓ ′, v′) whereℓ ′ ≠ s, such that there is a run ′ ′ ′ ′ from(ℓ, v) to(ℓ , v ) over the wordw 0. Reading⋆ from(ℓ , v ) leads to(ℓ i,0), from which ′′ ′′ ′′ ′′ ′′ there is a run over the wordw 1 that goes to a state(ℓ , v ) withℓ ∈F . From(ℓ , v ), reading# leads to(d,0). Sincew 2 contains no⋆, there is no path from(d,0) to locations byw 2. This means that we have found a state(ℓ, v) from which readingw does not lead ′ tos, contradicting the fact thatw is synchronizing inA . Hencew 1 is not accepted byA andA is thus not synchronizing. The same reduction is used to show undecidability of the location-synchronizing problem; note that all transitions going tos (the only possible location to synchronize) always

154 reset all clocks. Therefore, the TAA ′ is synchronizing if and only if it is location synchronizing. !

155 156 Synchronization in Weighted Au- 8tomata

First sight. In this chapter, we introduce variants of synchronizing words for weighted automata. Weighted automata arefinite automata where transitions are augmented with some weights that are mostly considered as the resource (or energy) consumptions. We define safe synchronization for weighted automata where the aim is to synchronize the set of safe states while forbidding the automaton to visit states outside the safety- set during synchronization. One of the challenges to synchronize weighted automata is having a possibly infinite set of states to synchronize, as it is for timed automata. We focus on deciding the existence of different variants of synchronizing words for weighted automata (with safety conditions). For deterministic weighted automata, the synchronizing problem is proven PSPACE-complete under energy constraints, and in 3-EXPSPACE under general safety constraints. The results presented in this chapter are summarized in Table 8.1.

Contents 8.1 Preliminaries ...... 158 8.1.1 Weighted automata ...... 158 8.1.2 Minsky machine and vector addition systems with states . . . 159 8.2 Synchronization in WA ...... 160 8.3 Location-synchronization in deterministic WAs ...... 162 8.3.1 Location-synchronization under lower-bounded safety condition 162 8.3.2 Location-synchronization under general safety condition . . . 169

157 Synchronization Location-synchronization No cond. Safety cond. No cond. Safety cond. 3-EXPSPACE Deterministic Trivial PSPACE-complete NLOGSPACE-complete energy condition: (always false) PSPACE-complete

Non-deterministic Trivial PSPACE-complete PSPACE-complete ? (always false)

Table 8.1: Computational complexity of the synchronizing and location-synchronizing problem in WAs. 8.1 Preliminaries

In this section, we provide the deﬁnitions of weighted automata, Minsky machine and vector addition systems with states.

8.1.1 Weighted automata We present the definition of weighted automata as an instance of a labeled transition system with possibly infinite states. Let usfirst recall the definition of labeled transition systems, which we have provided in Definition 7.1. A labeled transition system over an alphabetΓ is a pair⟨Q, R⟩ whereQ is a set of states andR⊆Q×Γ×Q is a transition relation. The state spaceQ=L×X consists of afinite setL of locations and a possibly infinite setX of quantitative values. Given a stateq=(ℓ, x), let loc(q) =ℓ be the location ofq, and fora∈Γ, let post(q,a)+ ={q ′ | ′ (q, a, q )∈R}. ForP⊆Q, let loc(P)={loc(q)|q∈P} and post(P,a) = q∈P post(q,a). For nonempty wordsw∈Γ +, define inductively post(q, aw) = post(post(q,a), w).A run (or path) in a labeled transition system⟨Q, R⟩ overΓ is afinite sequenceq 0q1 ···q n such ∗ that there exists a worda 0a1 ···a n−1 ∈Γ for which(q i, ai, qi+1)∈R for all0≤i

Definition 8.1 (Weighted automata).A weighted automaton (WA) over afinite al- phabetA is a tupleA=⟨L, E⟩ consisting of afinite setL of locations, and a set E⊆L×A×Z×L of edges.

a z WhenE is clear from the context, we denote byℓ −→: ℓ ′ the edge(ℓ, a, z,ℓ ′)∈E, which represents a transition on lettera from locationℓ toℓ ′ with weightz. We view the weights as the resource (or energy) consumption of the system. The semantics of a WAA=⟨L, E⟩ is the labeled transition system!A"=⟨Q, R⟩ on the alphabetA whereQ⊆L×Z and((ℓ,e), a,(ℓ ′, e′))∈R if(ℓ, a, e ′ − e,ℓ ′)∈E. In a state(ℓ, e), we calle the energy level. The WAA is deterministic if for all edges(ℓ, a, z 1,ℓ 1),

158 d:−2 a, b, c, d:0 ℓ0 ℓ1 a, b:0 b:1 b:−10 c:0

c:−10, a, d:0 ℓ2 ℓ3 a:1, c, d:0

Figure 8.1: A complete deterministic WA with the synchronizing worda 10 ·b·(c·b) 2 ·d under the energy safety condition.

(ℓ, b, z2,ℓ 2)∈E, ifa=b, thenz 1 =z 2 andℓ 1 =ℓ 2; it is complete if for allℓ∈L and all a∈A, there exists an edge(ℓ, a, z,ℓ ′)∈E.

Example 8.1. The WA drawn in Figure 8.1 has four locationsℓ 0,ℓ 1,ℓ 2,ℓ 3 and four letters a, b, c, d. The stateℓ 0 is an absorbing state where all transitions have weight0. In the stateℓ 1, thed-transition brings the automaton to the absorbing stateℓ 0 with weight−2, while thec-transition is redirected toℓ 2. Thea andb-transitions are self-loops inℓ 1. The b:+1 b-transition inℓ 2 goes toℓ 1:ℓ 2 −−→ℓ 1; other transitions inℓ 2 are however self-loops with weight0 except the self-loop of thec-transition that has weight−10. All transitions inℓ 3 are self-loops with weights0 or1; except theb-transition that leaves this state and goes toℓ 1 by decreasing the energy level by−10. One can verify that the WA in Figure 8.1 is complete and deterministic. Thus, for all wordsw there is only one possible run of the automaton overw. Starting in the locationℓ 3 with the energy level1, the run of the WA over the wordw=a 10 ·b·(c·b) 2 ·d is

a:+1 a:+1 a:+1 b:−10 c:0 (ℓ3, 1) −−→(ℓ 3, 2) −−→··· −−→(ℓ 3, 11) −−−→(ℓ 1, 1) −→(ℓ 2, 1)

b:+1 c:0 b:+1 d:−2 −−→(ℓ 1, 2) −→(ℓ 2, 2) −−→(ℓ 1, 3) −−→(ℓ 0, 1). ▹

8.1.2 Minsky machine and vector addition systems with states We recall the deﬁnitions of Minsky machine and vector addition systems, plus two decision problems that are used in this chapter.

Minsky machine

A Minsky machine has two non-negative countersc 0 andc 1, and it consists of a sequence of numbered instructions1: inst 1; 2 : inst2;...n: inst n; where instn = halt and other

159 instructions insti are an increment or a guarded decrement:

(increment)i:c j :=c j + 1; gotok; ′ (guarded decrement)i: ifc j = 0 gotok else(c j :=c j − 1; gotok );

where0

Decision problem(s). The halting problem asks, given a Minsky machine, whether it halts.

It is well-known that the halting problem for (two-counter) Minsky machines is undecidable [Min67].

Vector addition systems with states A Vector addition system with states (VASS) is afinite-state machine VASS=⟨Q, T⟩ where the transitions carry vectors of integers of somefixed dimensiond. A configuration of a VASS is(s, v) wheres∈Q is a state andv is ad-dimension vector of non-negative v′ integers. A transitiont:s −→s ′ can be taken from the configuration(s, v) to(s ′, v+v ′) if v+v ′ is bigger than the0-vector (the vector with0 for alld dimensions).

Decision problem(s). The coverability problem asks, given a VASS with an initial configuration(s init, vinit) andfinal configuration(s end, vend), whether starting from(s init, vinit), a configuration(s, v) withs=s end andv≥v end is reachable.

If the answer to coverability problem is positive and in particular such conﬁgura- tion(s, v) exists, we say that(s end, vend) is covered by(s init, vinit). It is known that the coverability problem in VASSs is EXPSPACE-complete [Lip76, Rac78].

8.2 Synchronization in WA

We generalize theﬁnite synchronizing words in NFAs, presented in Deﬁnition 3.1 on page 38, to WAs.

Deﬁnition 8.2 (Synchronizing and location-synchronizing words for WAs). Given a complete WAA=⟨L, E⟩ over the alphabetA and with the semantic (the labeled transition system)!A"=⟨Q, R⟩, a wordw∈A + is synchronizing in the labeled transition system⟨Q, R⟩ if post(Q, w) is a singleton, and it is location-synchronizing if loc(post(Q, w)) is a singleton.

160 A synchronizing word can be read from every state and bring the WA to a single state. As weights are merely accumulated in WA, it is impossible to synchronize to a single state: consider two states(ℓ, e 1) and(ℓ, e 2) involving the same locationℓ but different initial energiese 2 > e1. At least two of the runs starting from these two states go through the states involving the same location and the energy levels preserving the differencee 2 −e 1 (for deterministic WAs there are only two runs, but for non-deterministic WAs there might be more than two runs starting from those states). So two states(ℓ, e 1) and(ℓ, e 2) can never be synchronized, and thus WAs trivially never have a synchronizing word. Instead we define location-synchronizing words; inputting such words result in the WA reaching states that agree only on the location. A location-synchronizing word brings the WA to a (set of states where all have a) single location with possibly different energy levels. In this setting, the location-synchronization in WAs is equivalent to synchronization offinite-state automata (i.e., weights play no role). Consequently, deciding whether, a given WA, has a location-synchronizing word is NLOGSPACE-complete if WA is deterministic, and is PSPACE-complete if WA is non- deterministic. A more realistic extension is to consider safe synchronization where we add a safety condition insisting that during synchronization the accumulated weight (energy) must be safe, e.g. a non-negative safety condition that requires the system to never run out of power while synchronizing. Considering the safety condition is what distinguishes our setting from the one presented in [FV13]; moreover, in that work WAs are restricted to have only non-negative weights on transitions. GivenU⊆Q, a wordw is synchronizing (resp., location-synchronizing) in⟨Q, R⟩ with safety conditionU if – post(U, w) is a singleton (resp., loc(post(U, w)) is a singleton), and – post(U, v)⊆U for all prefixesv ofw. The safety conditionU requires that the states inQ\U are never visited while reading the synchronizing word. LetI be the set of intervals with integer or infinite endpoints. We consider safety conditions of the form Safe:L→I, therefore the safe set isU={(ℓ, x)∈ Q|x∈ Safe(ℓ)}. We denote an interval[y, z] byy≤e≤z, an interval[z,+∞) bye≥z, etc. wheree is an energy variable. A WAA is (location-)synchronizing under the safety condition if there exists some (location-)synchronizing word under the safety condition for it.

Decision problem(s). The (location-)synchronizing problem with a safety condition asks, given a WAA and a safety condition Safe, whetherA is (location-)synchronizing under Safe.

Example 8.2. Consider the WAA described in Example 8.1, and let the safety condition Safe be non-negativee≥0 for all four locations. We sometimes call such safety conditions an energy condition.

161 In order to location-synchronize the automaton, we have to deal with infinitely many states(ℓ i, e) wheree∈R ≥0. The only way to location-synchronize some state(ℓ 3, e) with ′ states(ℓ, e ) involving other locationsℓ≠ ℓ 3 is to inputb. However, ifb is provided initially, this will drop the energy level by−10 violating the non-negative safety condition for(ℓ 3, 0). Fortunately, the lettera recharges the energy level atℓ 3 and has no negative effect at other 10 locations. After readinga ·b, all states are location-synchronized inℓ 0 andℓ 1 with energy at least0. Next, ad-transition can location-synchronize states involvingℓ 0 andℓ 1, but it drops the energy level atℓ 1 by−2. Again, we try tofind a word that recharges the energy atℓ 1. Supplyingc·b twice makes ad-transition safe to be taken to location-synchronize 10 2 safe states involvingℓ 0 andℓ 1. So, the worda ·b·(c·b) ·d location-synchronizes the automaton with non-negative safety condition. ▹ As mentioned, it is impossible to synchronize WAs in the absence of a safety condition. In the presence of safety conditions, synchronization is also most-often impossible, for the same reason: the only exception is when safety condition is punctual. A punctual safety condition assigns at most one safe energy level to each location, thus there would not be two safe states(ℓ, e 1) and(ℓ, e 2) with the same locationℓ but different energy levelse 1 ≠ e 2. The synchronization with a punctual safety condition in WAs is equivalent to synchronizing partial (not-complete)finite-state automata, which is PSPACE-complete [Mar10]. We thus focus on location-synchronizing problem with safety conditions; we establish complexity bounds for this problem for deterministic WAs. The location-synchronizing problem with safety conditions for non-deterministic WAs is left open, however we provide an example to explain why the used techniques for deterministic WA cannot be directly applied to the non-deterministic WAs (See Example 8.4).

8.3 Location-synchronization in deterministic WAs

Weﬁrst solve the location-synchronizing problem for deterministic WAs with lower- bounded safety condition, which are safety conditions that assign conditions of the forme≥ n withn∈Z, to all locations. We provide tight complexity bounds for the location- synchronizing problem for deterministic WAs with lower-bounded safety condition. Next, we study this problem with general safety condition (with no constraints on the condition).

8.3.1 Location-synchronization under lower-bounded safety condition In this subsection we assume that the safety condition is lower bounded where all the locations have some condition of the forme≥n withn∈Z. This is equivalent to having only safety conditions of the forme≥0: it suﬃces to add−n to the weight of all incoming transitions and to add+n to the weight of outgoing transitions. As an instance, consider b:−10 the WAA described in Example 8.1 and the transitionℓ 3 −−−→ℓ 1. Let the safety constraint

162 forℓ 1 bee≥3, and forℓ 3 bee≥−7. We change the weight of the transition to−20 when considering energy condition for both locations. Below, we thus study the location-synchronizing problem with safety conditions of the forme≥0, which we call non-negative safety conditions or energy condition. Lemma 8.1. The location-synchronizing problem under non-negative safety condition in WAs, is in PSPACE.

Proof. LetA=⟨L, E⟩ be a complete deterministic WA over the alphabetA whereZ is the maximum value appearing as weight in transitions, in absolute. Runs starting from two states(ℓ, e 1) and(ℓ, e 2) with same locationℓ but two different energy levelse 2 > e1, always go through the states involving the same locations and the energy levels preserving the differencee 2 −e 1. Therefore, to decide whetherA is location-synchronizing under non-negative safety condition, it suffices to check if there is a word that synchronizes all locations with the initial energy0, into a single location. We show that deciding whether such wordw exists is in PSPACE by providing an upper bound for the length ofw. Below, we assume thatA has a location-synchronizing word. For all subsetsS⊆L with cardinalitym>2, there is a word that synchronizesS into some strictly smaller set. To characterize the properties of such words, we consider the weighted digraphG m where m=|S|. The digraphG m is induced by the product betweenm copies ofA, where all vertices with at least two identical locations, are replaced with a new vertex synch. All ingoing transitions to some vertice(ℓ 1,...,ℓ m) with at least two identical locations are redirected to synch. There is only a self-loop transition in synch. Formally,G m =⟨V m,Em⟩ is a weighted digraph where the set of vertices

m Vm ={(ℓ 1,ℓ 2,...,ℓ m)∈L |ℓ i ≠ ℓ j ↔i≠ j}∪{synch}

includes allm-tuples of distinct locations and another vertex synch. – For all pairs of vertices x, y∈V m, there is an edge fromx=(ℓ 1,...,ℓ m) toy= ′ ′ (ℓ1,...,ℓ m) with weight⟨z 1, . . . , zm⟩ if there exists some lettera such that for all a:zi ′ 0

163 i i family of wordsw 0 ·(w 1) (i∈N) such that inputting the wordw 0 ·(w 1) accumulates energyi for the run starting in some locationℓ∈S, while having non-negative eﬀects along the runs starting from the other locations inS. As an example consider the WA described in Examples 8.1 and 8.2. Since in the i digraphG 2, there is no safe path from(ℓ 0,ℓ 2) to synch, there is a family of words(b·c) such that each iteration ofb·c increase the energy level inℓ 2 by1.

Claim1. For all1

Proof of Claim 1. SinceA has a location-synchronizing word, then from all vertices of the digraphG m, there must be a safe path to synch. Take a vertexx∈V m and assume that all simple paths fromx to synch are unsafe. WriteG for the digraph obtained fromG m by removing all negative edges. Thus, there is no path fromx to synch inG. Consider one of the bottom SCCs reached fromx inG. Since there is no path from the bottom SCC to synch inG, we see that one of the edges in this bottom SCC must be positive. Otherwise, if all edges in this bottom SCC are zero-eﬀect then for all verticesy of the bottom SCC, there is no way to synchronize them-locations ofy with initial energy0. Thus the statement of claim holds, and the proof of Claim1 is complete.

The next claim states thatA has a location-synchronizing word if it has a short one, 2 of length at mostZ |L| × |L|3+|L| .

Claim2. For the synchronizing WAA, there exists a short location-synchronizing word.

Proof of Claim 2. The proof is by an induction: we prove that for all2≤k≤|L| 2 k 3+k and all subsetsS of locations with|S|=k, there is a wordw S of length(k)≤Z |L| that location-synchronizes (under non-negative safety condition) the subsetS into a single location.

Base case. We prove that for all subsetsS of locations with|S|=2, the length of the 2 6 wordw S is at most4Z |L| . LetS={ℓ 1,ℓ 2} and consider the digraphG 2. If there is a safe simple path from(ℓ 1,ℓ 2) to synch, the base of induction trivially holds. Otherwise by Claim1, for one of the locations inS, sayℓ 1, for alli>0 there exists ani-recharging i wordw 0 ·(w 1) . Recall that ani-recharging word that recharges energy inℓ 1 is a word such that inputting this word keeps the energy levels non-negative along the runs starting from ′ the states inS; however it accumulates energyi along the run starting fromℓ 1. Letℓ 1 ′ andℓ 2 be the locations reached fromℓ 1 andℓ 2, accordingly, after inputting thei-recharging word withi=Z(2|L| 2 +Z|L| 4). A slightly diﬀerent argument from the one used to prove j ′ Claim1 gives us another wordw 2 ·(w 3) that recharges energy in run starting inℓ 2 at least

164 to to an arbitrary valuej by not considering the negative eﬀect it may cause on the other run. ′′ ′′ ′ ′ Letℓ 1 andℓ 2 be the locations reached fromℓ 1 andℓ 2, accordingly, after inputting the j 2 ′′ ′′ wordw 2 ·(w 3) withj=Z|L| . Letw 4 be a shortest synchronizing word forℓ 1 andℓ 2 i j inA by interpreting it as a DFA. We argue that the wordw=w 0 ·(w 1) ·w 2 ·(w 3) ·w 4 i is a location-synchronizing word for the subsetS. By reading the wordw 0 ·(w 1) , two ′ ′ states(ℓ 1, 0) and(ℓ 2, 0) go to(ℓ 1, e1 +i) and(ℓ 2, e2) wheree 1, e2 ≥0. Since there is a high ′ j energy atℓ 1, it can tolerate the negative eﬀect caused after reading the wordw 2 ·(w 3) . j ′ ′ ′′ Next, readingw 2 ·(w 3) leads two states(ℓ 1, e1 +i) and(ℓ 2, e2) to the states(ℓ 1, e3 +j) ′′ ′′ ′′ and(ℓ 2, e4 +j) wheree 3, e4 ≥0. Both states(ℓ 1, e3 +j) and(ℓ 2, e4 +j) can tolerate the wordw 4, and get synchronized while maintaining the energy level positive meaning that w is a location-synchronizing word forS. Hence, length(2)≤4Z 2 |L|6 and the base of the induction holds.

Inductive step. We prove the inductive step forn. Assume that for allk0 there i exists an i-recharging wordw 0 ·(w 1) . A location-synchronizing word can start with a [Z· length(n− 1)]-recharging word to accumulate at leastZ· length(n− 1) energy in the ′ run starting fromℓ n. For all0

length(n)≤|L| n [2 +Z· length(n− 1)] + length(2).

2 So, length(n)≤Z n |L|3+n and the induction holds. ThusA has a location-synchronizing word with length at most length(|L|). This concludes the proof of Claim2.

Claim2 proves thatA has a location-synchronizing word if it has one of length at 2 mostZ |L| × |L|3+|L| . Since this value can be stored in polynomial space, an (N)PSPACE algorithm can decide whether the given WAA is location-synchronizing. !

We use an intuitive reduction from synchronizing problem for partialﬁnite automaton to establish the matching lower bound.

165 Lemma 8.2. The location-synchronizing problem under non-negative safety condition in WAs, is PSPACE-hard.

Proof. To show PSPACE-hardness, we use a reduction from synchronizing word problem for deterministicfinite automata with partially defined transition function that is PSPACE- complete [Mar10]. From a partialfinite state automatonN=⟨Q,A,∆⟩, we construct a WAA over the alphabetA such thatN has a synchronizing word if and only ifA has a location- synchronizing word under an energy condition. The construction ofA=⟨L, E⟩ is as follows. – All defined transitions ofN are augmented with the weight0 inA: for all statesq∈Q a and lettersa∈A, letq −→:0 q ′ if∆(q, a) =q ′ in the automatonN. – To completeA, all non-defined transitions are added as self-loops with weight−1: a for all statesq∈Q and lettersa∈A, letq −−→:−1 q if thea-transition inq is not defined inN. Since the safety condition is non-negative in all locations, none of the transitions with weight−1 are allowed to be used along synchronization inA. SoN has a synchronizing word if and only ifA has a location-synchronizing word. !

From previous Lemmas and arguments, Theorem 8.1 follows.

Theorem 8.1. The location-synchronizing problem under lower-bounded safety conditions in WAs, is PSPACE-complete.

We generalize the location-synchronizing problem to location-synchronization from a subset, where the aim is to synchronize a given subset of locations. This variant is used to decide location-synchronization under general safety condition. Given a subsetS⊆L of locations, we prove Lemma 8.3 and Lemma 8.4 using reductions to and from the coverability problem in vector-addition systems.

Lemma 8.3. The location-synchronizing problem from a subsetS of locations under non- negative safety conditions in WAs, is decidable in 2-EXPSPACE.

Proof. We present a construction to reduce location-synchronizing problem from a subsetS of locations under non-negative safety conditions in WAs, to the coverability problem in vector addition systems. The construction is exponential in the size of the WA, and by the fact that the coverability problem is in EXPSPACE, the 2-EXPSPACE membership of location-synchronizing problem from a subsetS of locations under non-negative safety conditions follows.

166 Given a WAA=⟨L, E⟩ over an alphabetA, a subsetS⊆L of locations and a non-negative safety condition Safe, we construct a VASS with two conﬁgura- tions(s init, vinit) and(s end, vend) such that the automatonA under Safe condition has a location-synchronizing word fromS if and only if the conﬁguration(s end, vend) is covered from(s init, vinit) in VASS. The encoding is straightforward. Let the setS have cardinality|S|=m, we thus write S={p 1,···,p m}. We construct the vector addition system VASS=⟨Q, T⟩ with vectors of dimensionm as follows. m m – The state spaceQ=L ∪{s end} whereL is allm-tuples of locations. ′ ′ ′ – From a states=(ℓ 1,···,ℓ m) there is a transition tos = (ℓ1,···,ℓ m) carrying the a,zi ′ vectorv=(z 1,···,z m), if for somea∈A and all0

Lemma 8.4. The location-synchronizing problem from a subsetS of locations under lower- bounded safety conditions in WAs, is EXPSPACE-hard.

Proof. The EXPSPACE-hardness proof is by a reduction from the coverability problem in vector addition systems. From a given VASS=⟨Q, T⟩ equipped with two conﬁgura- tions(s init, vinit) and(s end, vend), we construct a WAA, a lower-bounded safety condition Safe and a setS of locations such that(s end, vend) is covered from(s init, vinit) if and only if the automatonA under the Safe condition has a location-synchronizing word fromS. The intuition behind the reduction is that we addd copies of VASS toA such that the weights in thei-th copy is taken from thei-th dimension of vectors in VASS. An absorbing location called synch is added that is only reachable from the locationss end of each copy. The setS contains synch and the locationss init of all copies,A thus can only be location-synchronized in synch. Therefore, to location-synchronizeS alld copies must run in parallel and try to reach copies ofs end. We assume that all statess of VASS have exactly the same numberm of outgoing transitions (otherwise we add self-loops with0-vectors). We considerm lettersa 1, a2,···,a m and label each outgoing transitiont ins with a unique lettera i where0

167 WAA ⋆:1,#:−1 A:0 a1,···,a m : 0

sink # : 0 # : 0 synch # : 0 # : 0

sinit send ⟨sinit,1⟩ ⟨send,1⟩ ⟨sinit, d⟩ ⟨send, d⟩ ⇒ ··· ⋆:0 ⋆:0 ⋆:0 ⋆:0 VASS copy1 copyd

Figure 8.2: (Schematic) reduction from the coverability problem in vector addition systems with states to location-synchronizing problem from a subsetS of locations under lower- bounded safety conditions in WAs.

– The set of locationsL=Q×{1,2,···,d}∪{synch, sink} includesd copies of VASS and two new locations synch and sink. A state⟨s, i⟩ is in thei-th copy of VASS. – For all lettersa∈A, thea-transition in both locations synch and sink are self-loops implying that those locations are absorbing. The weights of all those transitions are0 except# and⋆-transitions in synch:

⋆ #:−1 synch −−→:+1 synch and synch −−−→ synch.

v ′ – For all transitionss −→s in VASS that is labeled bya and the vectorv=⟨z 1,···,z d⟩, there ared transitions inA such that for all0

a z ⟨s, i⟩ −−→⟨: i s′, i⟩

– Letv=v end −v init and writev=⟨z 1, z2,···,z d⟩. For alls∈Q and0

#:0 ⟨s, i⟩ −−→ sink

except ins end where the#-transition goes to synch:

#:−zi+1 ⟨send, i⟩ −−−−−→ synch.

– For alls∈Q and0

The construction is depicted in Figure 8.2. LetS={⟨s init, i⟩|0

168 First, assume that(s end, vend) is covered from(s init, vinit) in VASS. Thus, there is a sequence of transitionst 0t1 ···t n that are taken from(s init, vinit) to cover(s end, vend). Let w=a 0 ·a 1 ···a n be the sequence of the labels of those transitionst 0t1 ···t n wherea i is the label oft i for all0≤i≤n. The wordw·⋆·# reaches the location synch with non-negative energy level, no matter its origin (inS) and the initial energy. SoA has a location-synchronizing word under the condition Safe. Second, assume thatA has a location-synchronizing wordw. Since synch is absorbing, and since synch∈S thenA is location-synchronized in synch. Entering synch is only possible by reading#, so it must be the case thatw contains some#. Moreover, reading# in synch is only possible after at least one⋆; otherwise the safety condition in the location synch is violated (with energy level1). Letw ′ be the subword ofw that is obtained by omitting all letters⋆ from the prefix ofw until thefirst#. Thus, the wordw ′ has neither# nor⋆. Thefirst⋆ increases the energy level at location synch by1, and thus the #-transition becomes affordable in synch (to location-synchronize other states in synch). Since from all locations⟨s, i⟩ wheres≠ s end, the#-transitions lead to sink where there is no way to synchronize, thefirst# ofw must be read when all copies of VASS are in the locations⟨s end, i⟩ where0

Lemma 8.3 and Lemma 8.4 provide reductions to and from the coverability problem in vector addition systems to establish the following result.

Theorem 8.2. The location-synchronizing problem from a subsetS of locations under lower-bounded safety conditions in WAs, is decidable in 2-EXPSPACE, and it is EXPSPACE-hard.

8.3.2 Location-synchronization under general safety condition We now discuss location-synchronization under the general safety condition where the energy constraint for each location can be a bounded interval, lower or upper-bounded, or trivial (always true). We proceed in two steps:ﬁrst, we prove that the PSPACE- completeness results in case of non-negative safety condition is preserved in location- synchronization under safety condition with only lower-bounded or trivial constraints. Second, we extend our techniques to establish results for general safety conditions. To obtain results for the general case, we use the variant location-synchronization from a subset, that is discussed in all cases as well.

169 ℓ2 a:0 b:0 b:0

ℓ ℓ ℓ a, b:0 0 a, b:0 1 a:0 3

Figure 8.3: To location-synchronize the automaton withL ,→ ={ℓ 0,ℓ 2} andL ↔ ={ℓ 1,ℓ 3}, b:0 taking the back-edgeℓ 1 −→ℓ 2 is avoidable.

Location-synchronization under lower-bounded or trivial safety conditions. Let the safety condition Safe assign to each location ofL either an interval of the form[n,+∞) or true. Accordingly, we partition the setL of locations into two classesL ,→ andL ↔ such thatL ↔ is the set of all locations with true safety constraints, andL ,→ is the set of all locations with a lower-bounded safety constraints. A back-edge is a transitionℓ→ ′ ′ ℓ that goes from some locationℓ∈L ↔ with true safety constraints to some locationℓ ∈ L,→ with lower-bounded safety constraints. An important observation is that while location- synchronizing, all runs starting from some locationℓ∈L ↔ with true safety constraints can ′ never pass through some locationℓ ∈L ,→; otherwise, the runs starting in some state(ℓ, e) wheree<0 is a suﬃciently small integer, even after taking several transitions with positive weights to increase the energy level, arrive in the locationℓ ′ with negative energy level violating the safety condition. Those back-edge transitions though might beﬁred along the runs starting from locations with lower-bounded safety constraints. See Example 8.3. Example 8.3. Consider the WA drawn in Figure 8.3 with four locations and two letters. All transitions have weight0, and the WA is deterministic. From the stateℓ 0, all transitions bring the WA intoℓ 1. Thea-transition in the stateℓ 1 goes toℓ 3; and theb-transition goes toℓ 2. Theb-transition leavesℓ 2 and moves toℓ 3 while thea-transition stays there. The stateℓ 3 is absorbing. The safety condition is non-negative inℓ 0 andℓ 2, and is trivial inℓ 1 b:0 andℓ 3:L ,→ ={ℓ 0,ℓ 2} andL ↔ ={ℓ 1,ℓ 3}. Thus, the transitionℓ 1 −→ℓ 2 is a back-edge. b,0 The worda·b·b is a location-synchronizing word that takes the back-edgeℓ 1 −→ℓ 2 along the following runs starting inℓ 0:

a:0 b:0 b:0 (ℓ0, e) −→(ℓ 1, e) −→(ℓ 2, e) −→(ℓ 3, e) wheree≥0. b:0 While synchronizing by any location-synchronizing word, the back-edgeℓ 1 −→ℓ 2 cannot b:0 be used for the runs starting inℓ 1; otherwise the run(ℓ 1,−1) −→(ℓ 2,−1) would violate the non-negative safety condition atℓ 2. In this example, there exists an alternative worda·a·b that takes no back-edges and still location-synchronizes the automaton. ▹

170 In Example 8.3, we see that even though a back-edge might be taken along runs over some location-synchronizing words such asa·b·b, there are alternative location- synchronizing words that does notfire a back-edge transition, for instance the worda·a·b. We prove, by Lemma 8.5, that such alternative words always exist implying that taking back-edge transitions while synchronizing is avoidable in deterministic WAs. Lemma 8.5. LetA be a deterministic WA and Safe be a safety condition with only lower- bounded or trivial constraints. There is a location-synchronizing word inA if and only if there is one in the automaton obtained fromA by removing all back-edge transitions. Proof. First direction is trivial. To prove the reverse, assume thatA has a location- synchronizing wordw under Safe, such that there exists some state starting in which the b run overw takes a back-edge, say the back-edgeℓ 1 −→ℓ 2 withℓ 1 ∈L ↔ andℓ 2 ∈L ,→. We denote by[n,+∞) the lower-bounded safety constraint inℓ 2. Let(ℓ 0, e0) be some b state such that starting from(ℓ 0, e0), the run overw takes the back-edgeℓ 1 −→ℓ 2 earliest (among all the runs starting from all safe states). We split the wordw=w 0 ·b·w 1 such b thatb is the letterfiring the back-edgeℓ 1 −→ℓ 2 in the run starting from(ℓ 0, e0)first. We prove thatℓ 0 ∈L ,→ meaning thatℓ 0 has lower-bounded safety condition. Towards contradiction, assume thatℓ 0 ∈L ↔. Sincew location-synchronizes all safe states, then the runs starting from all states(ℓ 0, e) wheree∈Z would never violate the safety condition. Lete=n−Z·|w 0|−1 whereZ is the maximum value appearing as weight in transitions in absolute, and wheren is the minimum allowed energy level at the locationℓ 2. Inputting ′ ′ the wordw 0 ·b from the state(ℓ 0, e) bringsA to the state(ℓ 2, n ) withn < n, violating the safety condition atℓ 2. It contradicts with the fact thatw is a location-synchronizing word under Safe. It proves thenℓ 0 ∈L ,→. We denote byU the set of all safe states:U={(ℓ, e)|e∈ Safe(ℓ)}. LetS= loc(post(U, w0)) be the set of all reached locations after inputtingw 0. Since the runs starting from the locationℓ 0 ∈L ,→ are ending inℓ 1 ∈L ↔ by the wordw 0, and sincew 0 does notfire any back-edge transition, thenS∩L ,→ ⊂L ,→ meaning that the number of locations with a lower-bounded safety constraint is decreased after readingw 0. Since the wordw 0 does not violate the safety condition and keeps the automaton in the safe set post(U, w0)⊆U, then w0 ·w is also a location-synchronizing word. Therefore,w 0 ·w is a location-synchronizing word such that there are less runs over itfiring a back-edge, compared to the number of runs overw. We can repeat the above argument for all back-edges (at most|L ,→| times) that are taken while synchronizingA byw, and construct a location-synchronizing wordw ′ ·w such that no back-edge transition isfired while readingw ′. Moreover, all reached locations ′ ′ ′ after readingw have trivial safety constraints loc(post(U, w )⊆L ↔. Hence, the wordw ·w is a location-synchronizing word that no runs starting from some safe state over it takes a back-edge transition. It completes the proof. ! Lemma 8.5 does not hold when synchronizing from a subsetS of the locations. Indeed, consider the one-letter automaton drawn in Figure 8.4. The automaton has three states,

171 ℓ0 a:0

ℓ2

a:0 a:0 ℓ1

Figure 8.4: To location-synchronize the automaton fromS={ℓ 0,ℓ 1} whereL ,→ ={ℓ 0,ℓ 1} a:0 andL ↔ ={ℓ 2}, taking the back-edgeℓ 2 −→ℓ 1 is unavoidable.

but we are only interested to location-synchronize it from two locationsS={ℓ 0,ℓ 1}. The safety condition for those two locations is non-negativeL ,→ =S, however the safety constraint atℓ 2 is trivial. Obviously, it is possible to location-synchronize fromS, and this a:0 would not be possible without taking the back-edgeℓ 2 −→ℓ 1.

Lemma 8.5 also fails for non-deterministic WAs. We provides a non-deterministic WA in Example 8.4 such that for all location-synchronizing word, a back-edge transition must beﬁred.

Example 8.4. Consider the non-deterministic WAA depicted in Figure 8.5.a. The au- tomatonA has four states and three letters. All transitions have weight0, except two transitions. In the stateℓ 0, thea-transitions non-deterministically go to eitherℓ 0 orℓ 2. Theb-transition is a self-loop and thec-transition goes toℓ 3 that is an absorbing state. In b:0 the locationℓ 1, thea-transition is a self-loop whereasb andc-transitions go toℓ 0:ℓ 1 −→ℓ 0 c:−1 a,c:0 b:+1 andℓ 1 −−→ℓ 0. All transitions inℓ 2 leave the location:ℓ 2 −−→ℓ 3 andℓ 2 −−→ℓ 1 The safety condition Safe for locations inL ,→ ={ℓ 0,ℓ 1} is non-negative, and for loca- b:+1 tions inL ↔ ={ℓ 2,ℓ 3} is trivial. Therefore, the transitionℓ 2 −−→ℓ 1 is a back-edge; we b:+1 prove that there is no way to location-synchronizeA unless taking the back-edgeℓ 2 −−→ℓ 1. Assume thatw=a 0 ·a 1 ···a n is a location-synchronizing word. Sinceℓ 3 is an absorbing state, soA is synchronized intoℓ 3. Theﬁrst lettera 0 ofw b:+1 – cannot beb because(ℓ 2,−2) −−→(ℓ 1,−1) violating the non-negative safety condition at the locationℓ 1, c:−1 – cannot bec because(ℓ 1, 0) −−→(ℓ 0,−1) violating the non-negative safety condition at the locationℓ 0. Thus,w starts with the lettera (See Figure 8.5.b.). Inputtinga shrinks the safe set

U={(ℓ i, e)|0≤i≤1, e∈N}∪{(ℓ i, e)|2≤i≤3, e∈Z}

into the set U1 ={(ℓ i, e)|0≤i≤2, e∈N}∪{(ℓ 3, e)|e∈Z}.

172 ⟨ℓ0,N⟩ ⟨ℓ1,N⟩ ⟨ℓ2,Z⟩ ⟨ℓ3,Z⟩

a a, b:0 a:0 ⟨ℓ ,N⟩ ⟨ℓ ,N⟩ ⟨ℓ ,N⟩ ⟨ℓ ,Z⟩ b:0, c:−1 0 1 2 3 ℓ0 ℓ1 b

c:0 b:1 ⟨ℓ0,N⟩ ⟨ℓ1,N+1⟩ ⟨ℓ3,Z⟩ a:0 c ℓ ℓ 3 a, c:0 2 ⟨ℓ0,N⟩ ⟨ℓ3,Z⟩ a, b, c:0 c

a. b. ⟨ℓ3,Z⟩

Figure 8.5: Unavoidable back-edges to synchronize non-deterministic WAs.

Inputting anothera is safe but it would not shrink the setU 1, and in particular would not help in synchronizing. The same reason as before, the second lettera 1 ofw cannot bec. Inputtingb shrinks the setU 1 into

U2 ={(ℓ 0, e)|e∈N}∪{(ℓ 1, e)|e≥1}∪{(ℓ 3, e)|e∈Z}

b:+1 andﬁres the back-edgeℓ 2 −−→ℓ 1. It remains to ensure that there is indeed a way of location-synchronizing into the locationℓ 3, which is inputtingc twice. ▹ By Lemma 8.5, location-synchronizing a given WAA under a safety condition with only lower-bounded or trivial constraints can be achieved in two steps: –ﬁrst location-synchronizing all the states with lower-bounded safety constraints into the set of states with trivial constraints. It can be done in PSPACE by using Theo- rem 8.1, – second location-synchronizing the states with trivial safety constraints where the weights play no role. Lemma 8.6. The location-synchronizing problem in WAs under lower-bounded or trivial safety condition is PSPACE-complete.

Proof. LetA=⟨L, E⟩ be a complete deterministic WA over the alphabetA whereZ is the maximum value appearing as weight in transitions in absolute. Let Safe be the safety

173 condition with only lower-bounded or trivial constraints. By Lemma 8.5, we can assume thatA has no back-edge. We also assume thatL ↔ is not empty. Since there is no back-edge inA, the automaton is synchronized in some locationℓ∈L ↔ with trivial safety constraint. Thus, there exists some wordw such that by reading this wordw, the set of states(ℓ, e) with some locationℓ∈L ,→ having a lower-bounded safety constraint end up in the setL ↔; formally, loc(post(S, w))⊆L ↔ whereS={(ℓ, e)| ℓ∈L ,→, e∈ Safe(ℓ)}. After reading such words, weights play no role while location- synchronizing the reached set post(S,w) of states. Thus, we propose following PSPACE algorithm: ′ – We obtain the WAA fromA by replacing all locationsℓ∈L ↔ with some absorbing location synch. All ingoing-transition to some locationℓ∈L ↔ is redirected to synch. At the location synch, we deﬁne the safety constrainte≥m−Z where m is the minimum allowed energy level for all locations with a lower-bounded safety constraints:

m = min{e| there existsℓ∈L ,→ such thate∈ Safe(ℓ)}.

We choosem−Z as the minimum allowed energy level at synch since the least energy level for locations inL ,→ ism, and the energy level cannot be decreased more thanZ by taking the transitions going to synch . Weﬁnd a location-synchronizing wordw forA ′ by using Lemma 8.1. – We obtain the DFAN fromA by removing all locationsℓ∈L ,→ and omitting the weights of transitions. Weﬁnd a synchronizing wordv for the DFAN. By above arguments, the wordw·v is a location-synchronizing word forA. The proof is complete. !

The proof of Lemma 8.6 carries on for synchronizing from a subset of locations, except using Lemma 8.3 instead of Theorem 8.1, and requiring that the automaton has no back- edge.

Lemma 8.7. Given a WAA with no back-edges, the location-synchronizing problem from a subsetS of locations under lower-bounded or trivial safety condition is decidable in 2- EXPSPACE, and it is EXPSPACE-hard.

Location-synchronization under general safety conditions. Let us relax the restrictions on the safety condition Safe, and let some of the locations have bounded intervals to indicate the safe range of energy. The setL of locations is partitioned intoL ,−!,L ,→ andL ↔ where locations inL ,−! have safety conditions such as e∈[n 1, n2] wheren 1, n2 ∈Z. In this setting, transitions from locations inL ,→ orL ↔ to locations inL ,−! are considered as back-edge too. Since taking back-edge transitions while synchronizing from a subsetS of locations is not avoidable, we can use bounded

174 safety conditions to establish a reduction from the halting problem in Minsky machines to provide the following undecidability result.

Theorem 8.3. The location-synchronizing problem from a setS of locations in WAs under a general safety condition is undecidable.

Proof. The proof is by a reduction from the halting problem in two-counter Minsky machine. Below, without loss of generality, we assume that the Minsky machine has at least two guarded decrement instructions, one on each counter. From a Minsky machine, we construct a deterministic WAA, a general safety condition Safe and a subsetS such that Minsky machine halts if and only ifA has a location- synchronizing word fromS under the condition Safe. The automatonA is constructed from two disjoint automatonB 0 andB 1 (and some other locations such as synch) with the same number of locations and transitions. A run over automatonB j simulate the value of counterc j along a sequence of conﬁgurations in the Minsky machine. The only way to synchronize these two disjoint automata is arriving to their halt, simultaneously, and then play a special letter# to reach the location synch. To let the counters to get freely any non-negative value, the safety condition in all location ofB 0 andB 1 are non-negative except some particular locations that are reserved to check the correctness of a guarded decrement. The guarded decrement instructions are simulated by taking a back-edge and visiting locations with the safety condition of forme=0. To synchronize all locations with diﬀerent kind of safety conditions, we add a gadget that forces all location-synchronizing words forA to always begin with the letter⋆. The construction ofA over the alphabetA is as follows. An instance of reduction is depicted in Figure 8.6. – The alphabet has four lettersA={a, b,#,⋆}. – For each increment instructionsi:c j :=c j + 1; gotok; we have two locationsi j, kj c,1 and the transitionsi j −→k j inB j and two locationsi 1−j, k1−j and the transitions c,0 i1−j −→k 1−j inB 1−j wherec∈{a, b}. See instructions1 and3 in Figure 8.6 and the corresponding locations and transitions in Figure 8.7. – For each guarded decrement instructionsi: ifc j = 0 gotok else(c j :=c j − 1; ′ ′ gotok ); we have four locationsi j, kj, kj, fi,j and the transitions

b:0 c:0 a:−1 ′ ij −→f i,j , fi,j −→k j , ij −−→k j

′ inB j and similarly, four locationsi j, kj, kj, fi,j and following transitions

a:0 b:0 c:0 ′ i1−j −→k 1−j , i1−j −→f i,1−j , fi,1−j −→k 1−j

inB 1−j wherec∈{a, b}. See instruction4 in Figure 8.6 and the corresponding locations and transitions in Figure 8.7.

175 WAA a, b ⋆:−1, a, b

⋆ 1 :c 1 :=c 1 + 1; goto 2; d s ··· # # 3 :c 0 :=c 0 + 1; goto 4; 4 : ifc 1 = 0 goto5 A sink synch A else ⇒ c1 :=c 1 − 1; goto 4; # #,⋆ ··· # ⋆ # ⋆ 7 : halt; 10 70 #,⋆ 11 71 Minsky machine ⋆ # f4,0 ⋆ f4,1 B0 B1

Figure 8.6: (Schematic) reduction from the halting problem in Minsky machines to location-synchronizing problem from a subsetS of locations under general safety conditions in WAs. To simplify theﬁgure, the weights of transitions with weight0 are omitted. See Figure 8.7 for the details ofB 0 andB 1.

– We add two new absorbing locations synch and sink meaning that all transitions are self-loops with weight0. – We have a gadget with two new locations “departure”d and “stay”s, and following transitions: c ⋆ ⋆ c d −−→:−1 d, d −→:0 s, s −−→:−1 s, s −→:0 s wherec={a, b}. – From the stay locations,n 0 andn 1 where then-th instruction is halt, the#-transition goes to synch with weight0, but from all other locations the#-transition goes to the location sink. – From all locations inB j, the⋆-transition leads to synch except in locationsf i,j if the instructioni is a guarded decrement on counterc j. From these locations, the ⋆-transition goes to location1 j (wherej∈{0,1}). The safety condition for synch and sink is trivial, and for all locationsf i,j inB j, it is of the forme=0 if the instructioni is a guarded decrement on the counterc j (for0

To establish the correctness of the reduction,ﬁrst assume that Minsky machine halts. Thus, there is a sequence ofm conﬁgurations starting from (1, (0, 0)) and reaching halt:

(inst1, v1)(inst2, v2)···(inst m, vm) where inst1 = 1 and instm = halt.

176 a, b a, b:1 10 20 30 11 21 31 a, b:1 a, b ⋆ a 40 a:−1 41

b b a, b a, b 70 50 f4,0 71 51 f4,1 B0 B1

Figure 8.7: The details ofB 0 andB 1 in Figure 8.6. To simplify theﬁgure, the weights of transitions with weight0 are omitted.

Consider the wordw=⋆·w 1 ···w m ·♯ where –w i =a if inst i is an increment or a guarded decrement onc j whene j >0 in the valuationv i = (e0, e1), –w i =b·b if instructioni is a guarded decrement onc j whene j = 0 inv i = (e0, e1). We see thatw is a location-synchronizing word forA. Second, assume thatA has a location-synchronizing wordw. Since in the departure locationd the only letter that does not violate the safety condition is⋆,w must start with the letter⋆. Reading⋆ shrinks the setS into the locations1 0,1 1 with energy level0,s with all non-negative energy levels, and synch with all integer energy levels:

S1 ={(1 0, 0), (11, 0)}∪{(s, e)|e∈N}∪{(synch,e)|e∈Z}.

Since synch is absorbing, thusA is location-synchronized in synch; this location is only reachable by# and⋆. On the other hand, due to the safety condition at the stay locations, inputting more⋆’s is impossible. Thus,w must have some occurrence of# to location- ′ synchronize the setS 1 into synch. Letw =a 1 ·a 2 ···a n be the subword ofw after the first⋆ and up to thefirst#,w ′ thus has neither# nor⋆. Below, we show that there is a sequence of configurations such that the Minsky machine halts by operating this sequence. Consider (1, (0, 0)) to be thefirst configuration of the Minsky machine. For all1

1. let(ℓ, e j) = post((1j, 0), a1 ···a k) in the automatonB j for bothj={1,2},

2. ifℓ is of the formf i,j skip the next step,

3. define the next configuration(ℓ, v) of the Minsky machine such thatv=(e 1, e2). We know that after the subwordw ′, there is an immediate# in the location- synchronizing wordw. The#-transitions in all locations ofB 1 andB 2 except halt lo- cationsn 1 andn 2, bringA to the location sink (where there is no way to be synchronized with synch). Thus, after readingw ′ both automaton must be in their halt; otherwisew is not a location-synchronizing word. It implies that the constructed sequence of configurations for Minsky machine halts. Moreover, that is a valid configuration thanks to the

177 zero safety constraints at the locations simulating the guarded decrement instructions and non-negative safety constraints in other locations ofB 0 andB 1. !

In the absence of back-edges, we can get rid of the bounded safety constraints by explicitly encoding the energy values in the locations at the expense of an exponential blowup. We thus assign non-negative safety constraints to the encoded location and reduce to Lemma 8.6. Lemma 8.8. Given a WAA with no back-edges, the location-synchronizing problem from a subsetS of locations under safety condition with bounded, lower-bounded or trivial constraints is decidable in 3-EXPSPACE, and it is EXPSPACE-hard.

Proof. Let the WAA=⟨L, E⟩ and the safety condition Safe be such that the set of locations are partitioned as followsL=L ,−! ∪L ,→ ∪L ↔. Without loss of generality, assume ′ thatL ↔ ⊆S. We build another WAA in which the locations ofL ,−! are augmented with the explicit value of the energy levels. The construction ofA ′ =⟨L ′,E′⟩ is as follows. The set of locations

′ L ={⟨ℓ, e⟩|ℓ∈L ,−!, e∈ Safe(ℓ)}∪L ,→ ∪L ↔ contains all locations inL ,→ ∪L ↔ and a location⟨ℓ, e⟩ for all pairs of locationsℓ∈L ,−! and possible safe energy level for that locationℓ. There are following transitions inA ′: – for all pairs⟨ℓ, e⟩,⟨ℓ ′, e+z⟩ ∈L ′, there is a transition(⟨ℓ,e⟩, a,0,⟨ℓ ′, e+z⟩)∈E ′ from⟨ℓ,e⟩ to⟨ℓ ′, e+z⟩ with weight0 inA ′ if(ℓ, a, z,ℓ ′)∈E, ′ ′ ′ ′ – for all⟨ℓ, e⟩ ∈L andℓ ∈L ,→ ∪L ↔, there is a transition(⟨ℓ, e⟩, a, e+z,ℓ )∈E from⟨ℓ,e⟩ toℓ ′ with weighte+z inA ′ if(ℓ, a, z,ℓ ′)∈E, ′ ′ ′ ′ – for all pairs of locationsℓ,ℓ ∈L ,→ ∪L ↔, there is a transition(ℓ, a, z,ℓ )∈E inA if(ℓ, a, z,ℓ ′)∈E. We then deﬁne the safety condition Safe′ overL ′ by letting Safe′((ℓ,e)) = [0,+∞), ′ and Safe (ℓ) = Safe(ℓ) for allℓ∈L ,→ ∪L ↔. Finally, given a set of locationsS⊆L containingL ↔, we let

′ ′ S ={⟨ℓ, e⟩ ∈L |ℓ∈S∩L ,−!}∪[S∩(L ,→ ∪L ↔)].

′ ′ Assuming thatL ,→ ∪L↔ ≠ ∅, we prove that a location-synchronizing word inA fromS under safety condition Safe′ is also a location-synchronizing word inA fromS under safety condition Safe, and vice-versa. ′ ′ First, assume thatA has a location-synchronizing wordw. Letℓ f ∈L be the location ′ ′ ′ whereA is synchronized in fromS :ℓ f = loc(post(S , w)). Consider a state(ℓ, e) ofA whereℓ∈S. There are two cases: (i) the locationℓ∈L ,→ ∪L ↔ has lower-bounded or trivial safety constraint, then(ℓ, e) is a valid state inA ′. Moreover, all states and transitions visited along the run ofw overA ′ are valid states and transitions inA too. Thus, inputtingw from(ℓ, e) inA brings the automaton in the same locationℓ f .(ii) the

178 locationℓ∈L ,−! has bounded safety constraint. The run ofA over the wordw from the state(ℓ, e) is mimicked by run ofA ′ but starting from(⟨ℓ,e⟩, 0). As a consequence, this run ends inℓ f too. The converse implication is similar. Assume that there is a location-synchronizing wordw inA that merges all runs starting fromS into the same locationℓ f . We show ′ thatw is location-synchronizing inA too. For all locations inS∩(L ,→ ∪L ↔), as well as for the states of the form((ℓ,e), 0), it is easy to see that the run ofA overw is mimicked with the run ofA ′. Moreover, since there are only lower-bound or trivial constraints in Safe′ condition forA ′, it is the case from states of the form((ℓ,e), f) withf>0 too. We then apply the methods used in Lemma 8.7 to the WAA ′. Since the construction ofA ′ involves an exponential blowup in the number of locations, we end up with a 3- EXPSPACE algorithm. The EXPSPACE lower bound proved in Lemma 8.4 still applies here. !

Now, we consider general safety conditions where the constraints can be bounded, lower-bounded, upper-bounded or trivial constraints. The setL of locations is partitioned intoL ,−!,L ,→,L ←! andL ↔ where locations inL ←! have upper-bounded safety constraints such ase≤n wheren∈Z. In this setting, transitions between locations inL ←! and locations in both ofL ,→ andL ↔ are taken as back-edge too. Lemma 8.9. The location-synchronizing problem in WAs under a general safety condition is decidable in 3-EXPSPACE.

Proof. LetA=⟨L, E⟩ be a WA over the alphabetA, and let Safe be a general safety condition. Considering the safety condition, we partition the set of locations intoL= L,−! ∪L ,→ ∪L ←! ∪L ↔. Using similar arguments as presented in the proof of Lemma 8.5, we remove all back-edges. Since all back-edges are removed, for all wordsw, starting from some locationℓ∈L ,−! the run overw either visit locations inL ←! or location inL ,→. We thus can turn the upper-bound constraints to the lower-bound constraints by negating the weights of transitions in all locationsℓ∈L ←! with such upper-bound constraints. The following non-deterministic algorithm decides whether the WAA has a location- synchronizing wordw under Safe. Itﬁrst non-deterministically partitionsL ,−! into two sets ←! ,→ L,−! andL ,−! such that the locations from which the run overw only visits location inL ←! are ←! guessed and are contained inL ,−! , and the locations from which the run overw only visits ,→ ′ ′ ′ location inL ,→ are contained inL ,−! . The algorithm next builds another WAA =⟨L ,E ⟩ fromA and Safe as follows. The set of locations is

′ L = (L,−! ∪L ←!)×{−1}∪(L ,−! ∪L ,→ ∪L ↔)×{1}

and – for allx∈{−1,1}, the transition(⟨ℓ, x⟩, a, x×z,⟨ℓ ′, x⟩)∈E ′ is inA ′ if and only if (ℓ, a, z,ℓ ′)∈E;

179 ′ ′ ′ – for all locationsℓ∈L ,−! ∪L ←! andℓ ∈L ↔, the transition(⟨ℓ,−1⟩, a,−z,⟨ℓ ,1⟩)∈E is inA ′ if and only if(ℓ, a, z,ℓ ′) is inE. Now we change the safety conditions and deﬁne Safe′ such thate∈ Safe ′(⟨ℓ,y⟩) if and only if ye∈ Safe(ℓ). Notice that Safe ′ then only has bounded, lower-bounded or trivial safety constraints. This construction has the following property: a wordw is a location-synchronizing word inA with safety constraint Safe if and only if it is also a location-synchronizing ′ ←! ,→ word inA from the subsetS=(L ,−! ∪L ←!)×{−1}∪(L ,−! ∪L ,→ ∪L ↔)×{1} with safety constraint Safe′. Both implications are straightforwardly proven, and sinceA ′ has no back-edges the 3-EXPSPACE result follows. !

From previous Lemmas and arguments, Theorem 8.4, stating the complexity bounds of location-synchronizing word in WAs under a general safety condition, follows. Regard- ing the PSPACE-hardness result, the reduction from partial transition function used in Lemma 8.2 can be adapted to use lower-bounded and interval safety constraint, which entails the result.

Theorem 8.4. The location-synchronizing problem in WAs under a general safety condition is decidable in 3-EXPSPACE, and it is PSPACE-hard.

180 Conclusion9

Summary. The main contribution of this thesis is a novel approach to the qualitative analysis and the synthesis problem of the distribution-outcomes in probabilistic systems. We introduced synchronizing properties defined on the distribution-outcomes of Markov decision processes (MDPs) and probabilistic automata (PAs), that require that the winning strategy (or the input word) brings the system in some (group of) state(s) with a large probability, possibly in limit. In other words, the probability mass in the distribution, that indicates the likelihood of the system to be in different states at specific states, is accumulated in some target state. We defined always, eventually, weakly and strongly synchronizing properties that respectively are in analogy with safety, reachability, Büchi and coBüchi conditions in state-semantics. We considered three qualitative winning modes: sure, almost-sure and limit-sure modes, and we provided matching upper and lower complexity bounds of the existence problem of a winning strategy for synchronizing conditions, as well as the memory requirement for optimal winning strategies. In addition to the existence problem of a winning word-strategy in PAs that has been considered as the emptiness problems of synchronizing languages, we also discussed the universality problems. We carried over results for synchronizing problems from MDPs and PAs to two-player turn-based games and non-deterministicfinite state automata. Along with the main results, we established new complexity results for alternatingfinite automata over a one-letter alphabet. As a further contribution, we introduced the synchronization for timed automata (TAs) and weighted automata (WAs). We provided results for several variants and combinations of these: including deterministic and non-deterministic timed and weighted automata, synchronization to unique location with possibly different clock valuations or accumulated

181 weights, as well as synchronization with a safety condition forbidding the automaton to visit states outside a safety-set during synchronization (e.g. energy constraints).

Future works. Partially-observable Markov decision processes (POMDPs) are a generalization of MDPs where the controller, which makes the strategic choices, is not able to distinguish states, but groups of states through an observation [CDH10]. Thus, the states of systems are grouped (or sometimes partitioned) in diﬀerent observations, and the controller choice only relies on partial information about the history of the system execution, namely, on the past sequence of observations. As one direction for further in- vestigation, synchronizing properties in POMDPs can be considered. In particular, since PAs are a kind of POMDPs where the set of observations is a singleton, the undecidability results for the emptiness problem of some synchronizing languages in PAs, immediately give the undecidability for the existence problem of a winning observation-based strategy in POMDPs. On the other hand, MDPs are a kind of POMDPs where the set of observations is equal to the state space, and we have seen that all synchronizing problems are decidable in MDPs. Going from decidability to undecidability of synchronizing problems in these two extreme kinds of POMDPs, one can see which type of observations functions arise a decidable class of synchronizing properties in POMDPs; particularly, studying the sets of (practically meaningful) restrictions on the class of winning strategies with decidable synchronizing problems is interesting.

1 MDPs can be viewed as1 2 -player stochastic games, and thus a natural generalization 1 of them is2 2 -player stochastic games where in addition to strategic choices of a controller (Player-1) and random responses of its environment, strategic choices for an adversary (Player-2) exist [CH12]. Along the results for the synchronizing properties in MDPs, we 1 have established results for turn based two player games, that is kind of2 2 -player stochastic games where the environment always responds deterministically. The links between synchronizing properties in MDPs to turn based two player games are established by the fact that the exact value of probabilities in transitions of MDPs do not matter while syn- 1 chronization. It would be very interesting to see whether this result holds in2 2 -player stochastic games, and to study the role of a de-synchronizing adversary and the memory requirement for the strategies of the adversary.

It is known that an irreduciblefinite Markov chain has a set of stationary distributions [KS83, Nor98, Ser13]. For an MDP and a strategy, a distributionX can be defined as a stationary distribution when in the distribution-outcome, there is an infinite subsequence of distributions converging toX. Given a set of target stationary distributions and an MDP, one can study limit conditions in analogy to Büchi and coBüchi conditions: does there exists a strategy for the MDP such that some or all stationary distributions are from the target set? The weakly and strongly synchronizing conditions are kind of limit conditions; for example, for the function max T the set of target distributions in limit condition is the set of Dirac distributions on states ofT . Another interesting point could be observing the behavior of the maximal subsequence of the distribution-outcome con-

182 verging to a stationary distributionX. In particular, taking intervals probabilities and assigning symbols to each interval distribution, the language obtained in the convergence of a maximal subsequence to a stationary distributionX, is irregular for Markov chains and if only one entry ofX is of interest [AAGT12]. It is interesting toﬁnd the class of strategies such that the MDP shows a regular behavior when converging to a stationary distribution.

In this thesis, we have studied synchronizing and location-synchronizing problems for TAs and WAs. We left an open question in the thesis: decidability of the location- synchronizing problem in non-deterministic WAs. We showed that the technique used to solve location-synchronizing problem in deterministic WAs fails when applying for non- deterministic WAs, that is namely avoiding back-edges. Avoiding back-edges is not also possible when location-synchronizing from a subset of locations in deterministic WAs; though in that case we provided the undecidability result. It would be interesting toﬁll this gap and complete the extensive study done on synchronization of WAs. Recently, the study of synchronizing strategies under partial observation has been motivated in turn- based games [LLS14]. In this direction, a further topic to explore is deﬁning synchronizing conditions in timed and weighted games, considering perfect-information and partial observation strategies for the players.

The concept of synchronization has been found useful in applications such as planning, control of discrete event systems, bio-computing, and robotics [BAPE+03, Vol08, DMS11a]. The usage of synchronization also shows promises in the verification of cryptographic protocols. Cryptographic protocols are small programs that are specifically designed to ensure the security of private communications on public networks such as wireless ad-hoc networks. Contrary to classical networks using routers and access-points, ad-hoc networks consist of communicating members that broadcast a message one to another. Messages sent by a source member of the network is typically forwarded by other members until the message reaches its destination. Onion protocols have been specifically developed to ensure the anonymity of the members sending messages, meaning that a malicious member of the network should not be able to determine the identity of the sender from the forwarded messages [CD13]. By design, before forwarding a message, a member adds an encrypted layer of routing information to the message which increases the size of the message. The malicious member thus can determine the length of the path the message has gone through from the sender to himself. Using the concept of synchronization in the topology of small examples of networks, one can see that a sender, from which all paths to the malicious member have the same length, can be identified implying a breach on the anonymity property supposedly guaranteed by onion protocols. A further work could be to formally specify the synchronization requirements on the topology of ad-hoc networks and combine them with classic verification techniques on cryptographic protocols [CD13] to ensure the security of onion protocols.

183 184 Bibliography

[AAGT12] Manindra Agrawal, S. Akshay, Blaise Genest, and P. S. Thiagarajan. Approx- imate verification of the symbolic dynamics of Markov chains. In Proceedings of the 27th Annual IEEE Symposium on Logic in Computer Science, LICS 2012, Dubrovnik, Croatia, June 25-28, 2012, pages 55–64. IEEE, 2012. [AD94] Rajeev Alur and David L. Dill. A theory of timed automata. Theoretical Computer Science, 126(2):183–235, 1994. [AGV10] Dimitry S. Ananichev, Vladimir V. Gusev, and Mikhail V. Volkov. Slowly synchronizing automata and digraphs. In Mathematical Foundations of Com- puter Science 2010, 35th International Symposium, MFCS 2010, Brno, Czech Republic, August 23-27, 2010. Proceedings, volume 6281 of Lecture Notes in Computer Science, pages 55–65. Springer, 2010. [AH90] James Aspnes and Maurice Herlihy. Fast randomized consensus using shared memory. Journal Algorithm, 11(3):441–461, 1990. [BAPE+03] Yaakov Benenson, Rivka Adar, Tamar Paz-Elizur, Zvi Livneh, and Ehud Shapiro. DNA molecule provides a computing machine with both data and fuel. Proceedings of the National Academy of Sciences, USA, 100:2191–2196, 2003. [BBG08] Christel Baier, Nathalie Bertrand, and Marcus Größer. On decision problems for probabilistic Büchi automata. In Foundations of Software Science and Computational Structures, 11th International Conference, FOSSACS 2008, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2008, Budapest, Hungary, March 29 - April 6, 2008. Pro- ceedings, volume 4962 of Lecture Notes in Computer Science, pages 287–301. Springer, 2008. [BBL08] Patricia Bouyer, Ed Brinksma, and Kim Guldstrand Larsen. Optimal infinite scheduling for multi-priced timed automata. Formal Methods in System Design, 32(1):3–23, 2008. [BBMR08] Roberto Baldoni, François Bonnet, Alessia Milani, and Michel Raynal. On the solvability of anonymous partial grids exploration by mobile robots. In Prin- ciples of Distributed Systems, 12th International Conference, OPODIS 2008, Luxor, Egypt, December 15-18, 2008. Proceedings, volume 5401 of Lecture Notes in Computer Science, pages 428–445. Springer, 2008. [BBS06] Christel Baier, Nathalie Bertrand, and Ph. Schnoebelen. On computingfix- points in well-structured regular model checking, with applications to lossy

185 channel systems. In Logic for Programming, Artificial Intelligence, and Rea- soning, 13th International Conference, LPAR 2006, Phnom Penh, Cambodia, November 13-17, 2006, Proceedings, volume 4246 of Lecture Notes in Com- puter Science, pages 347–361. Springer, 2006. [BCH+07] Christel Baier, Lucia Cloth, Boudewijn R. Haverkort, Matthias Kuntz, and Markus Siegle. Model checking Markov chains with actions and state labels. IEEE Transaction Software Engineering, 33(4):209–224, 2007. [BGB12] Christel Baier, Marcus Größer, and Nathalie Bertrand. Probabilisticω- automata. Journal ACM, 59(1):1, 2012. [BHK+12] Tomás Brázdil, Holger Hermanns, Jan Krcál, Jan Kretínský, and Vojtech Rehák. Verification of open interactive Markov chains. In IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science, FSTTCS 2012, December 15-17, 2012, Hyderabad, India, volume 18 of LIPIcs, pages 474–485. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, 2012. [BK98a] Christel Baier and Marta Z. Kwiatkowska. Model checking for a probabilistic branching time logic with fairness. Distributed Computing, 11(3):125–155, 1998. [BK98b] Christel Baier and Marta Z. Kwiatkowska. On the verification of qualitative properties of probabilistic processes under fairness constraints. Inf. Process. Lett., 66(2):71–79, 1998. [BK08] Christel Baier and Joost-Pieter Katoen. Principles of model checking. MIT Press, 2008. [BKHW05] Christel Baier, Joost-Pieter Katoen, Holger Hermanns, and Verena Wolf. Comparative branching-time semantics for Markov chains. Information and Computation, 200(2):149–214, 2005. [BS96] Eric Bach and Jeffrey Shallit. Algorithmic Number Theory, Vol. 1: Efficient Algorithms. MIT Press, 1996. [CD11] Krishnendu Chatterjee and Laurent Doyen. Games and Markov decision processes with mean-payoff parity and energy parity objectives. In Mathematical and Engineering Methods in Computer Science - 7th International Doctoral Workshop, MEMICS 2011, Lednice, Czech Republic, October 14-16, 2011, Revised Selected Papers, volume 7119 of Lecture Notes in Computer Science, pages 37–46. Springer, 2011. [CD13] Rémy Chrétien and Stéphanie Delaune. Formal analysis of privacy for routing protocols in mobile ad hoc networks. In Principles of Security and Trust - Sec- ond International Conference, POST 2013, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2013, Rome, Italy, March 16-24, 2013. Proceedings, volume 7796 of Lecture Notes in Computer Science, pages 1–20. Springer, 2013.

186 [CDH10] Krishnendu Chatterjee, Laurent Doyen, and Thomas A. Henzinger. Qualita- tive analysis of partially-observable Markov decision processes. In Mathemat- ical Foundations of Computer Science 2010, 35th International Symposium, MFCS 2010, Brno, Czech Republic, August 23-27, 2010. Proceedings, volume 6281 of Lecture Notes in Computer Science, pages 258–269. Springer, 2010. [Čer64] JánČerný. Poznámka k homogénnym experimentom s konečnými automatmi. Matematicko-fyzikálnyčasopis, 14(3):208–216, 1964. [CH12] Krishnendu Chatterjee and Thomas A. Henzinger. A survey of stochastic ω-regular games. Journal of Computer and System Sciences, 78(2):394–413, 2012. [CHJS13] Krishnendu Chatterjee, Monika Henzinger, Manas Joglekar, and Nisarg Shah. Symbolic algorithms for qualitative analysis of Markov decision processes with Büchi objectives. Formal Methods in System Design, 42(3):301–327, 2013. [CHP08] Krishnendu Chatterjee, Thomas A. Henzinger, and Nir Piterman. Algorithms for Büchi games. CoRR, abs/0805.2620, 2008. [CJS12] Krishnendu Chatterjee, Manas Joglekar, and Nisarg Shah. Average case analysis of the classical algorithm for Markov decision processes with Büchi objectives. In IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science, FSTTCS 2012, December 15-17, 2012, Hyderabad, India, volume 18 of LIPIcs, pages 461–473. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, 2012. [CKV+11] Rohit Chadha, Vijay Anand Korthikanti, Mahesh Viswanathan, Gul Agha, and YoungMin Kwon. Model checking MDPs with a unique compact invari- ant set of distributions. In Eighth International Conference on Quantitative Evaluation of Systems, QEST 2011, Aachen, Germany, 5-8 September, 2011, pages 121–130. IEEE Computer Society, 2011. [CSV08] Rohit Chadha, A. Prasad Sistla, and Mahesh Viswanathan. On the expressive- ness and complexity of randomization infinite state monitors. In Proceedings of the Twenty-Third Annual IEEE Symposium on Logic in Computer Science, LICS 2008, 24-27 June 2008, Pittsburgh, PA, USA, pages 18–29. IEEE Com- puter Society, 2008. [CT12] Krishnendu Chatterjee and Mathieu Tracol. Decidable problems for probabilistic automata on infinite words. In Proceedings of the 27th Annual IEEE Symposium on Logic in Computer Science, LICS 2012, Dubrovnik, Croatia, June 25-28, 2012, pages 185–194. IEEE, 2012. [CY95] Costas Courcoubetis and Mihalis Yannakakis. The complexity of probabilistic verification. Journal of the ACM, 42(4):857–907, 1995. [dA97] Luca de Alfaro. Formal Verification of Probabilistic Systems. PhD thesis, Stanford University, 1997.

187 [dAH00] Luca de Alfaro and Thomas A. Henzinger. Concurrent omega-regular games. In 15th Annual IEEE Symposium on Logic in Computer Science, Santa Bar- bara, California, USA, June 26-29, 2000, pages 141–154. IEEE Computer Society, 2000. [dAHK07] Luca de Alfaro, Thomas A. Henzinger, and Orna Kupferman. Concurrent reachability games. Theoretical Computer Science, 386(3):188–217, 2007. [Die12] Reinhard Diestel. Graph Theory, 4th Edition, volume 173 of Graduate texts in mathematics. Springer, 2012. [DJL+14] Laurent Doyen, Line Juhl, Kim G. Larsen, Nicolas Markey, and Mahsa Shir- mohammadi. Synchronizing words for weighted and timed automata. In Pro- ceedings of the 34th Conference on Foundations of Software Technology and Theoretical Computer Science, FSTTCS’14, Leibniz International Proceedings in Informatics. Leibniz-Zentrum für Informatik, 2014. To appear. [DMS11a] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Inﬁnite synchronizing words for probabilistic automata. In Mathematical Foundations of Computer Science 2011 - 36th International Symposium, MFCS 2011, War- saw, Poland, August 22-26, 2011. Proceedings, volume 6907 of Lecture Notes in Computer Science, pages 278–289. Springer, 2011. [DMS11b] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Synchroniz- ing objectives for Markov decision processes. In Proceedings International Workshop on Interactions, Games and Protocols, iWIGP 2011, Saarbrücken, Germany, 27th March 2011., volume 50 of EPTCS, pages 61–75, 2011. [DMS12] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Inﬁnite synchronizing words for probabilistic automata (erratum). CoRR, abs/1206.0995, 2012. [DMS14a] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Limit synchronization in Markov decision processes. In Foundations of Software Sci- ence and Computation Structures - 17th International Conference, FOSSACS 2014, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2014, Grenoble, France, April 5-13, 2014, Proceedings, volume 8412 of Lecture Notes in Computer Science, pages 58–72. Springer, 2014. [DMS14b] Laurent Doyen, Thierry Massart, and Mahsa Shirmohammadi. Robust synchronization in Markov decision processes. In CONCUR 2014 - Concur- rency Theory - 25th International Conference, CONCUR 2014, Rome, Italy, September 2-5, 2014. Proceedings, volume 8704 of Lecture Notes in Computer Science, pages 234–248. Springer, 2014. [EKVY08] Kousha Etessami, Marta Z. Kwiatkowska, Moshe Y. Vardi, and Mihalis Yan- nakakis. Multi-objective model checking of Markov decision processes. Logical Methods in Computer Science, 4(4), 2008.

188 [EL10] Stewart Ethier and Jiyeon Lee. A Markovian slot machine and parrondo’s paradox. Annals of Applied Probability, 20:1098–1125, 2010. [Epp90] David Eppstein. Reset sequences for monotonic automata. SIAM J. Comput., 19(3):500–510, 1990. [FJLS11] Uli Fahrenberg, Line Juhl, Kim G. Larsen, and Jirí Srba. Energy games in multiweighted automata. In Theoretical Aspects of Computing - ICTAC 2011 - 8th International Colloquium, Johannesburg, South Africa, August 31 - September 2, 2011. Proceedings, volume 6916 of Lecture Notes in Computer Science, pages 95–115. Springer, 2011. [FP06] Wan Fokkink and Jun Pang. Variations on Itai-Rodeh leader election for anonymous rings and their analysis in PRISM. Journal of Universal Computer Science, 12(8):981–1006, 2006. [FV97] Jerzy Filar and Koos Vrieze. Competitive Markov Decision Processes. Springer, 1997. [FV13] Fedor M. Fominykh and Mikhail V. Volkov. P(l)aying for synchronization. In- ternational Journal of Foundations of Computer Science, 24(6):765–780, 2013. [GJ79] M. R. Garey and David S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman, 1979. [GO10] Hugo Gimbert and Youssouf Oualhadj. Probabilistic automata onfinite words: Decidable and undecidable problems. In Automata, Languages and Program- ming, 37th International Colloquium, ICALP 2010, Bordeaux, France, July 6-10, 2010, Proceedings, Part II, LNCS 6199, pages 527–538. Springer, 2010. [GR88] A. Gibbons and W. Rytter.Efficient parallel algorithms. Cambridge University Press, 1988. [HKK14] Holger Hermanns, Jan Krcál, and Jan Kretínský. Probabilistic bisimulation: Naturally on distributions. In CONCUR 2014 - Concurrency Theory - 25th International Conference, Rome, Italy, September 2-5, 2014. Proceedings, volume 8704 of Lecture Notes in Computer Science, pages 249–265. Springer, 2014. [HMW09] Thomas A. Henzinger, Maria Mateescu, and Verena Wolf. Sliding window abstraction for infinite Markov chains. In Computer Aided Verification, 21st International Conference, CAV 2009, Grenoble, France, June 26 - July 2, 2009. Proceedings, volume 5643 of Lecture Notes in Computer Science, pages 337–352. Springer, 2009. [Hol95] Markus Holzer. On emptiness and counting for alternatingfinite automata. In Developments in Language Theory, pages 88–97, 1995. [IRS76] Harry B. Hunt III, Daniel J. Rosenkrantz, and Thomas G. Szymanski. On the equivalence, containment, and covering problems for the regular and context- free languages. Journal of Computer and System Sciences, 12(2):222–268, 1976.

189 [IS95] Balázs Imreh and Magnus Steinby. Some remarks on directable automata. Acta Cybern., 12(1):23–35, 1995. [IS99] Balazs Imreh and Magnus Steinby. Directable nondeterministic automata. Acta Cybernetica, 14(1):105–115, 1999. [JS07] Petr Jancar and Zdenek Sawa. A note on emptiness for alternatingﬁ- nite automata with a one-letter alphabet. Information Processing Letters, 104(5):164–167, 2007. [Kfo70] D.J. Kfoury. Synchronizing sequences for probabilistic automata. Studies in Applied Mathematics, 29:101–103, 1970. [KNP05] Marta Z. Kwiatkowska, Gethin Norman, and David Parker. Probabilistic model checking in practice: case studies with PRISM. SIGMETRICS Perfor- mance Evaluation Review, 32(4):16–21, 2005. [KNP08] Marta Z. Kwiatkowska, Gethin Norman, and David Parker. Using probabilistic model checking in systems biology. SIGMETRICS Performance Evaluation Review, 35(4):14–21, 2008. [Koz77] Dexter Kozen. Lower bounds for natural proofs systems. In Proceedings of 18th Symposium on the Foundations of Computer Science, pages 254–266, 1977. [KS83] John G. Kemeny and James Laurie Snell. Finite Markov chains. Springer- Verlag, New York, 1983. [KVAK10] Vijay Anand Korthikanti, Mahesh Viswanathan, Gul Agha, and YoungMin Kwon. Reasoning about MDPs as transformers of probability distributions. In QEST 2010, Seventh International Conference on the Quantitative Evaluation of Systems, Williamsburg, Virginia, USA, 15-18 September 2010, pages 199– 208. IEEE Computer Society, 2010. [Lip76] Richard J. Lipton. The reachability problem requires exponential space. 62, New Haven, Connecticut: Yale University, Department of Computer Science, Research, 1976. [LLS14] Kim Guldstrand Larsen, Simon Laursen, and Jirí Srba. Synchronizing strategies under partial observability. In CONCUR 2014 - Concurrency Theory - 25th International Conference, CONCUR 2014, Rome, Italy, September 2-5, 2014. Proceedings, volume 8704 of Lecture Notes in Computer Science, pages 188–202. Springer, 2014. [Mar90] Donald A. Martin. An extension of borel determinacy. Annals of Pure and Applied Logic, 49(3):279–293, 1990. [Mar98] Donald A. Martin. The determinacy of blackwell games. Journal of Symbolic Logic, 63(4):1565–1581, 1998. [Mar10] Pavel V. Martyugin. Complexity of problems concerning carefully synchronizing words for PFA and directing words for NFA. In Computer Science - Theory and Applications, 5th International Computer Science Symposium in

190 Russia, CSR 2010, Kazan, Russia, June 16-20, 2010. Proceedings, volume 6072 of Lecture Notes in Computer Science, pages 288–302. Springer, 2010. [Mar12] Pavel V. Martyugin. Synchronization of automata with one undefined or am- biguous transition. In Implementation and Application of Automata - 17th International Conference, CIAA 2012, Porto, Portugal, July 17-20, 2012. Pro- ceedings, volume 7381 of Lecture Notes in Computer Science, pages 278–288. Springer, 2012. [Min67] Marvin Lee Minsky. Computation: Finite and Infinite Machines. Prentice Hall, Inc., 1967. [Nor98] James R. Norris. Markov chains. Cambridge series in statistical and probabilistic mathematics. Cambridge University Press, 1998. [OU10] Jörg Olschewski and Michael Ummels. The complexity offinding reset words infinite automata. In Mathematical Foundations of Computer Science 2010, 35th International Symposium, MFCS 2010, Brno, Czech Republic, August 23-27, 2010. Proceedings, volume 6281 of Lecture Notes in Computer Science, pages 568–579. Springer, 2010. [Pin78] Jean-Eric Pin. Sur les mots synthronisants dans un automatefini. Elektron- ische Informationsverarbeitung und Kybernetik, 14(6):297–303, 1978. [Rac78] Charles Rackoff. The covering and boundedness problems for vector addition systems. Theoretical Computer Science, 6:223–231, 1978. [Ros93] Kenneth H. Rosen. Elementary number theory and its applications (3. ed.). Addison-Wesley, 1993. [San04] Sven Sandberg. Homing and synchronizing sequences. In Model-Based Test- ing of Reactive Systems, Advanced Lectures [The volume is the outcome of a research seminar that was held in Schloss Dagstuhl in January 2004], volume 3472 of Lecture Notes in Computer Science, pages 5–33. Springer, 2004. [Ser13] Bruno Sericola. Markov Chains: Theory and Applications. Wiley-ISTE, 2013. [Sip97] Michael Sipser. Introduction to the theory of computation. PWS Publishing Company, 1997. [SVW87] A. Prasad Sistla, Moshe Y. Vardi, and Pierre Wolper. The complementation problem for Büchi automata with appplications to temporal logic. Theoretical Computer Science, 49:217–237, 1987. [TBG09] Mathieu Tracol, Christel Baier, and Marcus Größer. Recurrence and tran- sience for probabilistic automata. In IARCS Annual Conference on Foun- dations of Software Technology and Theoretical Computer Science, FSTTCS 2009, December 15-17, 2009, IIT Kanpur, India, volume 4 of LIPIcs, pages 395–406. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, 2009. [Tra11] A. N. Trahtman. Modifying the upper bound on the length of minimal synchronizing word. In Fundamentals of Computation Theory - 18th International

191 Symposium, FCT 2011, Oslo, Norway, August 22-25, 2011. Proceedings, volume 6914 of Lecture Notes in Computer Science, pages 173–180. Springer, 2011. [Tra14] A. N. Trahtman. The length of a minimal synchronizing word and thečerny conjecture. CoRR, abs/1405.2435, 2014. [Var85] Moshe Y. Vardi. Automatic veriﬁcation of probabilistic concurrentﬁnite-state programs. In 26th Annual Symposium on Foundations of Computer Science, Portland, Oregon, USA, 21-23 October 1985, pages 327–338. IEEE Computer Society, 1985. [Vol08] Mikhail V. Volkov. Synchronizing automata and the Cerny conjecture. In Language and Automata Theory and Applications, Second International Con- ference, LATA 2008, Tarragona, Spain, March 13-19, 2008. Revised Papers, volume 5196 of Lecture Notes in Computer Science, pages 11–27. Springer, 2008.

192