Probabilities and Partial Observation from Probabilistic Omega-Automata to Stochastic Games of Imperfect Information

Home , Determinacy, Monty Hall problem

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Probabilities and partial observation From probabilistic omega-automata to stochastic games of imperfect information

Nathalie Bertrand

Inria Rennes + Liverpool visitor

based on joint work with Christel Baier & Marcus Großer¨ Blaise Genest & Hugo Gimbert

Probabilities and partial observation OUCL Veriﬁcation seminar 1 Monty Hall problem

Ethernet protocol: random choice slot, collisions, maximum nb of trials

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Motivation

Adversarial situations with I probabilities: random choices, uncertainties, losses I partial observation: distributed

Probabilities and partial observation OUCL Veriﬁcation seminar 2 Ethernet protocol: random choice slot, collisions, maximum nb of trials

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Motivation

Adversarial situations with I probabilities: random choices, uncertainties, losses I partial observation: distributed

Monty Hall problem

Probabilities and partial observation OUCL Veriﬁcation seminar 2 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Motivation

Adversarial situations with I probabilities: random choices, uncertainties, losses I partial observation: distributed

Monty Hall problem

Ethernet protocol: random choice slot, collisions, maximum nb of trials

Probabilities and partial observation OUCL Veriﬁcation seminar 2 Problems for POMDPs Given a POMDP ( , ), an LTL formula ϕ and p [0, 1] M ∼ ∈ Question is there a based scheduler with P (ϕ) ./ p? ∼ U U

State of the art (back in 2008) I based, P (F) > 0 is EXPTIME-Comp. [de Alfaro 99] ∃U ∼ U I based, P (^F) ./ p is undecidable [Giro D’Argenio 07] ∃U ∼ (p U(0, 1) and ./ , , = ) ∈ ∈ {≤ ≥ }

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Partially observable MDP

Partially Observable MDP A POMDP ( , ) consists of an MDP equipped with an equivalence relationM ∼over states of . M ∼ M

Probabilities and partial observation OUCL Veriﬁcation seminar 3 State of the art (back in 2008) I based, P (F) > 0 is EXPTIME-Comp. [de Alfaro 99] ∃U ∼ U I based, P (^F) ./ p is undecidable [Giro D’Argenio 07] ∃U ∼ (p U(0, 1) and ./ , , = ) ∈ ∈ {≤ ≥ }

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Partially observable MDP

Partially Observable MDP A POMDP ( , ) consists of an MDP equipped with an equivalence relationM ∼over states of . M ∼ M Problems for POMDPs Given a POMDP ( , ), an LTL formula ϕ and p [0, 1] M ∼ ∈ Question is there a based scheduler with P (ϕ) ./ p? ∼ U U

Probabilities and partial observation OUCL Veriﬁcation seminar 3 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Partially observable MDP

Probabilities and partial observation OUCL Veriﬁcation seminar 3 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Outline

1 Introduction

2 Probabilistic ω-automata Examples and expressiveness Emptiness problem Positive semantics (PBA>0) Almost-sure semantics (PBA=1)

3 Stochastic games of imperfect information Framework Qualitative determinacy Memory requirements Decidability

4 Conclusion

Probabilities and partial observation OUCL Veriﬁcation seminar 4 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Probabilistic Buchi-automata¨ [Baier Großer¨ 05]

Probabilistic Buchi¨ Automata NBA where nondeterminism is resolved by probabilities

ω >0( ) = w Σ P( ρ Runs(w) ρ = ^F ) > 0 L A { ∈ ω | { ∈ | | } } =1( ) = w Σ P( ρ Runs(w) ρ = ^F ) = 1 L A { ∈ | { ∈ | | } }

Probabilities and partial observation OUCL Veriﬁcation seminar 5 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Probabilistic Buchi-automata¨ [Baier Großer¨ 05]

Probabilistic Buchi¨ Automata NBA where nondeterminism is resolved by probabilities

ω >0( ) = w Σ P( ρ Runs(w) ρ = ^F ) > 0 L A { ∈ ω | { ∈ | | } } =1( ) = w Σ P( ρ Runs(w) ρ = ^F ) = 1 L A { ∈ | { ∈ | | } }

PBA are POMDP with trivial equivalence relation: p, q p q. ∀ ∼

word for PBA deterministic blind scheduler for MDP ≡

Probabilities and partial observation OUCL Veriﬁcation seminar 5 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Probabilistic Buchi-automata¨ [Baier Großer¨ 05]

Probabilistic Buchi¨ Automata NBA where nondeterminism is resolved by probabilities

ω >0( ) = w Σ P( ρ Runs(w) ρ = ^F ) > 0 L A { ∈ ω | { ∈ | | } } =1( ) = w Σ P( ρ Runs(w) ρ = ^F ) = 1 L A { ∈ | { ∈ | | } }

Example 1

ω >0 = (a + b)∗a = NBA L ω L =1 = b∗a L

Probabilities and partial observation OUCL Veriﬁcation seminar 5 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Probabilistic Buchi-automata¨ [Baier Großer¨ 05]

Probabilistic Buchi¨ Automata NBA where nondeterminism is resolved by probabilities

ω >0( ) = w Σ P( ρ Runs(w) ρ = ^F ) > 0 L A { ∈ ω | { ∈ | | } } =1( ) = w Σ P( ρ Runs(w) ρ = ^F ) = 1 L A { ∈ | { ∈ | | } }

Example 2

ω NBA = ((ac)∗(ab)) L ω >0 = (ab + ac)∗(ab) L ω =1 = (ab) L

Probabilities and partial observation OUCL Veriﬁcation seminar 5 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Probabilistic Buchi-automata¨ [Baier Großer¨ 05]

Probabilistic Buchi¨ Automata NBA where nondeterminism is resolved by probabilities

ω >0( ) = w Σ P( ρ Runs(w) ρ = ^F ) > 0 L A { ∈ ω | { ∈ | | } } =1( ) = w Σ P( ρ Runs(w) ρ = ^F ) = 1 L A { ∈ | { ∈ | | } }

Example 3

ω NBA = (ab + ac) L 0 = =1 = L> L ∅

Probabilities and partial observation OUCL Veriﬁcation seminar 5 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Expressiveness: PBA vs NBA

Expressiveness

PBA>0 are more expressive than NBA. PBA=1 and NBA are incomparable.

Probabilities and partial observation OUCL Veriﬁcation seminar 6 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Expressiveness: PBA vs NBA

Expressiveness

PBA>0 are more expressive than NBA. PBA=1 and NBA are incomparable.

I any NBA can be turned into an equivalent PBA ﬁrst turn NBA into an equivalent one deterministic in the limit → I example of a PBA>0 whose language is not ω-regular

Probabilities and partial observation OUCL Veriﬁcation seminar 6 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Expressiveness: PBA vs NBA

Expressiveness

PBA>0 are more expressive than NBA. PBA=1 and NBA are incomparable.

Probabilities and partial observation OUCL Veriﬁcation seminar 6 Lemma For 0 < λ < µ < 1, ( ) ) ( ). L Pλ L Pµ

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Probabilities matter

Y ( ) = ak1 bak2 b (1 λki ) > 0) L Pλ { · · · | − } i

Probabilities and partial observation OUCL Veriﬁcation seminar 7 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Probabilities matter

Y ( ) = ak1 bak2 b (1 λki ) > 0) L Pλ { · · · | − } i

Lemma For 0 < λ < µ < 1, ( ) ) ( ). L Pλ L Pµ

Probabilities and partial observation OUCL Veriﬁcation seminar 7 Proof sketch Reduction of the modiﬁed emptiness problem for PFA    w P (w) ε or PFA with ∀ R ≤ R  w P (w) > 1 ε ∃ R −

1 and↓ 2 PBA s.t. P P >ε ( ) = ( 1) ( 2) = L R ∅ ⇔ L P ∩ L P ∅

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Emptiness for PBA>0 [Baier B. Großer¨ 08]

Theorem The emptiness problem is undecidable for PBA.

Probabilities and partial observation OUCL Veriﬁcation seminar 8 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Emptiness for PBA>0 [Baier B. Großer¨ 08]

Theorem The emptiness problem is undecidable for PBA.

Proof sketch Reduction of the modiﬁed emptiness problem for PFA    w P (w) ε or PFA with ∀ R ≤ R  w P (w) > 1 ε ∃ R −

1 and↓ 2 PBA s.t. P P >ε ( ) = ( 1) ( 2) = L R ∅ ⇔ L P ∩ L P ∅

Probabilities and partial observation OUCL Veriﬁcation seminar 8 First undecidability results in qualitative veriﬁcation of POMDP.

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Consequences for POMDP

Undecidability results for POMDP The following problems are undecidable I Given ( , ) and F set of states of , is there a deterministic M ∼ M based such that P (^F) > 0. ∼ U U I Given ( , ) and F set of states of , is there a deterministic M ∼ M based such that P (^F) = 1. ∼ U U

Probabilities and partial observation OUCL Veriﬁcation seminar 9 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Consequences for POMDP

First undecidability results in qualitative veriﬁcation of POMDP.

Probabilities and partial observation OUCL Veriﬁcation seminar 9 Proof sketch 1. emptiness problem and almost-sure reachability are interreducible for PBA

w, Pw (^F) = 1 w, Pw (^M) = 1 ∃ P ≡ ∃ P

2. almost-sure reachability for POMDP is decidable

based, PU (^M) = 1 ∃U ∼ M

Skip proof

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Emptiness for PBA=1

Theorem The emptiness problem is decidable for almost-sure PBA.

Probabilities and partial observation OUCL Veriﬁcation seminar 10 2. almost-sure reachability for POMDP is decidable

based, PU (^M) = 1 ∃U ∼ M

Skip proof

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Emptiness for PBA=1

Theorem The emptiness problem is decidable for almost-sure PBA.

Proof sketch 1. emptiness problem and almost-sure reachability are interreducible for PBA

w, Pw (^F) = 1 w, Pw (^M) = 1 ∃ P ≡ ∃ P

Probabilities and partial observation OUCL Veriﬁcation seminar 10 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Emptiness for PBA=1

Theorem The emptiness problem is decidable for almost-sure PBA.

Proof sketch 1. emptiness problem and almost-sure reachability are interreducible for PBA

w, Pw (^F) = 1 w, Pw (^M) = 1 ∃ P ≡ ∃ P

2. almost-sure reachability for POMDP is decidable

based, PU (^M) = 1 ∃U ∼ M

Skip proof

Probabilities and partial observation OUCL Veriﬁcation seminar 10 I w, Pw (^F) = 1 w, Pw (^M) = 1 ∃ P ∃ P

w, Pw (^F) = 1 Pw (^M) = 1 ∀ P ⇐⇒ P0

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Proof in more details: step 1

I w, Pw (^M) = 1 w, Pw (^F) = 1 ∃ P ∃ P Hint: F = M and add self loops on F with probability one

Probabilities and partial observation OUCL Veriﬁcation seminar 11 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Proof in more details: step 1

I w, Pw (^M) = 1 w, Pw (^F) = 1 ∃ P ∃ P Hint: F = M and add self loops on F with probability one

I w, Pw (^F) = 1 w, Pw (^M) = 1 ∃ P ∃ P

w, Pw (^F) = 1 Pw (^M) = 1 ∀ P ⇐⇒ P0 Probabilities and partial observation OUCL Veriﬁcation seminar 11 Idea: reduction to almost-sure reachability for MDP

From POMDP ( , ) build MDP 0 by powerset construction with M ∼ M additional ﬁnal state F 0. I if δ(r, a) F = : traditional powerset construction ∩ ∅ I if δ(r, a) F , : go to F 0 with probability 1/2, rest of probability mass uniformely∩ ∅ distributed over non ﬁnal successors

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Proof in more details: step 2

Theorem Given ( , ) and F set of states of , it is decidable whether there M ∼ M exists an observation-based with P (^M) = 1. U U

Probabilities and partial observation OUCL Veriﬁcation seminar 12 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Proof in more details: step 2

Theorem Given ( , ) and F set of states of , it is decidable whether there M ∼ M exists an observation-based with P (^M) = 1. U U Idea: reduction to almost-sure reachability for MDP

Probabilities and partial observation OUCL Veriﬁcation seminar 12 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Outline

1 Introduction

2 Probabilistic ω-automata Examples and expressiveness Emptiness problem Positive semantics (PBA>0) Almost-sure semantics (PBA=1)

3 Stochastic games of imperfect information Framework Qualitative determinacy Memory requirements Decidability

4 Conclusion

Probabilities and partial observation OUCL Veriﬁcation seminar 13 Strategy for Player i Based on initial distribution and sequence of signals received so far, Player i chooses a distribution over actions.

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Framework [B. Genest Gimbert 09]

I two-player game I partial observation on both sides signals received by the players → I probability on next state given current one and players’ decisions I qualitative objectives

Probabilities and partial observation OUCL Veriﬁcation seminar 14 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Framework [B. Genest Gimbert 09]

I two-player game I partial observation on both sides signals received by the players → I probability on next state given current one and players’ decisions I qualitative objectives

Strategy for Player i Based on initial distribution and sequence of signals received so far, Player i chooses a distribution over actions.

Probabilities and partial observation OUCL Veriﬁcation seminar 14 ϕ: objective of the game for Player 1 (e.g. reachability, Buchi).¨ I From δ, Player 1 wins almost-surely if σ τ Pδ (ϕ) = 1. ∃ ∀ σ,τ I From δ, Player 1 wins positively if σ τ Pδ (ϕ) > 0. ∃ ∀ σ,τ I From δ, Player 2 wins almost-surely if τ σ Pδ (ϕ) = 0. ∃ ∀ σ,τ I From δ, Player 2 wins positively if τ σ Pδ (ϕ) < 1. ∃ ∀ σ,τ

Player i a.-s. or pos. winning from δ only depends on support: X Pδ (ϕ) = δ(s) P1s (ϕ) σ,τ · σ,τ s S ∈

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Winning almost-surely and positively

Initial distribution δ, and strategy proﬁle (σ, τ) induce a probability measure Pδ ( ) on maximal plays. σ,τ ·

Probabilities and partial observation OUCL Veriﬁcation seminar 15 Player i a.-s. or pos. winning from δ only depends on support: X Pδ (ϕ) = δ(s) P1s (ϕ) σ,τ · σ,τ s S ∈

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Winning almost-surely and positively

Initial distribution δ, and strategy proﬁle (σ, τ) induce a probability measure Pδ ( ) on maximal plays. σ,τ ·

ϕ: objective of the game for Player 1 (e.g. reachability, Buchi).¨ I From δ, Player 1 wins almost-surely if σ τ Pδ (ϕ) = 1. ∃ ∀ σ,τ I From δ, Player 1 wins positively if σ τ Pδ (ϕ) > 0. ∃ ∀ σ,τ I From δ, Player 2 wins almost-surely if τ σ Pδ (ϕ) = 0. ∃ ∀ σ,τ I From δ, Player 2 wins positively if τ σ Pδ (ϕ) < 1. ∃ ∀ σ,τ

Probabilities and partial observation OUCL Veriﬁcation seminar 15 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Winning almost-surely and positively

Initial distribution δ, and strategy proﬁle (σ, τ) induce a probability measure Pδ ( ) on maximal plays. σ,τ ·

Player i a.-s. or pos. winning from δ only depends on support: X Pδ (ϕ) = δ(s) P1s (ϕ) σ,τ · σ,τ s S ∈

Probabilities and partial observation OUCL Veriﬁcation seminar 15 Initial support: 1, 2 . Objective: reach{ t } Player 2 wins positively.

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Examples

Initial support: 1, 2 . Objective: reach{ t } Player 1 wins almost-surely.

Probabilities and partial observation OUCL Veriﬁcation seminar 16 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Examples

Initial support: 1, 2 . Objective: reach{ t } Player 1 wins almost-surely.

Initial support: 1, 2 . Objective: reach{ t } Player 2 wins positively.

Probabilities and partial observation OUCL Veriﬁcation seminar 16 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Consequence of undecidability of PBA>0

Undecidability Buchi¨ games with positive probability (and co-Buchi¨ games with probability one) are undecidable.

I 2-player Buchi¨ games with positive probability generalize PBA>0 I randomness for free in POMDP [Chatterjee Doyen Gimbert Henzinger 10]

Probabilities and partial observation OUCL Veriﬁcation seminar 17 Implies qualitative determinacy for reachability objectives as well.

NB: co-Buchi¨ games are not qualitatively determined. Details

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Determinacy

Qualitative determinacy In Buchi¨ games, every initial distribution is I almost-surely winning for Player 1, or I positively winning for Player 2.

Probabilities and partial observation OUCL Veriﬁcation seminar 18 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Determinacy

Qualitative determinacy In Buchi¨ games, every initial distribution is I almost-surely winning for Player 1, or I positively winning for Player 2.

Implies qualitative determinacy for reachability objectives as well.

NB: co-Buchi¨ games are not qualitatively determined. Details

Probabilities and partial observation OUCL Veriﬁcation seminar 18 (S): set of supports where Player 2 does not win positively L ⊆ P I for every L , Player 1 has at least one safe action, i.e. can stay in ∈ L L I σ strategy for Player 1 that, from L selects (uniformly at random) a safe set of actions for L ∈and L plays uniform distribution I σ ensures to stay in I σ is almost-surely winningL

I 1s M = s S τ Pσ,τ( F) = 1 I { (S∈ M|) ∃ ¬ } L ⊆ P \ 1 I N N s S M τ P s (^F) > 1/N ∃ ∈ ∀ ∈ \ ∀ σ,τ Player 1 wins almost-surely from L , with belief-based strategy. ⊆ L

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Determinacy: Proof sketch

Belief: possible states of the game according to signals received

Probabilities and partial observation OUCL Veriﬁcation seminar 19 I for every L , Player 1 has at least one safe action, i.e. can stay in ∈ L L I σ strategy for Player 1 that, from L selects (uniformly at random) a safe set of actions for L ∈and L plays uniform distribution I σ ensures to stay in I σ is almost-surely winningL

I 1s M = s S τ Pσ,τ( F) = 1 I { (S∈ M|) ∃ ¬ } L ⊆ P \ 1 I N N s S M τ P s (^F) > 1/N ∃ ∈ ∀ ∈ \ ∀ σ,τ Player 1 wins almost-surely from L , with belief-based strategy. ⊆ L

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Determinacy: Proof sketch

Belief: possible states of the game according to signals received

(S): set of supports where Player 2 does not win positively L ⊆ P

Probabilities and partial observation OUCL Veriﬁcation seminar 19 I σ ensures to stay in I σ is almost-surely winningL

I 1s M = s S τ Pσ,τ( F) = 1 I { (S∈ M|) ∃ ¬ } L ⊆ P \ 1 I N N s S M τ P s (^F) > 1/N ∃ ∈ ∀ ∈ \ ∀ σ,τ Player 1 wins almost-surely from L , with belief-based strategy. ⊆ L

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Determinacy: Proof sketch

Belief: possible states of the game according to signals received

(S): set of supports where Player 2 does not win positively L ⊆ P I for every L , Player 1 has at least one safe action, i.e. can stay in ∈ L L I σ strategy for Player 1 that, from L selects (uniformly at random) a safe set of actions for L ∈and L plays uniform distribution

Probabilities and partial observation OUCL Veriﬁcation seminar 19 I 1s M = s S τ Pσ,τ( F) = 1 I { (S∈ M|) ∃ ¬ } L ⊆ P \ 1 I N N s S M τ P s (^F) > 1/N ∃ ∈ ∀ ∈ \ ∀ σ,τ Player 1 wins almost-surely from L , with belief-based strategy. ⊆ L

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Determinacy: Proof sketch

Belief: possible states of the game according to signals received

Probabilities and partial observation OUCL Veriﬁcation seminar 19 I (S M) L ⊆ P \ 1 I N N s S M τ P s (^F) > 1/N ∃ ∈ ∀ ∈ \ ∀ σ,τ Player 1 wins almost-surely from L , with belief-based strategy. ⊆ L

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Determinacy: Proof sketch

Belief: possible states of the game according to signals received

Probabilities and partial observation OUCL Veriﬁcation seminar 19 1 I N N s S M τ P s (^F) > 1/N ∃ ∈ ∀ ∈ \ ∀ σ,τ Player 1 wins almost-surely from L , with belief-based strategy. ⊆ L

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Determinacy: Proof sketch

Belief: possible states of the game according to signals received

I 1s M = s S τ Pσ,τ( F) = 1 I { (S∈ M|) ∃ ¬ } L ⊆ P \

Probabilities and partial observation OUCL Veriﬁcation seminar 19 Player 1 wins almost-surely from L , with belief-based strategy. ⊆ L

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Determinacy: Proof sketch

Belief: possible states of the game according to signals received

I 1s M = s S τ Pσ,τ( F) = 1 I { (S∈ M|) ∃ ¬ } L ⊆ P \ 1 I N N s S M τ P s (^F) > 1/N ∃ ∈ ∀ ∈ \ ∀ σ,τ

Probabilities and partial observation OUCL Veriﬁcation seminar 19 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Determinacy: Proof sketch

Belief: possible states of the game according to signals received

I 1s M = s S τ Pσ,τ( F) = 1 I { (S∈ M|) ∃ ¬ } L ⊆ P \ 1 I N N s S M τ P s (^F) > 1/N ∃ ∈ ∀ ∈ \ ∀ σ,τ Player 1 wins almost-surely from L , with belief-based strategy. ⊆ L

Probabilities and partial observation OUCL Veriﬁcation seminar 19 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Memory requirements

Player 1 needs exponential memory to win almost-surely. has to remember belief states →

Player 2 needs doubly exponential memory to win positively. has to remember possible beliefs of Player 1 →(beliefs of beliefs)

Probabilities and partial observation OUCL Veriﬁcation seminar 20 I Player 1 better informed than Player 2: 2EXPTIME-complete I Player 2 better informed than Player 1: EXPTIME-complete I Player 1 perfectly informed: EXPTIME-complete

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Deciding stochastic games with signals

Decidability and complexity Deciding whether Player 1 almost-surely wins a reachability or Buchi¨ game is 2EXPTIME-complete.

Probabilities and partial observation OUCL Veriﬁcation seminar 21 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Deciding stochastic games with signals

Decidability and complexity Deciding whether Player 1 almost-surely wins a reachability or Buchi¨ game is 2EXPTIME-complete.

I Player 1 better informed than Player 2: 2EXPTIME-complete I Player 2 better informed than Player 1: EXPTIME-complete I Player 1 perfectly informed: EXPTIME-complete

Probabilities and partial observation OUCL Veriﬁcation seminar 21 Baier, B. and Großer.¨ Probabilistic ω-automata. Journal of the ACM, 2012. B., Genest and Gimbert. Qualitative determinacy and decidability of stochastic games with signals. Proceedings of LICS, 2009.

Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Concluding remarks

More results on probabilistic ω-automata in [Baier B. Großer¨ 12]

Results on stochastic games with signals very dependent on the precise framework. E.g. for deterministic strategies or deterministic memory updates, the memory size for Player 1 may be a tower of exponential [Chatterjee Doyen]

Probabilities and partial observation OUCL Veriﬁcation seminar 22 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Concluding remarks

More results on probabilistic ω-automata in [Baier B. Großer¨ 12]

Baier, B. and Großer.¨ Probabilistic ω-automata. Journal of the ACM, 2012. B., Genest and Gimbert. Qualitative determinacy and decidability of stochastic games with signals. Proceedings of LICS, 2009.

Probabilities and partial observation OUCL Veriﬁcation seminar 22 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion

Detour: co-Buchi¨ games

Co-Buchi¨ games are not qualitatively determined.

Initial state: t Player 1 perfectly informed Player 2 blind

Objective: avoid t from some point on

I Player 1 has no almost-surely winning strategy I Player 2 has no positively winning strategy

Back to main

Probabilities and partial observation OUCL Veriﬁcation seminar 23