Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Probabilities and partial observation From probabilistic omega-automata to stochastic games of imperfect information Nathalie Bertrand Inria Rennes + Liverpool visitor based on joint work with Christel Baier & Marcus Großer¨ Blaise Genest & Hugo Gimbert Probabilities and partial observation OUCL Verification seminar 1 Monty Hall problem Ethernet protocol: random choice slot, collisions, maximum nb of trials Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Motivation Adversarial situations with I probabilities: random choices, uncertainties, losses I partial observation: distributed Probabilities and partial observation OUCL Verification seminar 2 Ethernet protocol: random choice slot, collisions, maximum nb of trials Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Motivation Adversarial situations with I probabilities: random choices, uncertainties, losses I partial observation: distributed Monty Hall problem Probabilities and partial observation OUCL Verification seminar 2 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Motivation Adversarial situations with I probabilities: random choices, uncertainties, losses I partial observation: distributed Monty Hall problem Ethernet protocol: random choice slot, collisions, maximum nb of trials Probabilities and partial observation OUCL Verification seminar 2 Problems for POMDPs Given a POMDP ( , ), an LTL formula ϕ and p [0, 1] M ∼ ∈ Question is there a based scheduler with P (ϕ) ./ p? ∼ U U State of the art (back in 2008) I based, P (F) > 0 is EXPTIME-Comp. [de Alfaro 99] ∃U ∼ U I based, P (^F) ./ p is undecidable [Giro D’Argenio 07] ∃U ∼ (p U(0, 1) and ./ , , = ) ∈ ∈ {≤ ≥ } Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Partially observable MDP Partially Observable MDP A POMDP ( , ) consists of an MDP equipped with an equiva- lence relationM ∼over states of . M ∼ M Probabilities and partial observation OUCL Verification seminar 3 State of the art (back in 2008) I based, P (F) > 0 is EXPTIME-Comp. [de Alfaro 99] ∃U ∼ U I based, P (^F) ./ p is undecidable [Giro D’Argenio 07] ∃U ∼ (p U(0, 1) and ./ , , = ) ∈ ∈ {≤ ≥ } Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Partially observable MDP Partially Observable MDP A POMDP ( , ) consists of an MDP equipped with an equiva- lence relationM ∼over states of . M ∼ M Problems for POMDPs Given a POMDP ( , ), an LTL formula ϕ and p [0, 1] M ∼ ∈ Question is there a based scheduler with P (ϕ) ./ p? ∼ U U Probabilities and partial observation OUCL Verification seminar 3 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Partially observable MDP Partially Observable MDP A POMDP ( , ) consists of an MDP equipped with an equiva- lence relationM ∼over states of . M ∼ M Problems for POMDPs Given a POMDP ( , ), an LTL formula ϕ and p [0, 1] M ∼ ∈ Question is there a based scheduler with P (ϕ) ./ p? ∼ U U State of the art (back in 2008) I based, P (F) > 0 is EXPTIME-Comp. [de Alfaro 99] ∃U ∼ U I based, P (^F) ./ p is undecidable [Giro D’Argenio 07] ∃U ∼ (p U(0, 1) and ./ , , = ) ∈ ∈ {≤ ≥ } Probabilities and partial observation OUCL Verification seminar 3 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Outline 1 Introduction 2 Probabilistic ω-automata Examples and expressiveness Emptiness problem Positive semantics (PBA>0) Almost-sure semantics (PBA=1) 3 Stochastic games of imperfect information Framework Qualitative determinacy Memory requirements Decidability 4 Conclusion Probabilities and partial observation OUCL Verification seminar 4 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Probabilistic Buchi-automata¨ [Baier Großer¨ 05] Probabilistic Buchi¨ Automata NBA where nondeterminism is resolved by probabilities ω >0( ) = w Σ P( ρ Runs(w) ρ = ^F ) > 0 L A { ∈ ω | { ∈ | | } } =1( ) = w Σ P( ρ Runs(w) ρ = ^F ) = 1 L A { ∈ | { ∈ | | } } Probabilities and partial observation OUCL Verification seminar 5 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Probabilistic Buchi-automata¨ [Baier Großer¨ 05] Probabilistic Buchi¨ Automata NBA where nondeterminism is resolved by probabilities ω >0( ) = w Σ P( ρ Runs(w) ρ = ^F ) > 0 L A { ∈ ω | { ∈ | | } } =1( ) = w Σ P( ρ Runs(w) ρ = ^F ) = 1 L A { ∈ | { ∈ | | } } PBA are POMDP with trivial equivalence relation: p, q p q. ∀ ∼ word for PBA deterministic blind scheduler for MDP ≡ Probabilities and partial observation OUCL Verification seminar 5 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Probabilistic Buchi-automata¨ [Baier Großer¨ 05] Probabilistic Buchi¨ Automata NBA where nondeterminism is resolved by probabilities ω >0( ) = w Σ P( ρ Runs(w) ρ = ^F ) > 0 L A { ∈ ω | { ∈ | | } } =1( ) = w Σ P( ρ Runs(w) ρ = ^F ) = 1 L A { ∈ | { ∈ | | } } Example 1 ω >0 = (a + b)∗a = NBA L ω L =1 = b∗a L Probabilities and partial observation OUCL Verification seminar 5 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Probabilistic Buchi-automata¨ [Baier Großer¨ 05] Probabilistic Buchi¨ Automata NBA where nondeterminism is resolved by probabilities ω >0( ) = w Σ P( ρ Runs(w) ρ = ^F ) > 0 L A { ∈ ω | { ∈ | | } } =1( ) = w Σ P( ρ Runs(w) ρ = ^F ) = 1 L A { ∈ | { ∈ | | } } Example 2 ω NBA = ((ac)∗(ab)) L ω >0 = (ab + ac)∗(ab) L ω =1 = (ab) L Probabilities and partial observation OUCL Verification seminar 5 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Probabilistic Buchi-automata¨ [Baier Großer¨ 05] Probabilistic Buchi¨ Automata NBA where nondeterminism is resolved by probabilities ω >0( ) = w Σ P( ρ Runs(w) ρ = ^F ) > 0 L A { ∈ ω | { ∈ | | } } =1( ) = w Σ P( ρ Runs(w) ρ = ^F ) = 1 L A { ∈ | { ∈ | | } } Example 3 ω NBA = (ab + ac) L 0 = =1 = L> L ∅ Probabilities and partial observation OUCL Verification seminar 5 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Expressiveness: PBA vs NBA Expressiveness PBA>0 are more expressive than NBA. PBA=1 and NBA are incomparable. Probabilities and partial observation OUCL Verification seminar 6 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Expressiveness: PBA vs NBA Expressiveness PBA>0 are more expressive than NBA. PBA=1 and NBA are incomparable. I any NBA can be turned into an equivalent PBA first turn NBA into an equivalent one deterministic in the limit → I example of a PBA>0 whose language is not ω-regular Probabilities and partial observation OUCL Verification seminar 6 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Expressiveness: PBA vs NBA Expressiveness PBA>0 are more expressive than NBA. PBA=1 and NBA are incomparable. Probabilities and partial observation OUCL Verification seminar 6 Lemma For 0 < λ < µ < 1, ( ) ) ( ). L Pλ L Pµ Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Probabilities matter Y ( ) = ak1 bak2 b (1 λki ) > 0) L Pλ { · · · | − } i Probabilities and partial observation OUCL Verification seminar 7 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Probabilities matter Y ( ) = ak1 bak2 b (1 λki ) > 0) L Pλ { · · · | − } i Lemma For 0 < λ < µ < 1, ( ) ) ( ). L Pλ L Pµ Probabilities and partial observation OUCL Verification seminar 7 Proof sketch Reduction of the modified emptiness problem for PFA w P (w) ε or PFA with ∀ R ≤ R w P (w) > 1 ε ∃ R − 1 and↓ 2 PBA s.t. P P >ε ( ) = ( 1) ( 2) = L R ∅ ⇔ L P ∩ L P ∅ Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Emptiness for PBA>0 [Baier B. Großer¨ 08] Theorem The emptiness problem is undecidable for PBA. Probabilities and partial observation OUCL Verification seminar 8 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Emptiness for PBA>0 [Baier B. Großer¨ 08] Theorem The emptiness problem is undecidable for PBA. Proof sketch Reduction of the modified emptiness problem for PFA w P (w) ε or PFA with ∀ R ≤ R w P (w) > 1 ε ∃ R − 1 and↓ 2 PBA s.t. P P >ε ( ) = ( 1) ( 2) = L R ∅ ⇔ L P ∩ L P ∅ Probabilities and partial observation OUCL Verification seminar 8 First undecidability results in qualitative verification of POMDP. Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Consequences for POMDP Undecidability results for POMDP The following problems are undecidable I Given ( , ) and F set of states of , is there a deterministic M ∼ M based such that P (^F) > 0. ∼ U U I Given ( , ) and F set of states of , is there a deterministic M ∼ M based such that P (^F) = 1. ∼ U U Probabilities and partial observation OUCL Verification seminar 9 Introduction Probabilistic ω-automata Stochastic games of imperfect information Conclusion Consequences for POMDP Undecidability results for POMDP The following problems are undecidable I Given ( , ) and F set of states of , is there a deterministic M ∼ M based such that P (^F) > 0. ∼ U U I Given ( , ) and F set of states of , is there a deterministic M ∼ M based such that P (^F) = 1. ∼ U U First undecidability results in qualitative verification of POMDP. Probabilities and partial observation