The Logic of Probability Theory

Deriving Bayesian Statistics from Boolean Algebra

William Sipes July 14, 2010 Probability Theory as Extended Logic

• George Boole • The Laws of Thought • Algebraic expression of Aristotelian logical propositions • Full Title: An Investigation of the Laws of Thought on Which are Founded the Mathematical Theories of Logic and Probabilities (1854) Probability Theory as Extended Logic

• Cox and Jaynes • The Algebra of Probable Inference (1961) • Probability Theory: The Logic of Science (2003) • Boolean Logic and Three Desiderata necessitate Bayesian Probability Boolean Algebra

• Finite Field • Commutative ring wrt operations of disjunction and conjunction • Equivalence classes of [0] and [1] (representing FALSE and TRUE) • Foundation of Computer Science Boolean Algebra

Disjunction

Conjunction

Negation Boolean Algebra

Disjunction

Conjunction

Negation

Though there are three distinct operations, it can be shown that any combination of two that includes negation is complete. Boolean Algebra

F F F F F F F T F F T T T F F T F T T T T T T T

F T T F Boolean Algebra

Idempotence

Associativity

Commutativity

De Morgan’s Laws

Double Negation Bayesian Probability

• Conditional Probability • Differs from frequentist approaches • Based on prior distributions • Models can be updated with new information Bayesian Probability

Conjunction: A and B

Conditional: A given B The Desiderata

1) Representation of plausibility with real numbers 2) Qualitative correspondence with common sense 3) Structural Consistency

These uniquely determine the allowable operations of all probabilistic theory.

Assumption of twice differentiability in functions Product Rule

1. Require that any plausibilities obey all of the desiderata simultaneously, unambiguously, and completely 2. Assume that there is a functional relation for conjoined propositions 3. Argue (using the requirements imposed by the desiderata) that there is only one form for this rule 4. Derive the form of the rule using differential equations Product Rule

• Most basic assumption • Only functional form that does not degenerate when tested at “extremes” • Key feature of familiar probability theory Product Rule Product Rule Sum Rule

1. Using the product rule, derive a function that relates propositions and their negations 2. Impose the conditions of the desiderata to derive another functional equation 3. Argue analytically about the functional equation 4. Reduce the functional equation to a differential equation 5. This leaves a functional relation for complementary plausibilities Sum Rule Sum Rule Sum Rule Sum Rule Further Developing the Theory

• Conditional probability and the sum rule gives definition of independence • Product rule can be used to derive Laplace’s definition of probability (frequentist) • Demonstrate agreement with Kolmogorov’s axioms of probability