Random Variables and Densities

Review: Random variables X represents outcomes or states of world. • Instantiations of variables usually in lower case: x Probability and Statistics We will write p(x) to mean probability(X = x). Sample Space: the space of all possible outcomes/states. • (May be discrete or continuous or mixed.) Probability mass (density) function p(x) 0 • ≥ Sam Roweis Assigns a non-negative number to each point in sample space. Sums (integrates) to unity: x p(x) = 1 or x p(x)dx = 1. Intuitively: how often does x occur, how much do we believe in x. P R Ensemble: random variable + sample space+ probability function • October 2, 2002

Probability Expectations, Moments

We use probabilities p(x) to represent our beliefs B(x) about the Expectation of a function a(x) is written E[a] or a • states x of the world. • h i There is a formal calculus for manipulating uncertainties E[a] = a = p(x)a(x) • h i x represented by probabilities. X e.g. mean = xp(x), variance = (x E[x])2p(x) Any consistent set of beliefs obeying the Cox Axioms can be x x − • mapped into probabilities. Moments are Pexpectations of higher oPrder powers. • (Mean is first moment. Autocorrelation is second moment.) 1. Rationally ordered degrees of belief: Centralized moments have lower moments subtracted away if B(x) > B(y) and B(y) > B(z) then B(x) > B(z) • 2. Belief in x and its negation x¯ are related: B(x) = f[B(x¯)] (e.g. variance, skew, curtosis). Deep fact: Knowledge of all orders of moments 3. Belief in conjunction depends only on conditionals: • B(x and y) = g[B(x), B(y x)] = g[B(y), B(x y)] completely defines the entire distribution. | | Joint Probability Conditional Probability

Key concept: two or more random variables may interact. If we know that some event has occurred, it changes our belief • Thus, the probability of one taking on a certain value depends on • about the probability of other events. which value(s) the others are taking. This is like taking a ”slice” through the joint table. • We call this a joint ensemble and write p(x y) = p(x, y)/p(y) • p(x, y) = prob(X = x and Y = y) |

z z

p(x,y|z)

p(x,y,z) y

x y

x

Marginal Probabilities Bayes’ Rule

We can ”sum out” part of a joint distribution to get the marginal Manipulating the basic definition of conditional probability gives • distribution of a subset of variables: • one of the most important formulas in probability theory: p(x) = p(x, y) p(y x)p(x) p(y x)p(x) p(x y) = | = | y | p(y) p(y x )p(x ) X x0 | 0 0 This is like adding slices of the table together. This gives us a way of ”reversing”conditional probabilities. • Σ p(x,y) P z • Thus, all joint probabilities can be factored by selecting an ordering z • for the random variables and using the ”chain rule”: y p(x, y, z, . . .) = p(x)p(y x)p(z x, y)p(. . . x, y, z) | | |

y x

x Another equivalent definition: p(x) = p(x y)p(y). • y | P Independence & Conditional Independence Entropy

Two variables are independent iff their joint factors: Measures the amount of ambiguity or uncertainty in a distribution: • • p(x, y) = p(x)p(y) H(p) = p(x) log p(x) p(x,y) − x p(x) X Expected value of log p(x) (a function which depends on p(x)!). x • − = H(p) > 0 unless only one possible outcomein which case H(p) = 0. • Maximal value when p is uniform. • Tells you the expected ”cost” if each event costs log p(event) p(y) • − Two variables are conditionally independent given a third one if for • all values of the conditioning variable, the resulting slice factors: p(x, y z) = p(x z)p(y z) z | | | ∀

Be Careful! Cross Entropy (KL Divergence)

Watch the context: An assymetric measure of the distancebetween two distributions: • e.g. Simpson’s paradox • KL[p q] = p(x)[log p(x) log q(x)] Define random variables and sample spaces carefully: k x − • X e.g. Prisoner’s paradox KL > 0 unless p = q then KL = 0 • Tells you the extra cost if events were generated by p(x) but • instead of charging under p(x) you charged under q(x). Statistics (Conditional) Probability Tables Probability: inferring probabilistic quantities for data given fixed For discrete (categorical) quantities, the most basic parametrization • • th models (e.g. prob. of events, marginals, conditionals, etc). is the probability table which lists p(xi = k value). Statistics: inferring a model given fixed data observations Since PTs must be nonnegative and sum to 1, for k-ary variables • • (e.g. clustering, classification, regression). there are k 1 free parameters. − Many approaches to statistics: If a discrete variable is conditioned on the values of some other • • frequentist, Bayesian, decision theory, ... discrete variables we make one table for each possible setting of the parents: these are called conditional probability tables or CPTs.

z z

p(x,y,z) p(x,y|z)

y y

x x

Some (Conditional) Probability Functions Exponential Family

Probability density functions p(x) (for continuous variables) or For (continuous or discrete) random variable x • • probability mass functions p(x = k) (for discrete variables) tell us p(x η) = h(x) exp η>T (x) A(η) how likely it is to get a particular value for a random variable | 1 { − } = h(x) exp η>T (x) (possibly conditioned on the values of some other variables.) Z(η) { } We can consider various types of variables: binary/discrete • is an exponential family distribution with (categorical), continuous, interval, and integer counts. natural parameter η. For each type we’ll see some basic probability models which are Function T (x) is a sufficient statistic. • parametrized families of distributions. • Function A(η) = log Z(η) is the log normalizer. • Key idea: all you need to know about the data is captured in the • summarizing function T (x). Bernoulli Multinomial

For a binary random variable with p(heads)=π: For a set of integer counts on k trials • • x 1 x p(x π) = π (1 π) − | − k! x1 x2 xn π p(x π) = π1 π2 πn = h(x) exp xi log πi = exp log x + log(1 π) | x1!x2! xn! · · ·   1 π − · · · Xi    −   Exponential family with: But the parameters are constrained: i πi = 1.   • • n 1 π So we define the last one πn = 1 − π . η = log − Pi=1 i 1 π n 1 πi − p(x π) = h(x) exp − logP x + k log πn T (x) = x | i=1 πn i A(η) = log(1 π) = log(1 + eη) Exponential family with: nP   o − − • h(x) = 1 η = log π log πn i i − The logistic function relates the natural parameter and the chance T (xi) = xi • η of heads A(η) = k log πn = k log e i 1 − i π = η h(x) = k!/x1!x2! xn! 1 + e− · · · P

Poisson The softmax function relates the basic and natural parameters: • eηi For an integer count variable with rate λ: π = • i eηj λxe λ j p(x λ) = − | x! P 1 = exp x log λ λ x! { − } Exponential family with: • η = log λ T (x) = x A(η) = λ = eη 1 h(x) = x! e.g. number of photons x that arrive at a pixel during a fixed • interval given mean intensity λ Other count densities: binomial, exponential. • Gaussian (normal) Important Gaussian Facts For a continuous univariate random variable: All marginals of a Gaussian are again Gaussian. • 1 1 • Any conditional of a Gaussian is again Gaussian. p(x µ, σ2) = exp (x µ)2 | √ −2σ2 − 2πσ   1 µx x2 µ2 = exp 2 2 2 log σ √2πσ (σ − 2σ − 2σ − ) p(y|x=x0) Exponential family with: p(x,y) • x0 η = [µ/σ2 ; 1/2σ2] − T (x) = [x ; x2] A(η) = log σ + µ/2σ2 Σ h(x) = 1/√2π

Note: a univariate Gaussian is a two-parameter distribution with a p(x) • two-component vector of sufficient statistis.

Multivariate Gaussian Distribution Gaussian Marginals/Conditionals For a continuous vector random variable: • To find these parameters is mostly linear algebra: 1/2 1 1 • p(x µ, Σ) = 2πΣ exp (x µ)>Σ (x µ) Let z = [x>y>]> be normally distributed according to: | | |− −2 − − −   x a A C z = ; y ∼ N b C> B       where C is the (non-symmetric) cross-covariance matrix between x Exponential family with: y x • and which has as many rows as the size of and as many 1 1 η = [Σ− µ ; 1/2Σ− ] columns as the size of y. − The marginal distributions are: T (x) = [x ; xx>] 1 x (a; A) A(η) = log Σ /2 + µ>Σ− µ/2 ∼ N | |n/2 y (b; B) h(x) = (2π)− ∼ N and the conditional distributions are: Sufficient statistics: mean vector and correlation matrix. 1 1 • x y (a + CB (y b); A CB C>) Other densities: Student-t, Laplacian. | ∼ N − − − − • y x b C A 1 x a B C A 1C For non-negative values use exponential, Gamma, log-normal. ( + > − ( ); > − ) • | ∼ N − − Moments Generalized Linear Models (GLMs)

For continuous variables, moment calculations are important. Generalized Linear Models: p(y x) is exponential family with • • | We can easily compute moments of any exponential family conditional mean µ = f(θ>x). • distribution by taking the derivatives of the log normalizer A(η). The function f is called the response function. • The qth derivative gives the qth centred moment. If we chose f to be the inverse of the mapping b/w conditional • • mean and natural parameters then it is called the canonical dA(η) response function. = mean dη η = ψ(µ) 2 d A(η) 1 = variance f( ) = ψ− ( ) dη2 · · · · · When the sufficient statistic is a vector, partial derivatives need to • be considered.

Parameterizing Conditionals Potential Functions

When the variable(s) being conditioned on (parents) are discrete, We can be even more general and define distributions by arbitrary • • we just have one density for each possible setting of the parents. energy functions proportional to the log probability. e.g. a table of natural parameters in exponential models or a table p(x) exp Hk(x) of tables for discrete models. ∝ {− } Xk When the conditioned variable is continuous, its value sets some of A common choice is to use pairwise terms in the energy: • the parameters for the other variables. • H(x) = aixi + wijxixj A very common instance of this for regression is the • Xi pairsXij “linear-Gaussian”: p(y x) = gauss(θ>x; Σ). | For discrete children and continuous parents, we often use a • Bernoulli/multinomial whose paramters are some function f(θ>x). Special variables Likelihood Function

If certain variables are always observed we may not want to model So far we have focused on the (log) probability function p(x θ) • | • their density. For example inputs in regression or classification. which assigns a probability (density) to any joint configuration of This leads to conditional density estimation. variables x given fixed parameters θ. If certain variables are always unobserved, they are called hidden or But in learning we turn this on its head: we have some fixed data • • latent variables. They can always be marginalized out, but can and we want to find parameters. make the density modeling of the observed variables easier. Think of p(x θ) as a function of θ for fixed x: (We’ll see more on this later.) • | L(θ; x) = p(x θ) | `(θ; x) = log p(x θ) | This function is called the (log) “likelihood”. Chose θ to maximize some cost function c(θ) which includes `(θ): • c(θ) = `(θ; ) maximum likelihood (ML) D c(θ) = `(θ; ) + r(θ) maximum a posteriori (MAP)/penalizedML D (also cross-validation, Bayesian estimators, BIC, AIC, ...)

Multiple Observations, Complete Data, IID Sampling Maximum Likelihood

A single observation of the data X is rarely useful on its own. For IID data: • • Generally we have data including many observations, which creates p( θ) = p(xm θ) • a set of random variables: = x1, x2, . . . , xM D| m | D { } Y m Two very common assumptions: `(θ; ) = log p(x θ) • D m | 1. Observations are independently and identically distributed X Idea of maximum likelihod estimation (MLE): pick the setting of according to joint distribution of graphical model: IID samples. • 2. We observe all random variables in the domain on each parameters most likely to have generated the data we saw: observation: complete data. θ = argmax `(θ; ) ML∗ θ D Very commonly used in statistics. • Often leads to “intuitive”, “appealing”, or “natural” estimators. Example: Bernoulli Trials Example: Univariate Normal

We observe M iid coin flips: =H,H,T,H,. . . We observe M iid real samples: =1.18,-.25,.78,. . . • D • D Model: p(H) = θ p(T ) = (1 θ) Model: p(x) = (2πσ2) 1/2 exp (x µ)2/2σ2 • − • − {− − } Likelihood: Likelihood (using probability density): • • `(θ; ) = log p( θ) `(θ; ) = log p( θ) D D|xm 1 xm D D| = log θ (1 θ) − M 1 (xm µ)2 − = log(2πσ2) − m 2 2 2 Y − − m σ = log θ xm + log(1 θ) (1 xm) X − − Take derivatives and set to zero: m m • X X ∂` 2 = log θNH + log(1 θ)NT = (1/σ ) (xm µ) − ∂µ m − Take derivatives and set to zero: ∂` = M + 1 (x µ)2 • ∂σ2 2σ2 P2σ4 m m ∂` NH NT − − = µ = (1/M) xm ∂θ θ − 1 θ ⇒ ML m P N − σ2 = (1/M) x2 µ2 θ = H ML Pm m − ML ML∗ N + N ⇒ H T P

Example: Multinomial Example: Univariate Normal

We observe M iid die rolls (K-sided): =3,1,K,2,. . . • D Model: p(k) = θk k θk = 1 • m Likelihood (for binaryPindicators [x = k]): • B `(θ; ) = log p( θ) D D| [xm=1] [xm=k] = log θxm = log θ1 . . . θk A m m Y mY = log θk [x = k] = Nk log θk m Xk X Xk Take derivatives and set to zero (enforcing θ = 1): • k k ∂` N = k M P ∂θk θk − N x θ = k ⇒ k∗ M Example: Linear Regression Sufficient Statistics

In linear regression, some inputs (covariates,parents) and all A statistic is a function of a random variable. • • outputs (responses,children) are continuous valued variables. T (X) is a “sufficient statistic” for X if • For each child and setting of discrete parents we use the model: 1 2 1 2 • T (x ) = T (x ) L(θ; x ) = L(θ; x ) θ 2 ⇒ ∀ p(y x, θ) = gauss(y θ>x, σ ) | | Equivalently (by the Neyman factorization theorem) we can write: The likelihood is the familiar “squared error” cost: • • p(x θ) = h (x, T (x)) g (T (x), θ) 1 | m xm 2 `(θ; ) = 2 (y θ> ) Example: exponential family models: D −2σ m − • X p(x θ) = h(x) exp η>T (x) A(η) The ML parameters can be solved for using linear least-squares: | { − } • ∂` m m m = (y θ>x )x ∂θ − m − X 1 θ = (X>X) X>Y ⇒ ML∗ −

Example: Linear Regression Sufficient Statistics are Sums

In the examples above, the sufficient statistics were merely sums • (counts) of the data: Bernoulli: # of heads, tails y X Multinomial: # of each type x Gaussian: mean, mean-square x x Regression: correlations x x x x x x x As we will see, this is true for all exponential family models: x x x x x • x x sufficient statistics are average natural parameters. x x x x Only exponential family models have simple sufficient statistics. x • x

Y x MLE for Exponential Family Models Fundamental Operations with Distributions

Recall the probability function for exponential models: Generate data: draw samples from the distribution. This often • • involves generating a uniformly distributed variable in the range p(x θ) = h(x) exp η>T (x) A(η) | { − } [0,1] and transforming it. For more complex distributions it may For iid data, sufficient statistic is T (xm): • m involve an iterative procedure that takes a long time to produce a single sample (e.g. Gibbs sampling, MCMC). P m m `(η; ) = log p( η) = log h(x ) MA(η)+ η> T (x ) D D| − Compute log probabilities. m ! m ! • X X When all variables are either observed or marginalized the result is a Take derivatives and set to zero: • single number which is the log prob of the configuration. ∂` m ∂A(η) ∂η = m T (x ) M ∂η Inference: Compute expectations of some variables given others − • ∂A(η) 1 m which are observed or marginalized. ∂η = PM m T (x ) ⇒ Learning. η = 1 T (xm) ML M Pm • Set the parameters of the density functions given some (partially) recalling that the natural momentsP of an exponential distribution observed data to maximize likelihood or penalized likelihood. are the derivatives of the log normalizer.

Basic Statistical Problems Learning

Let’s remind ourselves of the basic problems we discussed on the In AI the bottleneck is often knowledge acquisition. • • first day: density estimation, clustering classification and regression. Human experts are rare, expensive, unreliable, slow. • Density estimation is hardest. If we can do joint density estimation But we have lots of data. • then we can always condition to get what we want: • Want to build systems automatically based on data and a small Regression: p(y x) = p(y, x)/p(x) • amount of prior information (from experts). Classification: p|(c x) = p(c, x)/p(x) Clustering: p(c x)|= p(c, x)/p(x) c unobserved | Known Models

Many systems we build will be essentially probability models. • Assume the prior information we have specifies type & structure of • the model, as well as the form of the (conditional) distributions or potentials. In this case learning setting parameters. • ≡ Also possible to do “structure learning” to learn model. •

Jensen’s Inequality

For any concave function f() and any distribution on x, • E[f(x)] f(E[x]) ≤ f(E[x])

E[f(x)]

e.g. log() and √ are concave • This allows us to bound expressions like log p(x) = log p(x, z) • z P