Nonparametric Bayes and Exchangeability: Part II Tamara Broderick Associate Professor Electrical Engineering & Computer Science MIT http://www.tamarabroderick.com/tutorials.html Roadmap Roadmap • Example problem: clustering Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? • What does an infinite/growing number of parameters really mean (in NPBayes)? Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? • What does an infinite/growing number of parameters really mean (in NPBayes)? • Why is NPBayes challenging but practical? Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? • What does an infinite/growing number of parameters really mean (in NPBayes)? • Why is NPBayes challenging but practical? Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? • What does an infinite/growing number of parameters really mean (in NPBayes)? • Why is NPBayes challenging but practical? Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? • What does an infinite/growing number of parameters really mean (in NPBayes)? • Why is NPBayes challenging but practical? • How does thinking about exchangeability help us use NPBayes in practice? Dirichlet process mixture model 1 d G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X iid µ⇤ G n ⇠ 11 Dirichlet process mixture model • Gaussian mixture model 1 d G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X iid µ⇤ G n ⇠ 11 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢ , ⇢ ) Beta(a ,a ) 1 2 ⇠ 1 2 1 2 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X iid • i.e. µ⇤ G n ⇠ 11 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢ ,...,⇢ ) Dir(a ) 1 K ⇠ 1:K 1 2 3 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X iid • i.e. µ⇤ G n ⇠ 11 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … 1 2 3 4 … 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X iid • i.e. µ⇤ G n ⇠ 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X iid • i.e. µ⇤ G n ⇠ 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X iid • i.e. µ⇤ G n ⇠ 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X ⇢2 µ D 2 R iid • i.e. µ⇤ G n ⇠ 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X ⇢2 µ D 2 R iid • i.e. µ⇤ G n ⇠ 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X ⇢2 µ D 2 R iid • i.e. µ⇤ G n ⇠ 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X ⇢2 iid zn Categorical(⇢) ⇠ µ D 2 R iid • i.e. µ⇤ G n ⇠ 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X ⇢2 z iid Categorical(⇢) n ⇠ µ2 D µn⇤ = µzn R iid • i.e. µ⇤ G n ⇠ 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X ⇢2 z iid Categorical(⇢) n ⇠ µ2 D µn⇤ = µzn R iid • i.e. µ⇤ G n ⇠ 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X ⇢2 z iid Categorical(⇢) n ⇠ µ2 D µn⇤ = µzn R iid • i.e. µ⇤ G n ⇠ 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X ⇢2 z iid Categorical(⇢) n ⇠ µ2 D µn⇤ = µzn R iid • i.e. µ⇤ G n ⇠ indep x (µ⇤ , ⌃) n ⇠ N n 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X ⇢2 z iid Categorical(⇢) n ⇠ µ2 D µn⇤ = µzn R iid • i.e. µ⇤ G n ⇠ indep x (µ⇤ , ⌃) n ⇠ N n 12 Dirichlet process mixture model • Gaussian mixture model ⇢ =(⇢1, ⇢2,...) GEM(↵) ⇠ … µ iid (µ , ⌃ ),k =1, 2,... 1 2 3 4 … k ⇠ N 0 0 1 d • i.e. G = ⇢kδµ =DP(↵, (µ0, ⌃0)) k=1 k N X ⇢2 z iid Categorical(⇢) n ⇠ µ2 D µn⇤ = µzn R iid • i.e. µ⇤ G n ⇠ indep x (µ⇤ , ⌃) n ⇠ N n [demo] 12 Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? • What does an infinite/growing number of parameters really mean (in NPBayes)? • Why is NPBayes challenging but practical? • How does thinking about exchangeability help us use NPBayes in practice? Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? • What does an infinite/growing number of parameters really mean (in NPBayes)? • Why is NPBayes challenging but practical? • How does thinking about exchangeability help us use NPBayes in practice? Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? • What does an infinite/growing number of parameters really mean (in NPBayes)? • Why is NPBayes challenging but practical? • How does thinking about exchangeability help us use NPBayes in practice? Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? Learn more from more data • What does an infinite/growing number of parameters really mean (in NPBayes)? • Why is NPBayes challenging but practical? • How does thinking about exchangeability help us use NPBayes in practice? Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? Learn more from more data • What does an infinite/growing number of parameters really mean (in NPBayes)? Components vs clusters; latent vs realized • Why is NPBayes challenging but practical? • How does thinking about exchangeability help us use NPBayes in practice? Roadmap • Example problem: clustering • Example NPBayes model: Dirichlet process (DP) • De Finetti for clustering: Kingman Paintbox • De Finetti for networks/graphs • Big questions • Why NPBayes? Learn more from more data • What does an infinite/growing number of parameters really mean (in NPBayes)? Components vs clusters; latent vs realized • Why is NPBayes challenging but practical? Infinite dimensional parameter but finitely many realized (in practice, e.g., can integrate out or truncate the infinity) • How does thinking about exchangeability help us use NPBayes in practice? Clustering 1 Clustering “Clusters” 1 Clustering Econ Sports 1 Art “Clusters” Clustering Econ Art SportsTech Science Article 1 • Groups: clusters Article 2 Article 3 Article 4 Article 5 Article 6 Article 7 2 Clustering Econ Art SportsTech Science Article 1 • Groups: clusters • Exchangeable Article 2 Article 3 Article 4 Article 5 Article 6 Article 7 2 Clustering 0 1 3 [Kingman 1978] Clustering 0 1 3 [Kingman 1978] Clustering 3 [Kingman 1978] Clustering • Finite example: Dirichlet 3 [Kingman 1978] Clustering • Finite example: Dirichlet • Infinite example: GEM / DP 3 [Kingman 1978] Clustering … • Finite example: Dirichlet • Infinite example: GEM / DP 3 [Kingman 1978] Clustering … • Finite example: Dirichlet • Infinite example: GEM / DP 3 [Kingman 1978] Clustering 1 … • Finite example: Dirichlet • Infinite example: GEM / DP 1 3 [Kingman 1978] Clustering 1 2 … • Finite example: Dirichlet • Infinite example: GEM / DP 1 2 3 [Kingman 1978] Clustering 3 1 2 … • Finite example: Dirichlet • Infinite example: GEM / DP 1 2 3 3 [Kingman 1978] Clustering 3 1 2 4 … • Finite example: Dirichlet • Infinite example: GEM / DP 1 2 3 4 3 [Kingman 1978] Clustering 3 5 1 2 4 … • Finite example: Dirichlet • Infinite example: GEM / DP 1 2 3 4 5 3 [Kingman 1978] Clustering 3 5 1 2 6 4 … • Finite example: Dirichlet • Infinite example: GEM / DP 1 2 3 4 5 6 3 [Kingman 1978] Clustering 3 5 7 1 2 6 4 … • Finite example: Dirichlet • Infinite example: GEM / DP 1 2 3 4 5 6 7 3 [Kingman 1978] Thm (Kingman).