sign in sign up
<<
Home , Chinese restaurant, Dirichlet process, Markov chain

An elementary derivation of the Chinese process from Sethuraman’s stick-breaking process

Jeffrey W. Miller Harvard University, Department of Biostatistics October 14, 2018

Abstract The process (CRP) and the stick-breaking process are the two most commonly used representations of the . However, the usual proof of the connection between them is indirect, relying on abstract properties of the Dirichlet process that are difficult for nonexperts to verify. This short note provides a direct proof that the stick-breaking process leads to the CRP, without using any theory. We also discuss how the stick-breaking representation arises naturally from the CRP.

1 Introduction

Sethuraman(1994) showed that the Dirichlet process has the following stick-breaking rep- iid Qk−1 resentation: if v1, v2,... ∼ Beta(1, α), πk = vk i=1 (1 − vi) for k = 1, 2,..., and iid θ1, θ2,... ∼ H, then the random discrete measure

∞ X P = πkδθk (1) k=1 is distributed according to the Dirichlet process DP(α, H) with α and base distribution H. This representation has been instrumental in the development of many nonparametric models (MacEachern, 1999, 2000; Hjort, 2000; Ishwaran and Zarepour, 2000; Ishwaran and James, 2001; Griffin and Steel, 2006; Dunson and Park, 2008; Chung and Dunson, 2009; Rodriguez and Dunson, 2011; Broderick et al., 2012), has facilitated the understanding of these models (Favaro et al., 2012; Teh et al., 2007; Thibaux and Jordan, 2007; Paisley et al., 2010), and is central to various inference algorithms (Ishwaran and James, 2001; Blei and Jordan, 2006; Papaspiliopoulos and Roberts, 2008; Walker et al., 2007; Kalli et al., 2011). It is well-known that, as shown by Antoniak(1974), the Dirichlet process induces a distribution on partitions as follows: if P ∼ DP(α, H) where H is nonatomic (i.e., H({θ}) =

1 iid 0 for any θ), x1,..., xn|P ∼ P , and C is the partition of {1, . . . , n} induced by x1,..., xn, then

α|C|Γ(α) Y (C = C) = Γ(|c|). (2) P Γ(α + n) c∈C The sequential sampling process corresponding to this partition distribution is known as the Chinese restaurant process (CRP), or Blackwell–MacQueen urn process. The following key fact is a direct consequence of these two results (Sethuraman’s and iid Antoniak’s): if π = (π1, π2,...) is defined as above, z1,..., zn|π ∼ π, and C is the partition induced by z1,..., zn, then the distribution of C is given by Equation2. This can be seen by noting that when H is nonatomic, the distribution of C is the same as when it is induced by x1,..., xn|P . While this key fact follows directly from the results of Sethuraman and Antoniak, the proofs of their results are rather abstract and are not easy to verify, especially for those without expertise in measure theory. The purpose of this note is to provide a proof of this connection between the CRP and the stick-breaking representation using only elementary, non-measure-theoretic arguments. Our proof is completely self-contained and does not rely on any properties of the Dirichlet process or other theoretical results. Conversely, we also provide a sketch of how the CRP naturally leads to the stick-breaking representation. In previous work, Broderick et al.(2013) used De Finetti’s theorem to provide an elegant derivation of the stick-breaking weights from the CRP. Also, Paisley(2010) showed by ele- mentary calculations that if the base distribution H is a discrete distribution on {1,...,K}, and P is defined by the stick-breaking process as in Equation1, then ( P (1),..., P (K)) ∼ Dirichlet(αH(1), . . . , αH(K)); thus, despite the similar sounding title of the article by Pais- ley(2010), the result shown there is altogether different from what we show here.

2 Main result

We use [n] to denote the set {1, . . . , n}, and N to denote {1, 2, 3,...}. As is standard, we represent a partition of [n] as a set C = {c1, . . . , ct} of nonempty disjoint sets c1, . . . , ct such St that i=1 ci = [n]. Thus, t = |C| is the number of parts in the partition, and |c| is the number of elements in a given part c ∈ C. We say that C is the partition of [n] induced by z1, . . . , zn if it has the property that for any i, j ∈ [n], i and j belong to the same part c ∈ C if and only if zi = zj. We use bold font to denote random variables. Theorem 2.1. Suppose

iid v1, v2,... ∼ Beta(1, α) k−1 Y πk = vk (1 − vi) for k = 1, 2,..., i=1 iid z1,..., zn|π = π ∼ π, that is, P(zi = k | π) = πk,

2 and C is the partition of [n] induced by z1,..., zn. Then

α|C|Γ(α) Y (C = C) = Γ(|c|). P Γ(n + α) c∈C Our proof of the theorem relies on the following lemmas. Let us abbreviate z = n (z1, . . . , zn). Given z ∈ N , let Cz denote the partition [n] induced by z.

Lemma 2.2. For any z ∈ Nn,

m Γ(α)  Y  Y α  P(z = z) = Γ(|c| + 1) Γ(n + α) gk + α c∈Cz k=1

where m = max{z1, . . . , zn} and gk = #{i : zi ≥ k}. The proofs of the lemmas will be given in Section3. We use 1(·) to denote the , that is, 1(E) = 1 if E is true, and 1(E) = 0 otherwise.

Lemma 2.3. For any partition C of [n],

|C| m(z) α X 1 Y α Q = (Cz = C) c∈C |c| n gk(z) + α z∈N k=1

where m(z) = max{z1, . . . , zn} and gk(z) = #{i : zi ≥ k}. Proof of Theorem 2.1. X P(C = C) = P(C = C | z = z)P(z = z) z∈Nn m(z) (a) X Γ(α)  Y  Y α  = 1(Cz = C) Γ(|c| + 1) n Γ(n + α) gk(z) + α z∈N c∈Cz k=1 m(z) Γ(α)  Y  X  Y α  = Γ(|c| + 1) 1(Cz = C) Γ(n + α) gk(z) + α c∈C z∈Nn k=1 |C| (b) Γ(α)  Y  α = Γ(|c| + 1) Γ(n + α) Q |c| c∈C c∈C (c) Γ(α)  Y  = Γ(|c|) α|C| Γ(n + α) c∈C

where (a) is by Lemma 2.2, (b) is by Lemma 2.3, and (c) is since Γ(|c| + 1) = |c|Γ(|c|).

3 3 Proofs of lemmas

Proof of Lemma 2.2. Letting ek = #{i : zi = k}, we have n m Y Y ek P(z = z | π1, . . . , πm) = πzi = πk i=1 k=1 and thus m m  ek Y Qk−1 Y ek fk P(z = z | v1, . . . , vm) = vk i=1 (1 − vi) = vk (1 − vk) k=1 k=1 where fk = #{i : zi > k}. Therefore, Z P(z = z) = P(z = z | v1, . . . , vm)p(v1, . . . , vm)dv1 ··· dvm Z  m  Y ek fk = vk (1 − vk) p(v1) ··· p(vm)dv1 ··· dvm k=1 m Z Y ek fk = vk (1 − vk) p(vk)dvk k=1 m (a) Y = αB(ek + 1, fk + α) k=1 m Y αΓ(ek + 1)Γ(fk + α) = Γ(ek + fk + α + 1) k=1 m (b) Y αΓ(ek + 1)Γ(gk+1 + α) = Γ(gk + α + 1) k=1 m m m (c)  Y  Y α  Y Γ(gk+1 + α) = Γ(ek + 1) gk + α Γ(gk + α) k=1 k=1 k=1 m  Y  Y α  Γ(α) = Γ(|c| + 1) gk + α Γ(n + α) c∈Cz k=1 where step (a) holds since Z B(r + 1, s + α) xr(1 − x)s Beta(x|1, α)dx = = αB(r + 1, s + α), B(1, α)

step (b) since fk = gk+1 and gk = ek + fk, and step (c) since Γ(x + 1) = xΓ(x).

Let St denote the set of t! permutations of [t].

Lemma 3.1. For any n1, . . . , nt ∈ N, X 1 1 = a1(σ) ··· at(σ) n1 ··· nt σ∈St

where ai(σ) = nσi + nσi+1 + ··· + nσt .

4 Proof. Consider an urn containing t balls of various sizes—specifically, suppose the balls are labeled 1, . . . , t and have sizes n1, . . . , nt. Consider the process of sampling without replacement t times from the urn, supposing that the of drawing any given ball is proportional to its size. This defines a distribution on permutations σ ∈ St such that, Pt letting n = i=1 ni,

nσ1 nσ1 p(σ1) = = , n a1(σ)

nσ2 nσ2 p(σ2|σ1) = = , n − nσ1 a2(σ)

nσ3 nσ3 p(σ3|σ1, σ2) = = , n − nσ1 − nσ2 a3(σ)

and so on. Therefore, since nσ1 ··· nσt = n1 ··· nt,

n1 ··· nt p(σ) = p(σ1)p(σ2|σ1) ··· p(σt|σ1, . . . , σt−1) = . (3) a1(σ) ··· at(σ) Since p(σ) is a distribution on S by construction, we have P p(σ) = 1; applying this to t σ∈St Equation3 and dividing both sides by n1 ··· nt gives the result.

Proof of Lemma 2.3. Let t = |C|, and suppose c1, . . . , ct are the parts of C. For σ ∈ St, n define ai(σ) = |cσi | + ··· + |cσt |. For any z ∈ N such that Cz = C, if k1 < ··· < kt are the distinct values taken on by z1, . . . , zn, then

m(z) Y α  α k1  α k2−k1  α kt−kt−1 = ··· gk(z) + α gk (z) + α gk (z) + α gk (z) + α k=1 1 2 t  α d1  α d2  α dt = ··· a1(σ) + α a2(σ) + α at(σ) + α

where di = ki −ki−1, with k0 = 0, and σ is the permutation of [t] such that cσi = {j : zj = ki}. Note that the definition of d = (d1, . . . , dt) and σ sets up a one-to-one correspondence (that n t is, a bijection) between {z ∈ N : Cz = C} and {(σ, d): σ ∈ St, d ∈ N }. Therefore,

m(z) t X Y α X X Y  α di 1(Cz = C) = n gk(z) + α ai(σ) + α z∈N k=1 σ∈St d∈Nt i=1 t X Y X  α di = ai(σ) + α σ∈St i=1 di∈N t (a) X Y α = ai(σ) σ∈St i=1 t t (b) α α = = Qt Q i=1 |ci| c∈C |c| P∞ k where step (a) follows from the geometric series, k=1 x = 1/(1 − x) − 1 for x ∈ [0, 1), and step (b) is by Lemma 3.1.

5 4 Deriving the stick-breaking process from the CRP

We have provided an elementary derivation of the CRP from the stick-breaking process. What about going the other direction? Starting from the CRP, how might one arrive at the stick-breaking representation? Here, we sketch out how the stick-breaking process arises naturally from the CRP. This section should be viewed as a concise exposition of existing re- sults; see Pitman(2006) for reference. In this section only, we appeal to Kingman’s paintbox representation of exchangeable partitions, but otherwise our treatment is self-contained. The CRP is a sequential allocation of customers i = 1, 2,... to tables k = 1, 2,... in which customer 1 sits at table 1, and each successive customer sits at a currently occupied table with probability proportional to the number of customers at that table, or sits at the next unoccupied table with probability proportional to α. The resulting random partition of customers by table is distributed according to Equation2. First, consider table 1. Let yi = 1 if customer i sits at table 1, and yi = 0 otherwise. Then y1 = 1, and y2, y3,... is a two-color P´olya urn process in which yi | y1,..., yi−1 ∼ Pi−1  Bernoulli j=1 yj/(α + i − 1) . Thus,

1 · 2 ··· (s − 1) α(α + 1) ··· (α + n − s − 1) p(y , . . . , y ) = n n 1 n (α + 1)(α + 2) ··· (α + n − 1) B(s , α + n − s ) Z = n n = vsn−1(1 − v)n−sn Beta(v | 1, α)dv, B(1, α)

Pn where sn = i=1 yi. Therefore, the same distribution on y1, y2,... can be generated by iid drawing v ∼ Beta(1, α), then setting y1 = 1 and drawing y2, y3,... |v ∼ Bernoulli(v). 1 Pn Letting v1 = limn→∞ n i=1 yi (the limit exists with probability 1), it follows that v1 ∼ Beta(1, α). Note that v1 is the asymptotic proportion of customers at table 1. Now, consider table k. Given the indices of the subsequence of customers that do not sit at tables 1, . . . , k − 1, the customers in this subsequence sit at table k according to the same urn process as y1, y2,... above, independently of the corresponding urn processes for 1, . . . , k − 1. Thus, of the customers not at tables 1, . . . , k − 1, the proportion at table k converges to a Beta(1, α) , say vk, independent of v1,..., vk−1. Therefore, Qk−1 out of all customers, the proportion at table k converges to πk := vk j=1 (1 − vj) with probability 1. (The preceding urn-based derivation is adapted from Broderick et al., 2013.) Although we have arrived at the stick-breaking process, our derivation is not yet complete because the partition distribution given π as defined above is different than the partition iid distribution induced by assignments z1,..., zn|π ∼ π. To establish that they are equivalent, marginally, we use Kingman’s paintbox representation for exchangeable partitions. By Kingman(1978), there exists a random q = (q1, q2,...) with q1 ≥ q2 ≥ P∞ · · · ≥ 0 and j=1 qj = 1 (with probability 1) such that the random partition induced by iid z1,..., zn|q ∼ q is marginally distributed according to Equation2; for a concise proof, P∞ see Aldous(1985), Prop. 11.9. (Note that in general, j=1 qj < 1 is possible, but not in this case because in the CRP, clusters have probability 0, asymptotically.) Let iid 1 Pn 1 z1, z2,... |q ∼ q and qˆnj = n i=1 (zi = j). With probability 1, for all j ∈ N, qˆnj → qj as n → ∞, by the law of large . Let σ = (σ1, σ2,...) be the permutation of N

6 such that σk is the kth distinct value to appear in z1, z2,...; for reference, σ is called a size-biased permutation. In the CRP terminology, customer i is at table k when zi = σk, so the asymptotic proportion of customers at table k is qσk . Therefore, by the urn derivation Qk−1 above, (qσ1 , qσ2 ,...) is equal in distribution to (π1, π2,...) where πk = vk j=1 (1 − vj) and iid v1, v2,... ∼ Beta(1, α). Finally, note that permuting the entries of q (even via a random permutation that depends on q) does not affect the of the partition iid iid induced by z1,..., zn|q ∼ q. This shows that the partition induced by z1,..., zn|π ∼ π is marginally distributed according to Equation2.

Acknowledgments

Thanks to David Dunson, Garritt Page, Tamara Broderick, and Steve MacEachern for helpful conversations.

References

D. J. Aldous. Exchangeability and Related Topics. In Ecole´ d’Et´ede´ Probabilit´esde Saint- Flour XIII–1983, pages 1–198. Springer, 1985.

C. E. Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems. The Annals of , 2(6):1152–1174, 1974.

D. M. Blei and M. I. Jordan. Variational inference for Dirichlet process mixtures. Bayesian Analysis, 1(1):121–143, 2006.

T. Broderick, M. I. Jordan, and J. Pitman. Beta processes, stick-breaking and power laws. Bayesian Analysis, 7(2):439–476, 2012.

T. Broderick, M. I. Jordan, and J. Pitman. Cluster and feature modeling from combinatorial processes. Statistical Science, 28(3):289–312, 2013.

Y. Chung and D. B. Dunson. Nonparametric Bayes conditional distribution modeling with variable selection. Journal of the American Statistical Association, 104(488), 2009.

D. B. Dunson and J.-H. Park. Kernel stick-breaking processes. Biometrika, 95(2):307–323, 2008.

S. Favaro, A. Lijoi, and I. Pruenster. On the stick-breaking representation of normalized inverse Gaussian priors. Biometrika, 99(3):663–674, 2012.

J. E. Griffin and M. J. Steel. Order-based dependent Dirichlet processes. Journal of the American Statistical Association, 101(473):179–194, 2006.

N. L. Hjort. Bayesian analysis for a generalised Dirichlet process prior. Technical Report, University of Oslo, 2000.

7 H. Ishwaran and L. F. James. methods for stick-breaking priors. Journal of the American Statistical Association, 96(453), 2001.

H. Ishwaran and M. Zarepour. Monte Carlo in approximate Dirichlet and beta two-parameter process hierarchical models. Biometrika, 87(2):371–390, 2000.

M. Kalli, J. E. Griffin, and S. G. Walker. Slice sampling mixture models. Statistics and Computing, 21(1):93–105, 2011.

J. F. C. Kingman. The representation of partition structures. Journal of the Math- ematical Society, 2(18):374–380, 1978.

S. N. MacEachern. Dependent nonparametric processes. In ASA Proceedings of the Section on Bayesian Statistical Science, pages 50–55, 1999.

S. N. MacEachern. Dependent Dirichlet processes. Unpublished manuscript, Department of Statistics, The Ohio State University, 2000.

J. Paisley. A simple proof of the stick-breaking construction of the Dirichlet Process. Tech- nical report, Princeton University, Department of Computer Science, August 2010.

J. W. Paisley, A. K. Zaas, C. W. Woods, G. S. Ginsburg, and L. Carin. A stick-breaking construction of the beta process. In Proceedings of the 27th International Conference on , pages 847–854, 2010.

O. Papaspiliopoulos and G. O. Roberts. Retrospective methods for Dirichlet process hierarchical models. Biometrika, 95(1):169–186, 2008.

J. Pitman. Combinatorial Stochastic Processes. Springer–Verlag, , 2006.

A. Rodriguez and D. B. Dunson. Nonparametric Bayesian models through probit stick- breaking processes. Bayesian Analysis, 6(1), 2011.

J. Sethuraman. A constructive definition of Dirichlet priors. Statistica Sinica, 4:639–650, 1994.

Y. W. Teh, D. G¨or¨ur,and Z. Ghahramani. Stick-breaking construction for the Indian buffet process. In International Conference on Artificial Intelligence and Statistics, pages 556– 563, 2007.

R. Thibaux and M. I. Jordan. Hierarchical beta processes and the Indian buffet process. In International Conference on Artificial Intelligence and Statistics, pages 564–571, 2007.

S. G. Walker, A. Lijoi, and I. Pr¨unster.On rates of convergence for posterior distributions in infinite-dimensional models. The Annals of Statistics, 35(2):738–746, 2007.

8