Topic Modeling with Wasserstein Autoencoders
Total Page：16
File Type：pdf, Size：1020Kb
Topic Modeling with Wasserstein Autoencoders Feng Nany, Ran Ding ∗z , Ramesh Nallapatiy, Bing Xiangy Amazon Web Servicesy, Compass Inc.z fnanfen, rnallapa, [email protected], [email protected] Abstract (VAE) (Kingma and Welling, 2013). The key ad- vantage of such neural network based models is We propose a novel neural topic model in the that inference can be carried out easily via a for- Wasserstein autoencoders (WAE) framework. ward pass of the recognition network, without the Unlike existing variational autoencoder based models, we directly enforce Dirichlet prior on need for expensive iterative inference scheme per the latent document-topic vectors. We exploit example as in VB and collapsed Gibbs sampling. the structure of the latent space and apply a Topic models that fall in this framework include suitable kernel in minimizing the Maximum NVDM (Miao et al., 2016), ProdLDA (Srivastava Mean Discrepancy (MMD) to perform distri- and Sutton, 2017) and NTM-R (Ding et al., 2018). bution matching. We discover that MMD per- At a high level, these models consist of an en- forms much better than the Generative Adver- coder network that maps the Bag-of-Words (BoW) sarial Network (GAN) in matching high di- input to a latent document-topic vector and a de- mensional Dirichlet distribution. We further discover that incorporating randomness in the coder network that maps the document-topic vec- encoder output during training leads to signif- tor to a discrete distribution over the words in the icantly more coherent topics. To measure the vocabulary. They are autoencoders in the sense diversity of the produced topics, we propose a that the output of the decoder aims to reconstruct simple topic uniqueness metric. Together with the word distribution of the input BoW representa- the widely used coherence measure NPMI, we tion. Besides the reconstruction loss, VAE-based offer a more wholistic evaluation of topic qual- methods also minimize a KL-divergence term be- ity. Experiments on several real datasets show that our model produces significantly better tween the prior and posterior of the latent vector topics than existing topic models. distributions. Despite their popularity, these VAE- based topic models suffer from several conceptual 1 Introduction and practical challenges. First, the Auto-Encoding Variational Bayes (Kingma and Welling, 2013) Probabilistic topic models (Hoffman et al., 2010) framework of VAE relies on a reparameteriza- have been widely used to explore large collec- tion trick that only works with the “location-scale” tions of documents in an unsupervised manner. family of distributions. Unfortunately, the Dirich- They can discover the underlying themes and or- let distribution, which largely accounted for the ganize the documents accordingly. The most pop- modeling success of LDA, does not belong to this ular probabilistic topic model is the Latent Dirich- family. The Dirichlet prior on the latent document- let Allocation (LDA) (Blei et al., 2003), where the topic vector nicely captures the intuition that a authors developed a variational Bayesian (VB) al- document typically belongs to a sparse subset of gorithm to perform approximate inference; sub- topics. The VAE-basedtopic models have to resort sequently (Griffiths and Steyvers, 2004) proposed to various Gaussian approximations toward this an alternative inference method using collapsed effect. For example, NVDM and NTM-R simply Gibbs sampling. use Gaussian instead of Dirichlet prior; ProdLDA More recently, deep neural networks have been uses Laplace approximation of the Dirichlet dis- successfully used for such probabilistic models tribution in the softmax basis as prior. Second, the with the emergence of variational autoencoders KL divergence term in the VAE objective forces ∗This work was done when the author was with Amazon. posterior distributions for all examples to match 6345 Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6345–6381 Florence, Italy, July 28 - August 2, 2019. c 2019 Association for Computational Linguistics the prior, essentially making the encoder output • We introduce a uniqueness measure to evalu- independent of the input. This leads to the prob- ate topic quality more wholistically. lem commonly known as posterior collapse (He • W-LDA produces significantly better quality et al., 2019). Although various heuristics such as topics than existing topic models in terms of KL-annealing (Bowman et al., 2016) have been topic coherence and uniqueness. proposed to address this problem, they are shown to be ineffective in more complex datasets (Kim • We experiment with both the WAE-GAN and et al., 2018). WAE-MMD variants (Tolstikhin et al., 2017) In this work we leverage the expressive power for distribution matching and demonstrate and efficiency of neural networks and propose a key performance advantage of the latter with novel neural topic model to address the above a carefully chosen kernel, especially in high difficulties. Our neural topic model belongs to dimensional settings. a broader family of Wasserstein autoencoders (WAE) (Tolstikhin et al., 2017). We name our neu- • We discover a novel technique of adding ral topic model W-LDA to emphasize the connec- noise to W-LDA to significantly boost topic tion with WAE. Compared to the VAE-based topic coherence. This technique can potentially be models, our model has a few advantages. First, applied to WAE in general and is of indepen- we encourage the latent document-topic vectors to dent interest. follow the Dirichlet prior directly via distribution 2 Related Work matching, without any Gaussian approximation; by preserving the Dirichlet prior, our model repre- Adversarial Autoencoder (AAE) (Makhzani et al., sents a much more faithful generalization of LDA 2015) was proposed as an alternative to VAE. The to neural network based topic models. Second, main difference is that AAE regularizes the aggre- our model matches the aggregated posterior to the gated posterior to be close to a prior distribution prior. As a result, the latent codes of different ex- whereas VAE regularizes the posterior to be close amples get to stay away from each other, promot- to the prior. Wasserstein autoencoders (WAE) ing a better reconstruction (Tolstikhin et al., 2017). (Tolstikhin et al., 2017) provides justification for We are thus able to avoid the problem of posterior AAE from the Wasserstein distance minimization collapse. point of view. In addition to adversarial training To evaluate the quality of the topics from W- used in AAE, the authors also suggested using LDA and other models, we measure the coherence Maximum Mean Discrepancy (MMD) for distri- of the representative words of the topics using bution matching. Compared to VAE, AAE/WAEs the widely accepted Normalized Pointwise Mu- are shown to produce better quality samples. tual Information (NPMI) (Aletras and Stevenson, AAE has been applied in the task of unaligned 2013) score, which is shown to closely match hu- text style transfer and semi-supervised natural lan- man judgments (Lau et al., 2014). While NPMI guage inference by ARAE (Kim et al., 2017). To captures topic coherence, it is also important that be best of our knowledge, W-LDA is the first the discovered topics are diverse (not repetitive). topic model based on the WAE framework. Re- Yet such a measure has been missing in the topic cently, Adversarial Topic model (ATM) (Wang model literature.1 We therefore propose a sim- et al., 2018) proposes using GAN with Dirich- ple Topic Uniqueness (TU) measure for this pur- let prior to learn topics. The generator takes in pose. Given a set of representative words from all samples from Dirichlet distribution and maps to the topics, the TU score is inversely proportional a document-word distribution layer to form the to the number of times each word is repeated in fake samples. The discriminator tries to distin- the set. High TU score means the representative guish between the real documents from the fake words are rarely repeated and the topics are unique documents. It also pre-processes the BoW repre- to each other. Using both TU and NPMI, we are sentation of documents using TF-IDF. The evalu- able to provide a more wholistic measure of topic ation is limited to topic coherence. A critical dif- quality. To summarize our main contributions: ference of W-LDA and ATM is that ATM tries to perform distribution matching in the vocabulary 1Most papers on topic modeling only present a selected small subset of non-repetitive topics for qualitative evalua- space whereas W-LDA in the latent document- tion. The diversity among the topics is not measured. topic space. Since the vocabulary space has much 6346 higher dimension (size of the vocabulary) than the 3.2 Wasserstein Auto-encoder latent document-topic space (number of topics), The latent variable generative model posits that a we believe it is much more challenging for ATM target domain example (eg. document w) is gener- to train and perform well compared to W-LDA. ated by first sampling a latent code θ from a prior Our work is also related to the topic of learn- distribution PΘ and then passed through a decoder ing disentangled representations. A disentangled network. The resulting distribution in the target representation can be defined as one where sin- domain is Pdec with density: gle latent units are sensitive to changes in sin- Z gle generative factors, while being relatively in- pdec(w) = pdec(wjθ)p(θ)dθ: (1) variant to changes in other factors (Bengio et al., θ 2013). In topic modeling, such disentanglement The key result of (Tolstikhin et al., 2017) is that in means that the learned topics are coherent and order to minimize the optimal transport distance distinct. (Rubenstein et al., 2018) demonstrated between Pdec and the target distribution Pw, it is that WAE learns better disentangled representa- equivalent to minimizing the following objective tion than VAE.