Explaining Word Embeddings via Disentangled Representation Keng-Te Liao Cheng-Syuan Lee National Taiwan University National Taiwan University
[email protected] [email protected] Zhong-Yu Huang Shou-de Lin National Taiwan University National Taiwan University
[email protected] [email protected] Abstract In this work, we focus on word-level disentangle- ment and introduce an idea of transforming dense Disentangled representations have attracted in- word embeddings such as GloVe (Pennington et al., creasing attention recently. However, how to 2014) or word2vec (Mikolov et al., 2013b) into transfer the desired properties of disentangle- disentangled word embeddings (DWE). The main ment to word representations is unclear. In this work, we propose to transform typical dense feature of our DWE is that it can be segmented word vectors into disentangled embeddings into multiple sub-embeddings or sub-areas as il- featuring improved interpretability via encod- lustrated in Figure1. In the figure, each sub-area ing polysemous semantics separately. We also encodes information relevant to one specific topical found the modular structure of our disentan- factor such as Animal or Location. As an example, gled word embeddings helps generate more ef- we found words similar to “turkey” are “geese”, ficient and effective features for natural lan- “flock” and “goose” in the Animal area, and the guage processing tasks. similar words turn into “Greece”, “Cyprus” and 1 Introduction “Ankara” in the Location area. Disentangled representations are known to repre- sent interpretable factors in separated dimensions. This property can potentially help people under- stand or discover knowledge in the embeddings.