Self-Discriminative Learning for Unsupervised Document Embedding
Hong-You Chen∗1, Chin-Hua Hu∗1, Leila Wehbe2, Shou-De Lin1 1Department of Computer Science and Information Engineering, National Taiwan University 2Machine Learning Department, Carnegie Mellon University {b03902128, r07922028}@ntu.edu.tw, [email protected], [email protected]
Abstract ingful a document embedding as they do not con- sider inter-document relationships. Unsupervised document representation learn- Traditional document representation models ing is an important task providing pre-trained such as Bag-of-words (BoW) and TF-IDF show features for NLP applications. Unlike most competitive performance in some tasks (Wang and previous work which learn the embedding based on self-prediction of the surface of text, Manning, 2012). However, these models treat we explicitly exploit the inter-document infor- words as flat tokens which may neglect other use- mation and directly model the relations of doc- ful information such as word order and semantic uments in embedding space with a discrimi- distance. This in turn can limit the models effec- native network and a novel objective. Exten- tiveness on more complex tasks that require deeper sive experiments on both small and large pub- level of understanding. Further, BoW models suf- lic datasets show the competitiveness of the fer from high dimensionality and sparsity. This is proposed method. In evaluations on standard document classification, our model has errors likely to prevent them from being used as input that are relatively 5 to 13% lower than state-of- features for downstream NLP tasks. the-art unsupervised embedding models. The Continuous vector representations for docu- reduction in error is even more pronounced in ments are being developed. A successful thread scarce label setting. of work is based on the distributional hypothesis, and use contextual information for context-word 1 Introduction predictions. Similar to Word2Vec (Mikolov et al., 2013), PV (Le and Mikolov, 2014) is optimized by predicting the next words given their contexts Rapid advance in deep methods for natural lan- in a document, but it is conditioned on a unique guage processing has contributed to a growing document vector. Word2Vec-based methods for need for vector representation of documents as in- computing document embeddings achieve state- put features. Applications for such vector repre- of-the-art performance on document embedding. sentations include machine translation (Sutskever Such methods rely on one strong underlying as- et al., 2014), text classification (Dai and Le, 2015), sumption: it is necessary to train the document image captioning (Mao et al., 2015), multi-lingual embedding to optimize the prediction of the tar- document matching (Pham et al., 2015), question get words in the document. In other words, the answering (Rajpurkar et al., 2016), and more. This objective requires the model to learn to predict the work studies unsupervised training for encoders target words in surface text. We argue that there that can efficiently encode long paragraph of text are several concerns with such a self-prediction as- into compact vectors to be used as pre-trained fea- sumption. tures. Existing solutions are mostly based on the The strategy of predicting target words there- assumption that a good document embedding can fore only exploits in-document information, and be learned through modeling the intra-document do not explicitly model the inter-document dis- information by predicting the occurrence of terms tances. We believe an ideal embedding space inside the document itself. We argue that such an should also infer the relations among training doc- assumption might not be sufficient to obtain mean- uments. For example, if all documents in the ∗ Equally contribution. corpus are about machine learning, then the con-
2465 Proceedings of NAACL-HLT 2019, pages 2465–2474 Minneapolis, Minnesota, June 2 - June 7, 2019. c 2019 Association for Computational Linguistics cept of machine learning becomes less critical in of our knowledge, this is the first deep embedding the embedding. However, if the corpus contains work to explicitly model the inter-document rela- documents from different areas of computer sci- tionship. ence, then the concept of machine learning should Second, in our approach the predictions are in- be encoded in any document relevant to it. We ferred at the sentence level. This avoids the effect therefore claim that the embedding of a document of only predicting the surface meaning in word should not only depend on the document itself level (e.g. good vs. nice). Unlike previous work, but also the other documents in the corpus, even our model is explicitly optimized to represent doc- though previous work seldom makes this consid- uments as combinations of sequence embedding eration. beyond words seen in training. In addition, accurate predictions at the lexicon Below we summarize the key contributions: or word level do not necessarily reflect that the • ”true semantics” have been learned. For example, We present a deep and general framework in IMDB dataset review No.10007: and a novel objective for learning document representation unsupervisedly. Our models ”... the father did such a good job.” are end-to-end, easy to implement, and flexi- ble to extend. Obviously, good can be replaced with synonyms • We perform experiments through sentiment like nice without significantly altering the mean- analysis and topic classification to show that ing of the sentence. However, since the syn- our model, referred to as self-discriminative onyms are treated as independent tokens in PV and document embedding (SDDE), is competitive Doc2VecC, the lexicon good must be predicted ex- to the state-of-the-art solutions based on tra- actly. Moreover, to accurately predict the final ditional context-prediction objectives. word job, the embedding probably only needs to know that did a good job is a very common phrase, • Our extensive experiments quantitatively and without having to understand the true meaning of qualitatively show that SDDE learns more ef- job. This example shows that in order to accu- fective features that capture more document- rately predict a local lexicon, the embedding might level information. To the best of our knowl- opt to encode the syntactic relationship instead of edge, SDDE is the first deep network to true semantics. Enforcing document embeddings model inter-instance information at docu- to make predictions at the word level could be too ment level. strong of an objective. More specifically, we argue that the true semantics should not only depend on • We further propose to evaluate unsupervised a small context, but also the relations with other document embedding models in weakly- training documents at document level. supervised classification. That is, lots of un- To address the above concerns we propose a labeled documents with only few labels at- novel model for learning document embedding un- tached to some of them, which is a realistic supervisedly. In contrast with previous work (PV scenario that unsupervised embedding could and Doc2Vec), we model documents according to be particularly useful. two aspects. 2 Related Work First, we abandon the concept of context word prediction when training an embedding model. Here we give an overview of other related meth- Instead we propose a self-supervision learning ods on learning unsupervised text representations. framework to model inter-document information. Besides BoW and TF-IDF, Latent Dirichlet Al- Conceptually, we use the embedding to determine location models (Deerwester et al., 1990; Blei whether a sentence belongs to a document. Our et al., 2003) leverage the orthogonality of high- encoder is equipped with a discriminator to clas- dimensional BoW features by clustering a prob- sify whether a sentence embedding is derived from abilistic BoW for latent topics. a document given that document’s embedding. Several models extend from Word2Vec This explicitly enforces documents to be spread (Mikolov et al., 2013), using context-word reasonably in the embedding space without any la- predictions for training document embedding bels so that they can be discriminated. To the best end-to-end. PV (Le and Mikolov, 2014) keeps
2466 a document embedding matrix in memory and is ment size n = |X |, in which each document Xi jointly trained. The required training parameters is a set of sentences Si; are linear to the number of documents and thus 1 |xi| prohibit PV from being trained on a large corpus. •S i = {si , ··· , si }: a document divided into Moreover, expensive inference for new documents a set of sentences, of set size |Si|, in which j |V|×Tj is required during testing. To address the above each sentence si ∈ R contains a sequence concerns, Doc2VecC (Chen, 2017) combines PV of variable length Tj of word one-hot vectors 1 Tj |V|×1 and a denoising autoencoder (DEA) (Chen et al., wj , ··· , wj , each in R . S is the set of n 2012) with BoW vectors as global document in- S total sentences Si in the training corpus, of formation instead. The final document embedding i=1 are then produced by simply averaging the jointly size m = |S|; trained word embedding. • h U ∈ Another thread of work uses two-stage pipelines w: the size of the word embedding and Rhw×|V| to construct sentence/document embedding from : the word embedding projection ma- u U pre-trained word embedding. Arora et al.(2017) trix. We use w to denote the column in for w propose post-processing weighting strategies on word top of word embedding to build sentence represen- • hs: the size of the sentence embedding and es ∈ tations. WME (Wu et al., 2018) propose a random Rhs : the embedding of sentence s. feature kernel method based on distance between hs pairs of words, which also shows inter-document • di ∈ R : document Xi’s embedding. information helps. However, the cost scales with h ×n the size of training samples such that it is hard to Our goal is to learn a function F : X → R s be applied on large-scale dataset. that maps document Xi to di unsupervisedly. There have been more embedding work on sen- Next, we formulate how SDDE represents a tences compared to documents. These approaches document, then introduce our self-discriminative mostly learn the sentence embedding by model- learning procedure we use to train it. ing the sentence-level (Kiros et al., 2015; Tang 3.1 Document Representation et al., 2017b,a; Logeswaran and Lee, 2018) or word-level (Pagliardini et al., 2018; Kenter et al., We consider a document as mean of sentences, 2016; Arora et al., 2018) distribution hypothesis i.e., breaking a document into several subse- (Harris, 1954; Polajnar et al., 2015) in a large or- quences. We demonstrate several benefits of the dered corpus. We note that the main difference be- design in SDDE. First, decomposing long docu- tween learning embedding for sentences and doc- ments into shorter but reasonable semantic unit uments is that documents are not ordered in a cor- (e.g., sentences) makes encoding easier and faster pus. Some other work model sentences with RNN since they can be processed in parallel. Simi- autoencoders (Hill et al., 2016a; Gan et al., 2017). lar concepts of modeling documents hierarchically Documents often refer to long-length text contain- have shown benefits in some supervised tasks such ing multiple sentences, which might be hard to as text classification (Yang et al., 2016). It also model with RNNs (Pascanu et al., 2013; Jing et al., makes the model insensitive to document length, 2017) and time-consuming on large corpus. which is important because length varies greatly in real documents (see Table1). 3 Model and Design Rationale In training, we further propose to represent a document during training using the average To facilitate downstream tasks, a document em- of only a subset of its sentences. This special bedding is required to compress useful features sentence-level dropout is beneficial for training into a compact representation. It is not an easy by creating up to |Xi| combinations for each doc- task to learn discriminable features unsupervisedly q ument, where q is the number of sentences to keep. since validation information is not accessible for This enforces the local sentence representations to training. We first introduce some notations: capture global information of a document by rep- •V : the training corpus vocabulary of size |V|; resenting it with a subset of sentences in it. The word embedding is used as globally shared build- •X = {X1, ··· ,Xn}: a training corpus of docu- ing blocks for sentence embedding.
2467 j For a document Xi = {si }, or Si, the embed- Algorithm 1 Self-Discriminative Learning for ding is derived from averaging the respective rep- Unsupervised Document Embedding resentations of subsequences. Noted as: n Input: Documents X = {Xi}1 , p, k, hw, hs. hs q Output: Function F : X → R that maps 1 X hs d = ej, text of a document to an embedding d ∈ R . i s (1) n q S j=1, 1: Compute S = Si from set Si of each Xi. s∼P (s) Si i=1 2: Create and initialize D, U. where 3: Create and initialize E as in Eq.5 or Eq.6. es = E(s), (2) 4: while not converge do: where a sentence encoder E is introduced to pro- 5: for i = 1, . . . , n do j 6: Sample s from Si. duce sentence embedding for si . In practice, sen- p tences can be obtained by simply segmenting doc- 7: Get es ← E(s) as a positive sample. 1 q uments with punctuation. In testing, the document 8: Sample si ,..., si from Si \{s}. 9: for j = 1, . . . , q do embedding is obtained by averaging all the sen- j j tences in which: 10: Get es ← E(si ) 11: end for 1 j q X 12: Get di with Eq.1 given {es} . d = es. (3) j=1 |Si| ∀s∈S 13: for ` = 1, . . . , k do i 0 14: Sample s` from S\Si. ` 0 We note that averaging subsequences differs 15: Get es0 ← E(s`) as a negative sample. from averaging of words in two aspects. First, 16: end for p ` k each sentence is encoded individually before be- 17: Compute Eq.4 given di, es, {es0 }`=1. ing averaged, allowing incorporation of word or- 18: Backprop and update for E, D, U. der into design rationale at least in a reasonable 19: end for range. Second, subsequences may have different 20: end while lengths that reveal syntactic information. To il- 21: Return F (E, U), lustrate, BoW/mean-of-word models suffer from ambiguously modeling two different documents which are similar in word distributions but differ in a document. The overall procedure is summarized some aspects of interest. Mean-of-sentence model in Algorithm1. avoids such concern by modeling documents at the We propose an objective that explicitly opti- sentence level. It could be expected that it is much mizes SDDE towards representing a document less likely to find two documents with similar sen- with mean of (encoded) sentences. To optimize tence distribution than similar word distribution. the discriminator D, we formulate it as a binary Mean-of-sentences formulation can be smoothly classifier that takes pairs of document embedding reduced to mean-of-word models (by treating each d of a document Xi and a sentence embedding es, word as a sentence) or pure sequence models (by (d, es), as inputs. The discriminator is asked to treating each document as a very long sentence). discriminate using d whether the sentence s be- longs to the document Xi or the other documents 3.2 Self-Discriminative Learning 0 X ∈ X \ {Xi}. The loss then becomes: Unlike PV or Doc2VecC which emphasize mod- eling distributional information within individual k p X h ` i log(1 − D(d, e )) + log(D(d, e 0 )) , documents, we model relations across documents. s 0 E s s ∼PS , `=1 0 The basic idea is that we hope to learn an embed- s ∈S/ i ding for each sentence in the document as well as (4) p a discriminator that determines whether a sentence for each document with one positive sample es 0 belongs to a document. Self-discriminative learn- and k negative samples of sentences es0 , where s 0 ing uses a discriminator network D to determine are not in the sentence set Si of Xi, as s ∈/ Si. p whether a sentence belongs to a document. The Note that es is not used for d otherwise it would aim is to learn a suitable embedding and a good be trivial to be solved by the discriminator. discriminator to determine if a sentence belongs to The spirits of self-discriminative learning can
2468 be understood as unsupervisedly mimicking su- Dataset #Class #Train / #Test Doc Length Sent Length pervised inference without knowledge of any label IMDB 2 75k / 25k 124.6±8,856.7 11.6±105.5 Yelp P. 2 560k / 38k 70.0±4,117.8 8.2±48.7 information by treating sentences from other doc- AG’s News 4 120k / 7.6k 27.2±66.1 11.8±66.7 uments as fake/negative samples. One main con- DBPedia 14 560k / 70k 32.8±231.3 9.7±68.6 cern that it is possible to find similar sentences in Table 1: Statistics of the datasets. Length of document two different documents. Our discriminator par- and sentence in words (mean±variance), which could ticularly addresses this issue by optimizing for the be high for real-world scenarios such as online reviews. most discriminative sentences rather than similar Dataset k p hw hs ones that might not be critical to shape the em- IMDB 1 3 100 100 bedding. To minimize the loss, the encoder would Yelp P. 1 3 300 500 AG’s News 3 1 100 100 tend to preserve the most essential feature to facil- DBPedia 4 2 300 500 itate the discriminator to push away any two doc- uments, which should encourage the embedding Table 2: Hyperparameters used in experiments. The points spread even more widely across the space. document embedding trained with the same hyperpa- rameters are used for all the evaluations without task- This in turn should result in more ease in down- specific tuning. stream tasks: for example in learning a decision hyperplane in a classification task. adopt lightweight network structures for discrimi- nators. For the IMDB datasets, we find inner prod- 3.3 Sentence Encoder uct (dV)tE(s) with a learnable matrix V suffi- Next, we narrow down to sentence encoder E. cient. For the other datasets in Table1, two fully- Given a sequence of word one-hot vectors as a sen- connected layers with ReLU activations in latent tence s = [w1,..., wT ], we project them into an are used. embedding layer U to retrieve their corresponding 3.5 Sampling Sentences word embedding. Note that the word embedding are trained jointly. We relate our method to the Negative Sampling Our first method uses: (Mikolov et al., 2013) technique which is a simpli- fied objective of softmax approximation (Mikolov E(s) = φ(ReLU(G(Ust))), (5) et al., 2013; Mnih and Teh, 2012; Zoph et al., 2016). Negative sampling has been used as an where G is a single-layer RNN encoder using efficient and effective technique in learning word GRU cells to process the word embedding se- embedding. We reformulate it to train document quences and φ is a linear transform for dimension embedding by sampling in sentence level, which hs of sentence embedding. is easy to implement and efficient to train just like Our second method, we use a schema of mean- Word2Vec (Mikolov et al., 2013). of-word for advantage of fast generation, we av- In practice, when training with mini-batches the erage the word embedding w within a sentence s documents for negative samples are from the same along time axis as AVG encoder: mini-batch, which requires small extra computa- tion efforts. |s| SDDE requires a similar number of parameters 1 X E(s) = φ(ReLU( Uwt)), (6) as Doc2VecC does, but much less than PV. In ad- |s| i i=1 dition, the sentence encoder is flexible and can in- corporate other techniques of text processing such E Let us stress that the role of encoder is to extract as attention methods. local feature from every sentence, and the over- all objective encourages SDDE to represent docu- 4 Experiments ments as mean of sentence embedding. 4.1 Setup 3.4 Discriminator Public datasets on sentiment analysis and topic An undesired pitfall comes from a learned weak classification across diverse domains including encoder with a powerful discriminator causing the online reviews, news, and Wiki pages are used in- embedding produced by the encoder useless for cluding IMDB dataset (Maas et al., 2011) and the downstream tasks. To avoid such a pitfall, we others from Zhang et al.(2015). Table1 provides
2469 a summary. Only the training splits are used in Model Error% training embedding with subsampled training split Skip-thought* (Kiros et al., 2015) 17.4 SIF (GloVe) (Arora et al., 2017) 15.0 for cross-validations. RNN-LM* (Mikolov et al., 2010) 13.6 W2V-AVG* (Mikolov et al., 2013) 12.7 We preprocess the datasets by normalizing the DEA* (Chen et al., 2012) 12.5 text to lower class and replacing words appearing PV-DM 20.0 PV-DBoW 12.0 less than 10 times with a UNK token. Out-of- WME (Wu et al., 2018) 11.5 vocabulary words in testing set are also replaced. Doc2VecC* (Chen, 2017) 11.7 SDDE-AVG 10.6 All our baseline models use the same input data. SDDE-RNN 10.2 To define the sentences for experiment, we uti- lize the sentence tokenizer from NLTK. For the Table 3: Sentiment Classification on IMDB Bench- documents containing only one sentence we sim- mark. *Results are collected from Chen(2017). ply divide it into multiple subsequences for sam- SDDE-RNN Doc2VecC Pred pling. We use RMSProp method for optimiza- True False True False tion. Dropout 50% of input to the discriminator. #Sent 14.8 14.7 14.5 14.8 Weights are random-uniformly initialized between [-1, 1]. Table 4: Mean of #sentence per document in IMDB The other hyperparameters are summarized in dataset, in groups of classification correctness. Table2. All the models use the same embedding size for fair comparison. The trained document of both Distributed Memory (DM) and Distributed embedding are used for all the evaluations without Bag of Words (DBoW) are reported. For differ- specific tuning. ent encoders in SDDE, AVG is for averaging word embedding and RNN is for the RNN encoder. 4.2 Evaluation Generally, it is not easy to evaluate an unsuper- Self-Discriminative Learning is Effective vised embedding model. In Section 4.3 and 4.4, From the experiment result in Table3, we can see we evaluate the performance on standard docu- that our self-discriminative learning is effective ment classification following the common prac- and superior on the document embedding models tice used by previous work (Chen, 2017; Wu et al., for both AVG and RNN versions. SDDE-RNN 2018; Arora et al., 2017): a classification task with achieves best accuracy on IMDB dataset 1.5% a linear SVM (Fan et al., 2008) trained on the la- margin against Doc2VecC. bels in each dataset. Next, we study unsupervised document embedding on two novel aspects. In Study the Property of SDDE Unlike previous Section 4.5, we study a weakly-supervised clas- work modeling documents on the word or short sification setting that fits the realistic scenario of context level, SDDE operates on the sentence using unsupervised embedding with only a few la- level. We study the false and true predictions bels. In Section 4.6, we provide a metric to eval- output by SVM upon SDDE in comparison with uate the effectiveness of modeling inter-document Doc2VecC. Table5 show some examples that have information. the largest difference. We observed SDDE can better capture contradicting or contrasting opin- 4.3 Sentiment Analysis with IMDB Dataset ions. We observe some wrong predictions (Row We first compare our models with the others state- 3) are due to the ambivalent reviews. SDDE is in- of-the-art competitors. RNN-LM (Mikolov et al., sensitive to the number of sentences; we found the 2010) and Skip-thought (Kiros et al., 2015) are effect of the number of sentences per document RNN-based. SIF (Arora et al., 2017), W2V-AVG was trivial as shown in Table4. (Mikolov et al., 2013), and WME (Wu et al., 2018) are two-stage approach that post-processing on 4.4 Large-Scale Document Classification on word embedding. We collect the results reported More Dataset on the widely-used benchmark sentiment classifi- Next, we borrow some public large-scale dataset cation dataset IMDB. For PV, we use Gensim im- in Table1 to further validate the effectiveness plementation (Rehˇ u˚rekˇ and Sojka, 2010); versions of SDDE compared to the other models. For https://www.nltk.org/ Doc2vecC and SIF, we use the code from the au-
2470 Label: 1 i don t even like watching those late night talk shows , but i found this one really interesting . i imagine it s probably close to the truth SDDE: 0.89 — it feels like an honest account , if that means anything . kinda feel for the people somewhat when you watch it . a nice movie for a Doc2VecC: 0.27 saturday night . Label: 0 i m a boorman fan but this is arguably his least successful film . comedy has never been his strong suit , and here his attempts at SDDE: 0.12 screwball farce are clumsily done . still , it s almost worth seeing for boorman s eye for talent : this is one of uma thurman s first Doc2VecC: 0.75 starring roles , and as always she is ravishing to watch . on a sad side note boorman wrote the script with his daughter ,
Model Yelp P. AG DBP. PV-DM 17.6 39.6 21.6 PV-DBoW 12.1 16.8 10.4 Word2Vec AVG 8.0 14.2 2.7 SIF 8.4 13.8 2.9 Doc2VecC 7.4 10.4 2.0 SDDE-RNN 8.7 12.7 8.6 SDDE-AVG 6.7 9.8 1.8 (a) IMDB
Table 6: Testing error rate (%) with standard classifi- cation on public datasets. Bold text indicates passing hypothesis test with p-value < 0.05 with different ran- dom initialization.
Model IMDB Yelp P. AG DBP. (b) AG’s news Doc2VecC 12.5 11.5 13.9 3.6 PV-DBoW 13.7 18.6 13.9 13.1 Figure 1: t-SNE embedding Visualizations of weakly- SDDE 11.7 10.0 10.0 2.9 supervised learning experiments in Section 4.5. Col- ored points are labeled training data in the experiments Table 7: Testing error (%) in weakly-supervised set- and gray points are the unlabeled testing documents. ting. Only 1k labeled data per class were used. 4.5). thors. We use SIF to generate document embed- datasets we have used in previous experiments, but ding with Word2Vec trained on each dataset as its this time only when very few labels are available. inputs. Results are shown in Table6. SDDE-AVG We hypothesize that SDDE is particularly use- performs slightly better across different dataset. ful for classification with few labels since the self- We hypothesis SDDE gets larger improvement on discriminative learning has exploited the possible IMDB dataset since SDDE can handle longer doc- features to map the text onto the embedding space uments better by exploiting sentence embeddings. properly during the representation learning phase. On the other hand, the RNN version of SDDE per- The embedding is expected to be more discrim- forms significantly worse than the word-averaging inable to facilitate finding the classification deci- version. We may remind the reader that state-of- sion hyperplanes with fewer labeled data. the-art unsupervised document embedding models We randomly sample equal number of instances are not RNN-based. The effects of word order are from each class to train a SVM and verify with still unclear. Wieting and Gimpel(2017) provides the whole testing set. PV-DBoW, Doc2VecC, and a study of sentence embedding. We hypothesize SDDE-AVG are examined in this experiment. We that it may be difficult for an RNN encoder to use the same pre-trained document embedding as learn to incorporate multi-domain information in in the previous experiments. We repeat the whole datasets with many classes (e.g., DBpedia) unsu- procedure 30 times and report the means and tune pervisedly. This would be our future work. the penalty parameter C to find the best value for 4.5 Classification with Few Labeled Data each model. Results in Figure2 and Table7 show SDDEs outperform PV and Doc2VecC. We visu- Next, we consider a more real-world weakly- alize the training points with t-SNE. As shown in supervised learning scenario: classification on the Figure1, SDDE seems to be able to spread the em- https://github.com/mchen24/iclr2017 bedded data more widely, which eventually leads https://github.com/PrincetonML/SIF to better usage of scarce data for classification.
2471 Figure 2: Weakly-supervised learning on datasets in Table1. Training with [10, 20, 30, 40, 50, 100, 1000] instances per class (X-axis) and computing the accuracy on the whole testing set (Y-axis).
Model IMDB Yelp P. AG DBP. Metric A E A-E AE A-E AE A-E AE A-E Doc2VecC .78 .76 .02 .52 .49 .03 .57 .46 .11 .60 .40 .20 Word2Vec AVG .86 .85 .01 .54 .51 .03 .57 .44 .13 .75 .61 .14 PV-DBoW .28 .27 .01 .11 .10 .01 .71 .60 .09 .45 .36 .09 SDDE .36 .27 .09 .14 .08 .06 .20 .01 .19 .58 -.03 .61
Table 8: Distances of Intra & Inter-class cosine similarity. A for IntraCos and E for InterCos, note that they cannot be compared across different models. Instead, distance A-E defined in Equation7 is reported to study a method’s effectiveness of modeling inter-document features. The higher the number the better.
4.6 Distance of Intra&Inter-Class Pairwise Cosine Similarity
Definition We examine our assumption of the ability of SDDE to model inter-document feature. Similar to (Hill et al., 2016b), we consider pair- wise cosine similarity between documents with topic labels, this allows us to quantitatively eval- uate unsupervised document embedding at inter- document level. Our assumption is that: if pair- wise similarity between documents is calculated based on different kinds of embedding, the bet- ter embedding results should comply with the properties of both high similarities between those documents within the same underlying class, de- noted as IntraCos(d, d0) and low similarities be- tween document pairs from different classes, or InterCos(d, d˜). The mean distance: Figure 3: SDDE-AVG t-SNE visualization of class mean(IntraCos) − mean(InterCos), (7) ”World” in AG News dataset testing set. Titles of docu- ments are shown. Meaningful clusters such as (A) dis- is considered as our metric to avoid simply max- aster, (B) election, and (C) war. imizing IntraCos or minimizing InterCos. 5 Conclusion SDDE Provides High Separation Table8 shows the evaluation. The distances (Eq.7) for the base- Compared to mainstream unsupervised document line models are small, which support our assump- embedding models (trained to perform predictions tion that these methods are not able to model inter- on the lexicon level) SDDE embeddings capture document features properly. On the other hand, information at the inter-document level, as they distances for SDDE are significantly larger. With are trained to maximize the distance between a the classification experiments, we believe SDDE sentence and a corresponding document. We hope better preserves meaningful inter-document fea- the underlying idea of SDDE offers the document- tures. Figure3 shows some meaningful clusters in embedding community a new investigation direc- SDDE in the World class as cohesive sub-classes. tion. Self-discriminative learning shows potential
2472 for real-world scarcely-labeled scenarios, and our Felix Hill, KyungHyun Cho, Anna Korhonen, and future work will focus on joint training of repre- Yoshua Bengio. 2016b. Learning to understand sentations for semi-supervised learning. phrases by embedding the dictionary. TACL, 4:17– 30. Acknowledgements Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacic, and Yoshua Bengio. This material is based upon work supported by Mi- 2017. Gated orthogonal recurrent units: On learning crosoft Research Asia (MSRA) grant, and by Tai- to forget. CoRR, abs/1706.02761. wan Ministry of Science and Technology (MOST) Tom Kenter, Alexey Borisov, and Maarten de Rijke. under grant number 108-2634-F-002 -019. 2016. Siamese CBOW: optimizing word embed- dings for sentence representations. In ACL. References Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urta- Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. sun, and Sanja Fidler. 2015. Skip-thought vectors. A simple but tough-to-beat baseline for sentence em- In NIPS, pages 3294–3302. beddings. In ICLR. Quoc V. Le and Tomas Mikolov. 2014. Distributed rep- Sanjeev Arora, Yingyu Liang, Tengyu Ma, Mikhail resentations of sentences and documents. In ICML, Khodak, Nikunj Saunshi, and Brandon Stewart. pages 1188–1196. 2018. A la carte embedding: Cheap but effective in- duction of semantic feature vectors. In Proceedings Lajanugen Logeswaran and Honglak Lee. 2018. An of the 56th Annual Meeting of the Association for efficient framework for learning sentence represen- Computational Linguistics, ACL 2018, Melbourne, tations. In ICLR. Australia, July 15-20, 2018, Volume 1: Long Papers, pages 12–22. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2011. Learning word vectors for sentiment analy- 2003. Latent dirichlet allocation. Journal of Ma- sis. In The 49th Annual Meeting of the Association chine Learning Research, 3:993–1022. for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 Minmin Chen. 2017. Efficient vector representation for June, 2011, Portland, Oregon, USA, pages 142–150. documents through corruption. In ICLR. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Minmin Chen, Zhixiang Eddie Xu, Kilian Q. Wein- Huang, and Alan Yuille. 2015. Deep captioning berger, and Fei Sha. 2012. Marginalized denoising with multimodal recurrent neural networks (m-rnn). autoencoders for domain adaptation. In ICML. In ICLR.
Andrew M. Dai and Quoc V. Le. 2015. Semi- Tomas Mikolov, Martin Karafiat,´ Lukas´ Burget, Jan supervised sequence learning. In NIPS, pages 3079– Cernocky,´ and Sanjeev Khudanpur. 2010. Recur- 3087. rent neural network based language model. In IN- TERSPEECH 2010, 11th Annual Conference of the Scott Deerwester, Susan T. Dumais, George W. Fur- International Speech Communication Association, nas, Thomas K. Landauer, and Richard Harshman. Makuhari, Chiba, Japan, September 26-30, 2010, 1990. Indexing by latent semantic analysis. JASIS, pages 1045–1048. 41(6):391–407. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang- rado, and Jeffrey Dean. 2013. Distributed represen- Rui Wang, and Chih-Jen Lin. 2008. Liblinear: A tations of words and phrases and their composition- library for large linear classification. JMLR, pages ality. In NIPS, pages 3111–3119. 1871–1874. Andriy Mnih and Yee Whye Teh. 2012. A fast and sim- Zhe Gan, Yunchen Pu, Ricardo Henao, Chunyuan Li, ple algorithm for training neural probabilistic lan- Xiaodong He, and Lawrence Carin. 2017. Learning guage models. In Proceedings of the 29th Inter- generic sentence representations using convolutional national Conference on Machine Learning, ICML neural networks. In EMNLP. 2012, Edinburgh, Scotland, UK, June 26 - July 1, 2012. Zellig S. Harris. 1954. Distributional structure. WORD, 10(2-3):146–162. Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised learning of sentence embed- Felix Hill, Kyunghyun Cho, and Anna Korhonen. dings using compositional n-gram features. In 2016a. Learning distributed representations of sen- Proceedings of the 2018 Conference of the North tences from unlabelled data. In NAACL HLT, pages American Chapter of the Association for Computa- 1367–1377. tional Linguistics: Human Language Technologies,
2473 NAACL-HLT 2018, New Orleans, Louisiana, USA, embedding. In Proceedings of the 2018 Conference June 1-6, 2018, Volume 1 (Long Papers), pages 528– on Empirical Methods in Natural Language Pro- 540. cessing, Brussels, Belgium, October 31 - November 4, 2018, pages 4524–4534. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, networks. In ICML, pages 1310–1318. Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Hieu Pham, Minh-Thang Luong, and Christopher D. NAACL HLT, pages 1480–1489. Manning. 2015. Learning distributed representa- tions for multilingual text sequences. In NAACL- Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. HLT, pages 88–94. Character-level convolutional networks for text clas- sification. In NIPS, pages 649–657. Tamara Polajnar, Laura Rimell, and Stephen Clark. 2015. An exploration of discourse-based sentence Barret Zoph, Ashish Vaswani, Jonathan May, and spaces for compositional distributional semantics. Kevin Knight. 2016. Simple, fast noise-contrastive In Proceedings of the First Workshop on Linking estimation for large RNN vocabularies. In NAACL Computational Models of Lexical, Sentential and HLT, pages 1217–1222. Discourse-level Semantics, pages 1–11. Association for Computational Linguistics. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP.
Radim Rehˇ u˚rekˇ and Petr Sojka. 2010. Software Frame- work for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Val- letta, Malta. ELRA. http://is.muni.cz/ publication/884893/en. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural net- works. In NIPS, pages 3104–3112. Shuai Tang, Hailin Jin, Chen Fang, Zhaowen Wang, and Virginia R. de Sa. 2017a. Rethinking skip- thought: A neighborhood based approach. In Pro- ceedings of the 2nd Workshop on Representation Learning for NLP, Rep4NLP@ACL 2017, Vancou- ver, Canada, August 3, 2017, pages 211–218. Shuai Tang, Hailin Jin, Chen Fang, Zhaowen Wang, and Virginia R. de Sa. 2017b. Trim- ming and improving skip-thought vectors. CoRR, abs/1706.03148. Sida Wang and Christopher D. Manning. 2012. Base- lines and bigrams: Simple, good sentiment and topic classification. In The 50th Annual Meeting of the As- sociation for Computational Linguistics, Proceed- ings of the Conference, July 8-14, 2012, Jeju Island, Korea - Volume 2: Short Papers, pages 90–94. John Wieting and Kevin Gimpel. 2017. Revisiting re- current networks for paraphrastic sentence embed- dings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Vol- ume 1: Long Papers, pages 2078–2088. Lingfei Wu, Ian En-Hsu Yen, Kun Xu, Fangli Xu, Avinash Balakrishnan, Pin-Yu Chen, Pradeep Ravikumar, and Michael J. Witbrock. 2018. Word mover’s embedding: From word2vec to document
2474