Towards Controlled and Diverse Generation of Article Comments Linhao Zhang1, Houfeng Wang1 1MOE Key Lab of Computational Linguistics, Peking University, Beijing, 100871, China 2Baidu Inc., China fzhanglinhao, [email protected] Abstract Articles (truncated): With the popularity of the TV series Ode to Joy, the actors’ salaries are revealed recently. According to the news, Liu Tao’s salary is about 400K per episode, while Jiang Xin is paid about 100K to Much research in recent years has focused 300K. Yang Zi, who plays Qiu Yingying in the play, also earns about 100K on automatic article commenting. However, to 300K. Compared with them, Wang Ziwen and Guan Muer are relatively not so famous, and their salaries are only about tens of thousands. few of previous studies focus on the control- Seq2Seq I love this movie. lable generation of comments. Besides, they Disgust Does Wang Ziwen have any acting skills? tend to generate dull and commonplace com- Like Liu Tao is so beautiful. I am so relieved to see such news. ments, which further limits their practical ap- Ours Happiness Sadness I am wordless after watching Ode to Joy. plication. In this paper, we make the first step Anger What’s wrong with Jiang Xin? towards controllable generation of comments, by building a system that can explicitly control Figure 1: Comparison of output comments of the the emotion of the generated comments. To Seq2Seq baseline and our model. We can see that the achieve this, we associate each kind of emo- baseline cannot control the expressed emotion, and the tion category with an embedding and adopt generated comment is dull and irrelevant to the arti- a dynamic fusion mechanism to fuse this em- cle (red colored). By contrast, Our model is emotion- bedding into the decoder. A sentence-level controllable and can generate more diverse and relevant emotion classifier is further employed to bet- comments, thanks to the hierarchical copy mechanism ter guide the model to generate comments ex- (blue colored). pressing the desired emotion. To increase the diversity of the generated comments, we pro- pose a hierarchical copy mechanism that al- ments for users, who may later select one and refine lows our model to directly copy words from it (Zheng et al., 2018). the input articles. We also propose a restricted beam search (RBS) algorithm to increase intra- In addition to the practical importance of this sentence diversity. Experimental results show task, it also has significant research value. It can that our model can generate informative and di- be seen as a natural language generation (NLG) verse comments that express the desired emo- task, yet unlike machine translation or text summa- tions with high accuracy. rization, the comments can be rather diverse. That is, for the same article, there can be numerous ap- 1 Introduction propriate comments that are from different angles. Automatic article commenting is a valuable yet In this sense, this task is similar to dialog system, arXiv:2107.11781v1 [cs.CL] 25 Jul 2021 challenging task. It requires the machine first to yet because the input article is much longer and understand the articles and then generate coherent more complex than dialog text, it is hence more comments. The task was formally proposed by challenging. (Qin et al., 2018), along with a large-scale dataset. Despite the importance of this task, it is still rel- Much research has since focused on this task (Lin atively new and not well-studied. One of the major et al., 2018b; Ma et al., 2018a; Li et al., 2019). limitations of current article commenting systems The ability to generate comments is especially is that the generation process is not controllable, useful for online news platforms, for the comments meaning that the comments are conditioned entirely can encourage user engagement and interactions on the articles, and users can not further control (Qin et al., 2018). Besides, an automatic comment- the features of comments. In this work, we make ing system also enables us to build a comment- the first step towards controllable article comment- assistant which can generate some candidate com- ing and propose a model to control the emotion of generated comments, which has wide practical sentence diversity, respectively. application. Take comment-assistant (Zheng et al., 2018) as an example, it is far more desirable to 2 Method have candidate comments each expresses a differ- The overall architecture of our proposed model, ent emotion, and users can hence choose one that Controllable Commenting System (CCS), is shown matches their own emotion towards the article. in Figure2. We describe the details in the following Another problem of current commenting sys- subsections. tems arises from the limitation of the Seq2Seq framework (Sutskever et al., 2014), which has been 2.1 Task Defination known to suffer from generating dull and responses The task can be formulated as follows: Given an that are irrelevant to the input articles (Li et al., article D and an emotion category χ, the goal is to 2015; Wei et al., 2019; Shao et al., 2017). As shown generate an appropriate comment Y that expresses in Figure1, the Seq2Seq baseline generates I love the emotion χ. this movie for the input article, despite the fact that Specifically, D consists of ND sentences. Each Ode of joy is not a movie, but a TV series. sentence consists of T words, where T can vary In this work, we propose a controllable article among different sentences. For each word, we use commenting system that can generate diverse and ei to denote its word embedding. relevant comments. We first create two emotional 2.2 Basic Structure: Hierarchical Seq2Seq datasets based on (Qin et al., 2018). The first one is a find-grained dataset that contains fAnger, Dis- We encode an article by first building representa- gust, Like, Happiness and Sadness g and the sec- tions of sentences and then aggregating them into ond one is a coarse-grained one that contains only an article representation. fPositive and Negativeg. To incorporate the emo- Word-Level Encoder - We first encode the ar- tion information into the decoding process, we first ticle D on the word level. For each sentence in enc associate each kind of emotion with an embedding D, the word-level LSTM encoder LST M w reads and then feed the emotion embedding into the de- the words sequentially and updates its hidden state coder at each time step. A dynamic fusion mech- hi. Formally, anism is employed to utilize the emotion informa- h = LST M enc(e ; h ) (1) tion selectively at each decoding step. Besides, the i w i i−1 decoding process is further guided by a sequence- The last hidden state, hT , stores information of the level emotion loss term to increase the intensity of whole sentence and thus can be used to represent emotional expression. that sentence: To generate diverse comments, we propose a e^ = hT (2) hierarchical copy mechanism to copy words from Sentence-Level Encoder - Given sentence em- the input articles. This is based on the observation beddings (^e ; e^ ; :::; e^ ), we then encode the arti- that we can discourage the generation of dull com- 1 2 ND cle at the sentence-level: ments like I don’t know by enhancing the relevance between comments and articles. In this way, the enc gi = LST Ms (^ei; gi−1) (3) inter-sentence diversity gets increased. Moreover, enc we observe that the repetition problem of Seq2Seq where the LST Ms is the sentence-level LSTM framework (See et al., 2017; Lin et al., 2018a) can encoder and gi is its hidden state. The last hidden be seen as a lack of intra-sentence diversity, and state, gND , aggregates information of all the sen- we further adopt a restricted beam search (RBS) tences and is later used by the decoder to initialize algorithm to tackle this problem. its hidden state. To sum up, our contributions are two-folds: Decoder Similarly, the decoder LSTM dec 1) We make the first step to build a controllable LST M updates its hidden state st at time step article commenting system by injecting emotion t: into the decoding process. Features other than emo- dec tion can be controlled in a similar way. st = LST M (et; st−1) (4) 2) We propose the hierarchical copy mechanism where et is the embedding of the previous word. and RBS algorithm to increase the inter- and intra- While training, this is the previous word of the iPad Hierarchical P Emotion Copy emotion Loss step embedding MLP top k { Pfinal ∑ t step γij Patten × (1 − pg en) “iPad” a . zoo × pg en P βt Sentence vocab i LSTM A new MLP st Dynamic Fusion t αij Word LSTM new ⨀ ⊕ ⊗ first sentence second sentence ⨀ <emotion> Hierarchical Encoder Decoder Figure 2: The architecture of CCS. On the left is the encoder, which encodes articles at the word- and sentence- level. On the right is the decoder, with emotion information dynamically fused into the decoding process. We add a emotion loss item to further bias the decoding process. Besides, a hierarchical copy mechanism is proposed to improve the diversity of the generated comments. reference comment; at test time it is the previous ever, this method is overly simplistic. It uses emo- word emitted by the decoder. The attention mecha- tion information indiscriminately at each decoding nism (Luong et al., 2015) is also applied. Formally, step, yet not all steps require the same amount At each decoding step, we compute the attention of emotion information. Simply using the same weights between the current decoder hidden vector emotion embedding during the whole generation and all the outputs of the sentence-level encoder.