Affect-LM: a Neural Language Model for Customizable Affective Text Generation
Total Page:16
File Type:pdf, Size:1020Kb
Affect-LM: A Neural Language Model for Customizable Affective Text Generation Sayan Ghosh1, Mathieu Chollet1, Eugene Laksana1, Louis-Philippe Morency2 and Stefan Scherer1 1Institute for Creative Technologies, University of Southern California, CA, USA 2Language Technologies Institute, Carnegie Mellon University, PA, USA 1 sghosh,chollet,elaksana,scherer @ict.usc.edu { [email protected]} “… good about this.” Abstract Context Words ct 1 − “I feel so …” Affect-LM “… great about this.” Human verbal communication includes “… awesome about this.” affective messages which are conveyed Mid (optional) Inference through use of emotionally colored words. Automatic # % " There has been a lot of research in this $ ! Low High direction but the problem of integrat- Affect Category Affect Strength et 1 β ing state-of-the-art neural language mod- − els with affective information remains Figure 1: Affect-LM is capable of generating an area ripe for exploration. In this emotionally colored conversational text in five paper, we propose an extension to an specific affect categories (et 1) with varying − LSTM (Long Short-Term Memory) lan- affect strengths (β). Three generated example guage model for generating conversa- sentences for happy affect category are shown in tional text, conditioned on affect cate- three distinct affect strengths. gories. Our proposed model, Affect-LM enables us to customize the degree of emotional content in generated sentences 2012) and studies of correlation between function through an additional design parameter. words and social/psychological processes (Pen- Perception studies conducted using Ama- nebaker, 2011). People exchange verbal messages zon Mechanical Turk show that Affect- which not only contain syntactic information, but LM generates naturally looking emotional also information conveying their mental and emo- sentences without sacrificing grammatical tional states. Examples include the use of emo- correctness. Affect-LM also learns affect- tionally colored words (such as furious and joy) discriminative word representations, and and swear words. The automated processing of perplexity experiments show that addi- affect in human verbal communication is of great tional affective information in conversa- importance to understanding spoken language sys- tional text can improve language model tems, particularly for emerging applications such prediction. as dialogue systems and conversational agents. 1 Introduction Statistical language modeling is an integral component of speech recognition systems, with Affect is a term that subsumes emotion and longer other applications such as machine translation and term constructs such as mood and personality information retrieval. There has been a resur- and refers to the experience of feeling or emo- gence of research effort in recurrent neural net- tion (Scherer et al., 2010). Picard(1997) provides works for language modeling (Mikolov et al., a detailed discussion of the importance of affect 2010), which have yielded performances far supe- analysis in human communication and interaction. rior to baseline language models based on n-gram Within this context the analysis of human affect approaches. However, there has not been much from text is an important topic in natural language effort in building neural language models of text understanding, examples of which include senti- that leverage affective information. Current liter- ment analysis from Twitter (Nakov et al., 2016), ature on deep learning for language understand- affect analysis from poetry (Kao and Jurafsky, ing focuses mainly on representations based on 634 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 634–642 Vancouver, Canada, July 30 - August 4, 2017. c 2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1059 word semantics (Mikolov et al., 2013), encoder- before concluding the paper in Section6. decoder models for sentence representations (Cho et al., 2015), language modeling integrated with 2 Related Work symbolic knowledge (Ahn et al., 2016) and neural caption generation (Vinyals et al., 2015), but to the Language modeling is an integral component of best of our knowledge there has been no work on spoken language systems, and traditionally n- augmenting neural language modeling with affec- gram approaches have been used (Stolcke et al., tive information, or on data-driven approaches to 2002) with the shortcoming that they are unable to generate emotional text. generalize to word sequences which are not in the training set, but are encountered in unseen data. Motivated by these advances in neural language Bengio et al.(2003) proposed neural language modeling and affective analysis of text, in this pa- models, which address this shortcoming by gen- per we propose a model for representation and eralizing through word representations. Mikolov generation of emotional text, which we call the et al.(2010) and Sundermeyer et al.(2012) extend Affect-LM. Our model is trained on conversational neural language models to a recurrent architecture, speech corpora, common in language modeling where a target word wt is predicted from a con- for speech recognition applications (Bulyko et al., text of all preceding words w1, w2, ..., wt 1 with − 2007). Figure1 provides an overview of our an LSTM (Long Short-Term Memory) neural net- Affect-LM and its ability to generate emotionally work. There also has been recent effort on build- colored conversational text in a number of affect ing language models conditioned on other modali- categories with varying affect strengths. While ties or attributes of the data. For example, Vinyals these parameters can be manually tuned to gener- et al.(2015) introduced the neural image caption ate conversational text, the affect category can also generator, where representations learnt from an in- be automatically inferred from preceding context put image by a CNN (Convolutional Neural Net- words. Specifically for model training, the affect work) are fed to an LSTM language model to gen- category is derived from features generated using erate image captions. Kiros et al.(2014) used keyword spotting from a dictionary of emotional an LBL model (Log-Bilinear language model) for words, such as the LIWC (Linguistic Inquiry and two applications - image retrieval given sentence Word Count) tool (Pennebaker et al., 2001). Our queries, and image captioning. Lower perplexity primary research questions in this paper are: was achieved on text conditioned on images rather Q1:Can Affect-LM be used to generate affective than language models trained only on text. sentences for a target emotion with varying de- In contrast, previous literature on affective lan- grees of affect strength through a customizable guage generation has not focused sufficiently on model parameter? customizable state-of-the-art neural network tech- Are these generated sentences rated as emo- Q2: niques to generate emotional text, nor have they tionally expressive as well as grammatically cor- quantitatively evaluated their models on multiple rect in an extensive crowd-sourced perception ex- emotionally colored corpora. Mahamood and Re- periment? iter(2011) use several NLG (natural language gen- Does the automatic inference of affect cate- Q3: eration) strategies for producing affective medi- gory from the context words improve language cal reports for parents of neonatal infants under- modeling performance of the proposed Affect-LM going healthcare. While they study the difference over the baseline as measured by perplexity? between affective and non-affective reports, their The remainder of this paper is organized as fol- work is limited only to heuristic based systems and lows. In Section2, we discuss prior work in the do not include conversational text. Mairesse and fields of neural language modeling, and generation Walker(2007) developed PERSONAGE, a sys- of affective conversational text. In Section3 we tem for dialogue generation conditioned on ex- describe the baseline LSTM model and our pro- traversion dimensions. They trained regression posed Affect-LM model. Section4 details the ex- models on ground truth judge’s selections to au- perimental setup, and in Section5, we discuss re- tomatically determine which of the sentences se- sults for customizable emotional text generation, lected by their model exhibit appropriate extrover- perception studies for each affect category, and sion attributes. In Keshtkar and Inkpen(2011), the perplexity improvements over the baseline model authors use heuristics and rule-based approaches 635 for emotional sentence generation. Their gener- scribed by the following equation: ation system is not training on large corpora and P (wt = i ct 1, et 1) = they use additional syntactic knowledge of parts | − − T T of speech to create simple affective sentences. In exp (Ui f(ct 1) + βVi g(et 1) + bi) − − contrast, our proposed approach builds on state-of- V T T j=1 exp(Uj f(ct 1) + βVj g(et 1) + bj) − − the-art approaches for neural language modeling, (3) utilizes no syntactic prior knowledge, and gener- P ates expressive emotional text. et 1 is an input vector which consists of affect − category information obtained from the words in 3 Model the context during training, and g(.) is the output of a network operating on et 1.Vi is an embed- − 3.1 LSTM Language Model ding learnt by the model for the i-th word in the Prior to providing a formulation for our pro- vocabulary and is expected to be discriminative of posed model, we briefly