MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models

Home , Anima Anandkumar, Lexical analysis, Megatron, Transformers

Peng Xu‡∗, Mostofa Patwary§, Mohammad Shoeybi§, Raul Puri§, Pascale Fung‡, Anima Anandkumar§, Bryan Catanzaro§ ‡The Hong Kong University of Science and Technology, §NVIDIA [email protected], [email protected]

Abstract Context [FEMALE] was on a long road trip . Guan et al.(2020) she had just driven into town . she had been driving for hours . she was so tired she decided Existing pre-trained large language models to take a nap . when she woke up , she still have shown unparalleled generative capabili- had n’t slept well . ties. However, they are not controllable. In MEGATRON- she was on her way to see a strange country. CNTRL-124M she had never seen anything like that before. this paper, we proposeM EGATRON-CNTRL, keyword: driving finally she saw a small town that was so small. a novel framework that uses large-scale lan- it was a great little town with a nice park ! guage models and adds control to text gener- MEGATRON- she was driving in the middle of [PLACE]. all ation by incorporating an external knowledge CNTRL-8B of a sudden the tire pressure light came on. she base. Our framework consists of a keyword keyword: driving got out to check her tire. it was flat so she used the roadside assistance. predictor, a knowledge retriever, a contextual MEGATRON- she really wanted to see a few attractions. the knowledge ranker, and a conditional text gen- CNTRL-8B-ANT first one she saw was a giant water park. it was erator. As we do not have access to ground- keyword: attract amazing. it ended up being a fun experience. truth supervision for the knowledge ranker, we make use of weak supervision from sentence Table 1: Stories generated by models of increas- embedding. The empirical results show that ing capacity and controllability. As the model size our model generates more fluent, consistent, grows, story quality becomes increasingly coherent, and coherent stories with less repetition and fluent, and logically consistent. The last row demon- higher diversity compared to prior work on the strates howM EGATRON-CNTRL-8B-ANT model con- ROC story dataset. We showcase the controlla- trols the story generation with a new keyword, “attract”. bility of our model by replacing the keywords Note that [MALE] and [FEMALE] denote names and used to generate stories and re-running the [PLACE] denotes locations. generation process. Human evaluation results show that 77.5% of these stories are success- lack of knowledge which humans use to produce fully controlled by the new keywords. Further- natural text. For example, GPT-2 based models more, by scaling our model from 124 million produce degraded generations that are illogical and to 8.3 billion parameters we demonstrate that ungrammatical for knowledge-driven generation larger models improve both the quality of gen- tasks, such as story generation. Guan et al.(2020) eration (from 74.5% to 93.0% for consistency) therefore introduced commonsense knowledge to and controllability (from 77.5% to 91.5%). the pre-trained language model by further finetun- 1 Introduction ing on commonsense datasets. Although implicit encoding of knowledge is helpful for knowledge Text generation has recently attracted significant incorporation, there is still a lack of training mech- attention from the research community as large pre- anism to teach the model when and what to incor- trained language models, such as GPT-2 (Radford porate from external knowledge. et al., 2018, 2019) demonstrated promising results In addition, these large pre-trained language for generating long, grammatically correct, and flu- models are hard to control. Recently, plug-and-play ent text. Finetuning these models has shown signif- language models Dathathri et al.(2019) addressed icant improvements in downstream tasks, such as whole document controllability by adding a lin- persona chat (Wolf et al., 2019). However, one non- ear classifier on top of GPT-2 to predict whether negligible drawback of these large models is the generated text observes a particular style or prop- ∗ This work was done during the internship of Peng Xu at erty. Keskar et al.(2019) controlled a 1.2B pa- NVIDIA. Corresponding authors: Peng Xu, Mostofa Patwary. rameter language model generation via the use of

2831 Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2831–2845, November 16–20, 2020. c 2020 Association for Computational Linguistics

Figure 1: Overview of our generation process. Based on an input context, we generate keywords for future context, use the keywords to retrieve the relevant knowledge from an external knowledge-base, filter them based on their relevance to the context, and use the top scored knowledge sentences to guide the generation. control codes prepended to the model input. Boyd Summary of Contributions: et al.(2020) controlled the personality of a dialogue agent by conditioning it on prior conversations of a • We propose a novel generation framework that al- target actor. However, these controlling conditions lows dynamical incorporation of external knowl- are predefined, limited in their capability, and are edge into language model as well as control for only used once at the beginning to condition the text generation. generation of the rest of the document. They do not provide control granularity at either a sentence • Using both automatic metrics as well as human or sub-document level. evaluations, we demonstrate that our model gen- In this work, we address these shortcomings and erates more fluent, consistent, and coherent sto- develop an efficient controllable text generation ries with lower repetition rate and higher diversi- framework that we apply to the story generation ties compared to the previous state-of-the-art on task. In order to provide manual control to users ROC story datasets (Mostafazadeh et al., 2016). through a set of interpretable keywords, we first • We showcase the controllability of our model develop a keyword predictor model for the next by replacing the keywords used to generate sto- sentence. These keywords are then used to retrieve ries. Human evaluation results show that up to knowledge sentences from an external knowledge 91.5% of the generated stories are successfully base. Not all the retrieved knowledge is relevant to controlled by the new keywords . the story context and often it is noisy. To this end, we introduce a novel contextual ranker that ranks • We scale our model from 124 million to 8.3 bil- knowledge sentences based on the relevance to the lion parameters and demonstrate that both quali- context. As we do not have access to ground-truth ties, as well as controllability of the generations, supervision for this contextual knowledge ranker, improve as the model size increases. we make use of sentence embedding for weak supervision. The top-ranked knowledge sentences 2 Framework from the knowledge ranker are then fed to the conditional text generator to guide generation. By In our problem setup, we complete a story using giving the knowledge in addition to the context, we the first sentence as input, similar to Guan et al. provide rich information for the generator to attend (2020). We augment the generation process with an to and help the model better understand the ratio- external knowledge-base and develop a methodol- nale between sentences. Table1 shows an example ogy that can guide and control the story generation. of controllable story generation with increasing Our approach consists of the following four steps model capacity. connected together as shown in Figure1:

2832 i i 1. Given the story context, a keyword predictor as the knowledge associated with s , where rj de- model first predicts a set of keywords for the notes the j-th knowledge sentence associated si. next sentence yet to be generated. The number of knowledge sentences v varies per sentence and can be zero. Note that v 6= q because 2. A knowledge retriever then takes the gen- a keyword can have multiple knowledge triples erated keywords and queries an external associated with it. Given this notation, we de- knowledge-base where each knowledge triple fine the story context Xi = {x1, ··· , xi} where is converted into natural language “knowledge xi = [Ri, si]. The goal of this work is to generate sentences” using templates. xi given Xi−1, that is to first predict the knowledge i i i 3. A contextual knowledge ranker then ranks the R contained in s and then predict s itself. external knowledge sentences based on their 2.1 Keyword Predictor Model relevance to the story context. To provide manual control to users, we first develop 4. Finally, a generator takes both the story con- a keyword predictor model. Given the current story text as well as the top-ranked knowledge sen- context Xi−1, the model predicts a set of keywords tences as input and generates the next sentence Ki for the next sentence yet to be generated. The in the story. The output sentence is appended prediction of keywords instead of directly predict- to the story context and steps 1-4 are repeated. ing knowledge triples not only allows us to control the generation in an interpretable manner, but it This formulation naturally allows controllability by also helps to greatly reduce the search space for the replacing the keyword prediction process with man- knowledge triples. We formulate this keyword pre- ual external keywords. This work uses dynamic diction problem similar to a left-to-right language planning of the keywords and knowledge at each model where the goal is to predict the string of generation step. This allows the users to participate concatenated keywords: and control the generation on the go. As a result, q they don’t need to pre-specify the keywords explic- Y p(Ki|Xi−1) = p(ki |Xi−1,Ki ), (1) itly. We also note that it is challenging to statically j words are ambiguous si−1,USE encoder U,RAKE keywords extractor, and and can have multiple senses. We, therefore, con- knowledge base G textualize the knowledge sentences in Rî to obtain i Output: Pseudo Label of R relevant and useful ones under Xi−1. To do this, i i 1: Extract keywords K from s using RAKE we develop a contextual knowledge ranker. The R¯ = {T (t)|t ∈ G ∃ki ∈ Ki ki ∈ t} 2: Find and j , s.t. j model is trained with pseudo-labels extracted with ¯ r 3: Encode each r¯j ∈ R to Uj using USE i s access to the future sentence s as described in Sec. 4: Encode [si−1, si] to U r 2.3. 5: Compute cosine similarity score between each Uj and U s We use a BERT model to encode both the context 6: return r¯ s with the top N highest score i−1 i î j X and each knowledge sentence rˆj ∈ R . To adapt to the format of BERT, we append a [SEP] token to each Rj and sj inside the context Xi−1. A [CLS] token is then added to the beginning of 2.3 Building Pseudo Label of Ri Xi−1. For segment ids, we mark the tokens from the knowledge base as 0 and those from the story i−1 i The main challenge for controlling generation with as 1. The representation of X and rˆj are then knowledge is that we have no explicit access to the obtained after applying a linear layer on top of the hidden, latent controlling knowledge humans use embedding of the [CLS] token: i to supervise their story writing. That means R , i−1 V = W BERT (X ), the knowledge associated with si is not available. x 1 CLS i We, therefore, propose to use a weakly supervised Vj = W2 BERTCLS(ˆrj), signal to build the pseudo labels of Ri from si. where W and W are learnable weights. We then We hypothesize that Ri should 1) overlap with si 1 2 calculate the relevance score C between Xi−1 and in terms of keywords and 2) have strong connec- rî using the inner product between V and V as : tions to both the preceding sentence si−1 and si. j x j We include si−1 along with si because it is hard i i−1 i Cj = C(X , rˆj) = VxVj (2) to retrieve appropriate knowledge using only si i due to the ambiguity of natural language. We also We take R (Sec. 2.3) as positive samples and i i did not include other previous context beyond si−1 Rˆ \R as negative samples to train our ranker. as additional context overwhelms the information Given a positive and a negative knowledge sentence rp and rn, we define the ranking loss as contained in si. i−1 i−1 Following our hypothesis, we first extract key- L = max{0,M − C(X , rp) + C(X , rn)} (3) i i words K from s using RAKE (Rose et al., 2010) where M is a margin and determined empirically. i and then match K with all knowledge triples in Algorithms2 describe the ranker training process. G. Transforming the retrieved triples into knowl- At inference time, we simply calculate Cj for all edge sentences gives us our set of R¯i. We then take i î i rˆj ∈ R , sort them based on Cj score and pick the i i−1 the sentence s and s , concatenate them, and top N most relevant knowledge sentences as Ri. encode them using the Universal Sentence Encoder (USE)(Cer et al., 2018), a widely-used toolkit for 2.5 Conditional Generator s i−1 i semantic similarity, U = U([s , s ]), where we The conditional generator is a language model that i ¯i denote the encoder of USE as U. For each r¯j ∈ R , incorporates the controlling knowledge and gener- s we then calculate the cosine similarity between U ates the following sentences. It concatenates the r ¯i i−1 i and Uj = U(¯rj) and sort R based on this score. story context X and controlling knowledge R i i We take the top N highest scores r¯j as a pseudo as input and generates s .A GPT-2 transformer is label of Ri. Algorithm1 describes this process. used to model this conditional probability distribu- During the training phase of each following model, tion. We describe the concatenated input represen- we use this pseudo label of Ri to represent Ri. tation in the Appendix A.5.

2834 Algorithm 2 Knowledge Ranker Training the same settings. Parameters: BERT model parameters Θ and ranker To train our contextual knowledge ranker, we set model parameters W1 and W2 l the margin to 5.0. We set the number of knowledge Input: A story S with l sentences and a knowledge i base G sentences in R to 10. Therefore, for a given story 1: Initialize Θ using a pre-trained BERT model and context, the top 10 retrieved knowledge sentences W1,W2 randomly. from ConceptNet according to USE are chosen as 2: Dataset D = ∅ the positive samples. We further select 40 nega- 3: Call Algorithm1 to retrieve R1 from G using s1. 4: for i ∈ {2, . . . , l} do tive samples to compute our margin loss. We then 5: Call Algorithm1 to retrieve Ri using si. randomly sample 50 (positive, negative) pairs for 6: Get Rî using knowledge retrieval (Section 2.2) each story context to train our contextual knowl- 7: for j ∈ {1,...,N} do edge ranker. In total, we used ∼15 million pairs i i i 8: Sample rp from R and rn from Rˆ \R i−1 for training and ∼1 million pairs for validation. 9: D = D ∪ (X , rp, rn) 10: end for After training our ranker, we achieve a validation 11: end for accuracy of 0.9. 12: for (X, rp, rn) ∈ D do 13: Calculate loss L using Equation3 3.3 Controllability Experiment Setup 14: Optimize BERT,W1,W2 15: end for To test the controllability of our model, we perform 16: return BERT,W1,W2 experiments where we change keywords to their antonyms. With antonyms, we expect maximal change to the story generation. To do that, we first 3 Experimental Setup use MEGATRON-CNTRL-124M to generate key- 3.1 Datasets words K and corresponding full story S. Then we i i identify the first keyword ka ∈ K from K whose We use the ROC story dataset (Mostafazadeh antonym is available at WordNet (Miller, 1995). If et al., 2016) for our experiments. It consists of i multiple antonyms for ka are available we sample 98,161 stories, where each story contains five sen- one with a uniform random probability. Afterwards, tences. 88,344/4,908/4,909 stories are used for we provide the start of story {s1, s2, ··· , si−1}, train/validation/test sets, respectively. Following the keywords shared with our original story Guan et al.(2020), for each sentence, delexicaliza- 1 2 i−1 i {K ,K , ··· ,K }, and the antonym of ka to tion is performed by replacing all the names and en- either MEGATRON-CNTRL-124M or larger mod- tities in stories with special placeholders, [MALE], els (e.g. MEGATRON-CNTRL-355M). We then [FEMALE], and [NEUTRAL] for male, female and let the model finish the generation. We refer unknown names and entities, respectively. Given to these generations as MEGATRON-CNTRL-ANT, the first sentence of each story, our model’s task is for example, we call the antonym generations from to generate the rest of the story. For our external MEGATRON-CNTRL-355M model as MEGATRON- knowledge base, we use ConceptNet (Speer and CNTRL-355M-ANT. Havasi, 2012), consists of 600k knowledge triples. 3.4 Baselines 3.2 Models We compare our model with the following state- We used Megatron-LM (Shoeybi et al., 2019) for of-the-art story generation models. (1) Plan and pre-trained BERT and GPT-2 models to initialize write (Yao et al., 2019): The authors use an our contextual knowledge ranker and generative LSTM-based model to first generate a sequence models, respectively. For the model configurations, of keywords for planning the story. These key- hidden size, number of layers, and attention heads, words are then used to condition the generation. we used the configurations of BERT and GPT-2 as (2) Knowledge enhanced GPT-2 (Guan et al., in Megatron-LM. For generation with our GPT-2 2020): This work is currently the SOTA for ROC models, we used a top-k sampling scheme (Fan story generation. It finetunes a pre-trained GPT-2 et al., 2018) with k = 40 and a softmax tempera- model with knowledge triples from commonsense ture of 0.7. We detail the training hyperparameters datasets. Similar to our method, the knowledge and the input representations for GPT-2 and BERT triples are converted to sentences with templates. in Appendix A.1& A.2 . Both the keyword predic- A multitask learning framework is then developed tor and the conditional sentence generator follow to further finetune the story generation task and

2835 classify corrupted stories from real ones. We do We set up these three evaluations as independent not compare to Fan et al.(2019) because Guan et al. AMT tasks to make sure the tasks do not influ- (2020) has already shown their model significantly ence each other and introduce spurious correlations outperforms Fan et al.(2019) and in this work, we between labels. To reduce noise in our labeling pro- compare to Guan et al.(2020). (3) GPT-2-124M: cess, we only accepted workers with an approval This baseline finetunes a GPT-2 model with a next rating over 90% and have over 1k accepted jobs. token prediction loss on the story. We further limited the location of the annotators to the United States. For each example, we explicitly 3.5 Evaluation ask them to spend at least 15 seconds to evaluate We conduct both automatic as well as human eval- coherency and 10 seconds to evaluate the other two uations to assess our generation. properties. In total, we randomly sample 200 stories and assign five annotators for each story. We 3.5.1 Automatic Evaluation adopted majority voting to make final decisions We use the following metrics to compare differ- among the five annotators. ent models: Repeat: measures the redundancy of the generated story by reporting the percentage 3.5.3 Human Evaluation on Controllability of the stories that contain at least one repeated 4- gram (Shao et al., 2019). Distinct: measures the To evaluate how controllable our model is, we con- diversity of generated stories by reporting the ratio duct another human evaluation just for controlla- between distinct 4-grams to all generated 4-grams. bility. We show the annotators the start of a story, Perplexity: In the inference phase, our models in- original keywords, and the corresponding genera- volve two steps of generation: (i) generate set of tion. We then show the antonyms of the keywords, knowledge sentences, Ri from story context Xi−1, along with the corresponding generated story, and (ii) generate story sentence, si from Xi−1 and Ri. ask the annotators if the new story has changed To report the perplexity of the conditional generator compared to the original story in accordance with we sample Ri sequentially before generating each the meaning of the keyword’s antonyms. The rest story sentence si and report the total perplexity of of the AMT settings for these experiments are the all sentences si for i ∈ [2, l] where l is the number same as our consistency experiments. of sentences in the story. 4 Results 3.5.2 Human Evaluation on Quality We conduct human evaluations on Amazon Me- In this section, we first perform automatic and hu- chanical Turk1 (AMT) to analyze the quality of man evaluations with different model sizes and our generations on three aspects: Fluency, Co- compare our framework to the existing baselines. herence, and Consistency. To evaluate fluency, We then evaluate the controllability of our model we show the annotators a pair of generated sto- and finally show ablation study varying GPT-2 and ries from two models. We ask them to evaluate BERT model sizes. The detailed configuration of each sentence independently and choose the story the model sizes are shown in Table2. We provide with better overall fluency. Fluency of a story is several generated stories in Appendix A.7 varying defined as a measure of intra-sentence linguistic the length of the given context. We use M-CNTRL quality and grammatical correctness taken over all to denote MEGATRON-CNTRL in the tables due to sentences of the story. For coherence, we provide the limited space. the same stories as in fluency but ask to choose the one with better inter-sentence causal and temporal Conditional Keyword Knowledge dependencies. We let the annotators choose tie for Model Name Generator Generator Ranker (GPT-2) (GPT-2) ( BERT) both fluency and coherence. M-CNTRL-124M 124M 124M 336M Different from the settings of fluency and coher- M-CNTRL-355M 355M 355M 336M ence, we only show one generated story to anno- M-CNTRL-774M 774M 774M 336M tators to evaluate consistency. They are required M-CNTRL-2B 2.5B 2.5B 336M M-CNTRL-8B 8.3B 2.5B 336M to choose whether the story is logically consistent, based on whether the story self contradicts or not. Table 2: Number of parameters of our models (M- CNTRL is the short form forM EGATRON-CNTRL). 1https://www.mturk.com/

2836 Source A Coherence ↑ Fluency ↑ Source B M-CNTRL-124M 78.5% - 13.0% 66.5% - 22.5% Yao et al.(2018) M-CNTRL-124M 46.0% - 39.0% 44.5% - 43.5% Guan et al.(2020) M-CNTRL-355M 56.0% - 30.5% 46.5% - 30.5% Guan et al.(2020) M-CNTRL-355M 52.0% - 31.5% 46.5% - 39.0%M-C NTRL-124M M-CNTRL-774M 44.5% - 41.5% 56.0% - 33.5%M-C NTRL-355M M-CNTRL-2B 50.5% - 30.5% 53.0% - 39.0%M-C NTRL-774M M-CNTRL-8B 46.0% - 39.5% 46.5% - 46.5%M-C NTRL-2B

Table 3: Pairwise comparison between our models and baselines. Percentages in the format “A% - B%” indicate how often annotators rank the samples from source A better than from source B for a given category, and vice versa. Percentage pairs do not sum to 100% as the annotators were allowed to choose “tie” as being of equal quality.M EGATRON-CNTRL-124M achieves better results than all baselines. Scaling the models shows better coherence and ﬂuency.

Consistency ↑ 1 2 i−1 Name PPL ↓ Repeat ↓ Distinct ↑ computing perplexity, [s , s , ··· , s ] are given (Human Eval) Ri R Xi−1 GPT-2-124M 6.98 27.2 74.1 69.5 ground truth tokens, but and all in must Yao et al.(2018) NA 13.3 63.7 49.0 be sampled from a distribution that is learned with Guan et al.(2020) 7.04 22.1 77.1 67.0 weak supervision. This sampling introduces noise M-CNTRL-124M 9.37 20.0 80.1 74.5 M-CNTRL-355M 8.02 19.9 81.6 75.5 and non-determinism that results in higher perplex- M-CNTRL-774M 6.58 21.3 81.6 80.5 ity. This discrepancy is not an issue when analyzing M-CNTRL-2B 6.31 21.2 82.6 89.0 automatic evaluation metrics within our model fam- M-CNTRL-8B 6.21 21.2 82.8 93.0 ily. When scaling our model from 124M up to 8B Table 4: Evaluation results for the previous state-of- parameters we see a consistent drop in PPL and the-art models as well as our algorithm at different increase in distinct. This shows larger models can sizes. Perplexity, repeat, and distinct are evaluated auto- generate better stories with more diversity. matically whereas consistency is obtained using human evaluations. Our smallest model with 124M parame- Human evaluation results are presented in last ters achieves better distinct and consistency score com- column of Table4 (consistency) and in Table pared to prior work. Increasing model size up to 8B 3. Comparing MEGATRON-CNTRL-124M to Yao improves perplexity, distinct, and consistency scores. et al.(2019), we achieve much better coherence, For reference, the ground truth human writing gives 7.6 fluency, and consistency scores, which shows the score for repeat and 88.9 for distinct. benefit of large pre-trained transformer models. Comparing MEGATRON-CNTRL-124M to Guan 4.1 Automatic and Human Evaluations et al.(2020) which uses a similar base model, we Table4 shows that our smallest model, find that fluency is similar, however we should note that Guan et al.(2020) is not controllable and our MEGATRON-CNTRL-124M achieves better distinct and consistency scores compared to model has significantly better coherence (+7.0%) previous work. For repetition, our model is worse in Table3 and consistency (+7.5%) in Table4. We than Yao et al.(2019) which was also observed in attribute this to the use of the retrieved knowledge, Ri Guan et al.(2020). The reason could be their small . By explicitly providing facts pertinent to the 8M model is better at learning short term statistics next sentence, the conditional generative model (e.g. 4-grams), while large models are better at can focus on just generating text. By comparison, learning long term dependencies. Compared to a standard autoregressive GPT-2 model is tasked with predicting both the topics and the text of the other GPT-2 based models (i.e. GPT-2-124M and next sentence. Guan et al.(2020)), MEGATRON-CNTRL-124M achieves lower repeat and higher distinct scores, Scaling this up, and comparing MEGATRON- hence our model generates less repetitive stories. CNTRL-355M to Guan et al.(2020), our We notice from Table4 that our perplexity (PPL) model significantly outperforms in all aspects. score is much higher than other GPT-2-based mod- Furthermore, a thorough comparison among els. Our hypothesis for why this occurs is that other MEGATRON-CNTRL-355M, MEGATRON-CNTRL- GPT-2-based methods directly model and report 774M, MEGATRON-CNTRL-2B, MEGATRON- P (si|s1, s2, ··· , si−1) while our conditional gen- CNTRL-8B shows that scaling the model size fur- erator models and reports P (si|Xi−1,Ri). When ther almost always improves the quality of genera-

2837 tion in terms of fluency, coherence and consistency. Name Repeat ↓ Distinct ↑ For consistency, our best model at 8B parameters M-CNTRL-124M (D) 20.04 80.14 M-CNTRL-124M w/o knowledge (D) 23.59 79.39 achieves a score of 93%. M-CNTRL-124M (S) 23.87 79.45 M-CNTRL-124M w/o knowledge (S) 23.98 79.61 4.2 Controllability Evaluation We evaluate the controllability by changing key- Table 7: Ablation studies of static (S) vs dynamic (D) words to their antonyms as detailed in Section planning strategy, with and without knowledge. 3.3& 3.5. Table5 shows repeat and distinct for MEGATRON-CNTRL-124M as well as the con- study of model size can be found in the Appendix trolled versions at three different sizes. Altering A.3. control with antonym keywords gives lower repeat 4.3.1 Planning Strategy and higher distinct scores than the original genera- In this section, we investigate the effects of plan- tion. As the model size increases, the repeat stays ning strategy in our framework. Yao et al.(2019) almost constant while distinct improves. These showed that static planning works better than dy- results show that changing keywords manually re- namic planning for LSTM-based models. To intro- sults in distinct and not repeated text. duce the static planning in our model, we predicted Name Repeat ↓ Distinct ↑ all the keywords and relevant knowledge sentences M-CNTRL-124M 20.0 80.1 from the starting sentence and generated the en- M-CNTRL-124M-ANT 17.8 80.9 tire stories. When we compare these generations M-CNTRL-355M-ANT 18.0 81.6 with the stories generated by dynamic planning, we M-CNTRL-8B-ANT 18.5 82.8 see in Table7 (first and third rows) that dynamic Table 5: Comparing controllability of the models by planning outperforms the static planning strategy changing the keywords to their antonyms. Controlled with higher distinction (+0.7%) and lower repeti- generations show less repetition and higher diversity tion (-3.8%) scores. This is due to direct guidance compared to the original one. over each sentence provided by the retrieved knowledge from dynamic planning . In contrast, in static Further supporting this hypothesis, evaluation of planning, the retrieved knowledge sentences are all controllability in Table6 shows that MEGATRON- predicted together at the beginning using only the CNTRL-124M-ANT achieves a high controllabil- starting sentence, which makes the supervision for ity score of 77.5%. This means that by changing each story sentence weaker and noisier. the keywords to their antonym, 77.5% of newly generated stories change their semantic content to 4.3.2 External Knowledge follow the new antonym keywords. We also show In this section, we investigate the importance of that larger models are better able to leverage key- retrieved knowledge. Table7 (first and second word control. Scaling the model size from 124M rows) shows that, when excluding the knowledge to 355M and 8B model further improves the con- from our framework (i.e. MEGATRON-CNTRL- trollability score to 84.5% and 91.5%, respectively. 124M w/o knowledge), distinction score decreases We again observe the quality (e.g. coherence) of by 0.8% and repetition increases by 3.6%, high- our controlled generation improves as the model lighting the importance of external knowledge in size scales to 8B. our approach. Unlike dynamic planning, we ob- Name Controllability ↑ serve that in static planning, the external knowledge M-CNTRL-124M-ANT 77.5% does not play an important role in the quality of M-CNTRL-355M-ANT 84.5% the generations and using or not using the knowl- M-CNTRL-8B-ANT 91.5% edge leads to similar qualities. This observation Table 6: Human evaluation for controllability by also confirms that knowledge needs to be planned changing keywords to their antonyms. Over 77% of dynamically. our generation changes according to the keywords. 5 Future Work

4.3 Ablation Studies Short story sentences in ROC story dataset limits In this section, we conduct the ablation study on our exploration from several potential research di- the planning strategy and external knowledge. The rections. For example, how long the control signal

2838 would propagate for longer generations? Investi- models have been used to finetune on both story gating this issue using longer story datasets (e.g. completion datasets and commonsense knowledge WRITINGPROMPTS (Fan et al., 2018)) is a subject to further improve the quality of story completion for future work. Other interesting direction may (Guan et al., 2020). However, few works concern include incorporating the structure-level control- the controllability of language model generation, lability by adding it as either an extra input for especially for the large pre-trained models that are the conditional generator or a multitask learning common in today’s literature. supervision for each sequence. Controllable Generation Controllable text gen- We also observed that in some cases during the eration has a wide range of applications, including generation, our model simply mentions the given controlling through persona (Zhang et al., 2018; word in the sentence, and talks about things that are Boyd et al., 2020), politeness (Niu and Bansal, not strictly related to or restricted by the given word. 2018), etc. Wiseman et al.(2018) presented con- For example, the generated story of MEGATRON- trolling generations by learning latent, discrete tem- CNTRL-8B in Table 15 only mentions the keyword plates from data. Fu et al.(2019) discovered the “realize” instead of centering around it. This is importance of pivot words that determines the sen- caused by the RAKE keywords extractor, which tence attributes and presented a lexical analysis does not always extract the keywords that represent framework. To control large pre-trained models, the sentence well. One way to mitigate this issue is Keskar et al.(2019) demonstrated the ability to con- to leverage longer context information to identify trol text generation through a wide range of aspects, better keywords which is subject of the future work. such as domains and links. Plug-and-play language models Dathathri et al.(2019) also address whole 6 Related Work document controllability by adding a linear classifier on top of GPT-2 to predict whether generated Knowledge Incorporation of knowledge into lan- text observes a particular style or property. Prabhu- guage models has shown promising results for moye et al.(2020) provides a good survey of five downstream tasks, such as factual correct genera- modules for control. Differing from these works, tion (Logan et al., 2019) , commonsense knowledge we control the generation through keywords backed graph construction (Bosselut et al., 2019), entity by external knowledge. typing (Zhang et al., 2019) and etc. More recently, several works have shown that inclusion of learned 7 Conclusion mechanisms for explicit or implicit knowledge can In this paper, we proposed a novel framework that lead to the state-of-the-art results in Question An- adds control to text generation with external knowl- swering (Guu et al., 2020; Karpukhin et al., 2020; edge. Our model first generates a set of keywords Lee et al., 2019; Lewis et al., 2020) and dialogue and a knowledge retriever then queries an external modeling (Roller et al., 2020). knowledge base for triples related to the keywords. Storytelling There are several different story- Based on the relevance to the story context, a con- telling tasks described throughout the literature. textual knowledge ranker ranks the retrieved knowl- Storytelling can be classified into story completion edge sentences and feeds the top ones to a condi- (Chen et al., 2019), story ending generation (Guan tional generator to generate the next story sentence. et al., 2019), story generation from prompts (Fan Experimental results on the ROC story dataset et al., 2018) or titles (Yao et al., 2019), and story showed that our model outperforms state-of-the-art generation from a given sentence (Guan et al., models by generating less repetitive, more diverse 2020). Different approaches have been developed and logically consistent stories. Human evalua- to model the structure of stories with storylines tion of the controllability of our model shows that (Yao et al., 2019), skeletons (Xu et al., 2018), 91.5% of generated stories are successfully con- Conditional Variational AutoEncoders (Wang and trolled by changing keywords to their antonym. In Wan, 2019) and a coarse-to-fine framework (Fan line with current trends, we also demonstrate that et al., 2019). Other works focus on incorporat- using larger pre-trained language models consis- ing commonsense knowledge into story generation tently improves both the quality of the generated with attention-based models (Guan et al., 2019; stories and controllability. Chen et al., 2019). Recently, pre-trained language

2839 References Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming-Wei Chang. 2020. Realm: Retrieval- Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- augmented language model pre-training. arXiv tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. preprint arXiv:2002.08909. 2019. Comet: Commonsense transformers for automatic knowledge graph construction. In Proceed- Vladimir Karpukhin, Barlas Oguz,˘ Sewon Min, Ledell ings of the 57th Annual Meeting of the Association Wu, Sergey Edunov, Danqi Chen, and Wen- for Computational Linguistics, pages 4762–4779. tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint Alex Boyd, Raul Puri, Mohammad Shoeybi, Mostofa arXiv:2004.04906. Patwary, and Bryan Catanzaro. 2020. Large scale multi-actor generative dialog modeling. arXiv Nitish Shirish Keskar, Bryan McCann, Lav R. Varsh- preprint arXiv:2005.06114. ney, Caiming Xiong, and Richard Socher. 2019. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Ctrl: A conditional transformer language model for Nicole Limtiaco, Rhomni St John, Noah Constant, controllable generation. Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Diederik P Kingma and Jimmy Ba. 2014. Adam: A et al. 2018. Universal sentence encoder. arXiv method for stochastic optimization. arXiv preprint preprint arXiv:1803.11175. arXiv:1412.6980.

Jiaao Chen, Jianshu Chen, and Zhou Yu. 2019. In- Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. corporating structured commonsense knowledge in 2019. Latent retrieval for weakly supervised Proceedings of the AAAI Con- story completion. In open domain question answering. arXiv preprint ference on Artiﬁcial Intelligence , volume 33, pages arXiv:1906.00300. 6244–6251. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Petroni, Vladimir Karpukhin, Naman Goyal, Hein- Hung, Eric Frank, Piero Molino, Jason Yosinski, and rich Kuttler,¨ Mike Lewis, Wen-tau Yih, Tim Rosanne Liu. 2019. Plug and play language mod- Rocktaschel,¨ et al. 2020. Retrieval-augmented gen- els: a simple approach to controlled text generation. eration for knowledge-intensive nlp tasks. arXiv arXiv preprint arXiv:1912.02164. preprint arXiv:2005.11401. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep Robert Logan, Nelson F Liu, Matthew E Peters, Matt bidirectional transformers for language understand- Gardner, and Sameer Singh. 2019. Barack’s wife ing. arXiv preprint arXiv:1810.04805. hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Annual Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hi- Meeting of the Association for Computational Lin- erarchical neural story generation. In Proceedings guistics, pages 5962–5971. of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), George A Miller. 1995. Wordnet: a lexical database for pages 889–898. english. Communications of the ACM, 38(11):39– 41. Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In Pro- Nasrin Mostafazadeh, Lucy Vanderwende, Wen-tau ceedings of the 57th Annual Meeting of the Asso- Yih, Pushmeet Kohli, and James Allen. 2016. Story ciation for Computational Linguistics, pages 2650– cloze evaluator: Vector space representation evalu- 2660. ation by predicting what happens next. In Proceed- ings of the 1st Workshop on Evaluating Vector-Space Yao Fu, Hao Zhou, Jiaze Chen, and Lei Li. 2019. Re- Representations for NLP, pages 24–29. thinking text attribute transfer: A lexical analysis. In Proceedings of the 12th International Conference on Tong Niu and Mohit Bansal. 2018. Polite dialogue gen- Natural Language Generation, pages 24–33. eration without parallel data. Transactions of the As- sociation for Computational Linguistics, 6:373–389. Aaron Gokaslan and Vanya Cohen. 2019. Openweb- text corpus. Shrimai Prabhumoye, Alan W Black, and Rus- lan Salakhutdinov. 2020. Exploring control- Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and lable text generation techniques. arXiv preprint Minlie Huang. 2020. A knowledge-enhanced pre- arXiv:2005.01822. training model for commonsense story generation. arXiv preprint arXiv:2001.05139. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- Jian Guan, Yansen Wang, and Minlie Huang. 2019. standing by generative pre-training. Story ending generation with incremental encoding and commonsense knowledge. In Proceedings of Alec Radford, Jeffrey Wu, Rewon Child, David Luan, the AAAI Conference on Artiﬁcial Intelligence, vol- Dario Amodei, and Ilya Sutskever. 2019. Language ume 33, pages 6473–6480. models are unsupervised multitask learners.

2840 Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Lili Yao, Nanyun Peng, Ralph M. Weischedel, Kevin Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Knight, Dongyan Zhao, and Rui Yan. 2018. Plan- Kurt Shuster, Eric M Smith, et al. 2020. Recipes and-write: Towards better automatic storytelling. for building an open-domain chatbot. arXiv preprint CoRR, abs/1811.05701. arXiv:2004.13637. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Stuart Rose, Dave Engel, Nick Cramer, and Wendy Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Cowley. 2010. Automatic keyword extraction from Yejin Choi. 2019. Defending against neural fake individual documents. Text mining: applications news. In Advances in Neural Information Process- and theory, 1:1–20. ing Systems, pages 9051–9062.

Zhihong Shao, Minlie Huang, Jiangtao Wen, Wenfei Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Xu, et al. 2019. Long and diverse text generation Szlam, Douwe Kiela, and Jason Weston. 2018. Per- with planning-based hierarchical variational model. sonalizing dialogue agents: I have a dog, do you In Proceedings of the 2019 Conference on Empirical have pets too? In Proceedings of the 56th An- Methods in Natural Language Processing and the nual Meeting of the Association for Computational 9th International Joint Conference on Natural Lan- Linguistics (Volume 1: Long Papers), pages 2204– guage Processing (EMNLP-IJCNLP), pages 3248– 2213. 3259. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. Ernie: Enhanced Mohammad Shoeybi, Mostofa Patwary, Raul Puri, language representation with informative entities. In Patrick LeGresley, Jared Casper, and Bryan Catan- Proceedings of the 57th Annual Meeting of the Asso- zaro. 2019. Megatron-lm: Training multi-billion ciation for Computational Linguistics, pages 1441– parameter language models using gpu model paral- 1451. lelism. arXiv preprint arXiv:1909.08053. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Robert Speer and Catherine Havasi. 2012. Represent- Salakhutdinov, Raquel Urtasun, Antonio Torralba, ing general relational knowledge in conceptnet 5. In and Sanja Fidler. 2015. Aligning books and movies: LREC, pages 3679–3686. Towards story-like visual explanations by watching movies and reading books. CoRR, abs/1506.06724. Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847.

Tianming Wang and Xiaojun Wan. 2019. T-cvae: Transformer-based conditioned variational autoen- coder for story completion. In Proceedings of the 28th International Joint Conference on Artiﬁcial In- telligence, pages 5233–5239. AAAI Press.

Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2018. Learning neural templates for text generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3174–3187.

Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149.

Jingjing Xu, Xuancheng Ren, Yi Zhang, Qi Zeng, Xi- aoyan Cai, and Xu Sun. 2018. A skeleton-based model for promoting coherence among sentences in narrative story generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4306–4315.

Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan- and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artiﬁcial In- telligence, volume 33, pages 7378–7385.

2841 A Appendices Name (a-b-c) PPL ↓ Repeat ↓ Distinct ↑ 2B-2B-336M 6.31 21.2 82.6 2B-2B-1.2B 6.35 19.7 81.2 A.1 GPT-2 Hyperparameters: 2B-774M-1.2B 6.33 20.4 81.4

We used the BPE subword tokenizer from Radford Table 8: Ablation studies varying keyword prediction et al.(2019) to tokenize each sentence of the ROC model (b) and ranking model (c) keeping the condi- story dataset. The maximum sequence length is set tional generator fixed (a). Increasing keyword predic- to 1024 learned positional embeddings. An Adam tion model reduces repetition. Larger ranking models optimizer (Kingma and Ba, 2014) with learning does not give consistently better scores as it may overfit rate of 0.0001 is utilized. We added dropout to to noise due to the weakly supervised labels. both the embedding layer and multi-head attention layers with 0.1 probability. Gradients were clipped A.4 Datasets Used for pre-trained Models to a global gradient norm of 5.0. We finetuned the The pre-trained GPT-2 models were trained on a GPT-2 models for 5 epochs and selected the best 174GB corpora including: Wikipedia (Devlin et al., one with the lowest perplexity on the validation set. 2018), OpenWebText (Gokaslan and Cohen, 2019), RealNews (Zellers et al., 2019), and CC-Stories A.2 BERT Hyperparameters: (Trinh and Le, 2018). For BERT models we include BooksCorpus (Zhu et al., 2015) in the train- We set the maximum sequence length to ing dataset. 512 learned positional embeddings. We used a WordPiece tokenizer with the A.5 Input Format bert-large-uncased vocabulary for to- For the format of Rj, we add a “ SEP ” string to kenization. The model was also optimized with j separate different knowledge sentences r in Rj. an Adam optimizer with a learning rate of 0.0001, k We add a “ EOK ” string to denote the end of but it used a weight decay of 0.01. Gradients knowledge sentences. are clipped to a global norm of 1.0. We also For the story context Xi−1 = added dropout to embedding layer and multi-head {x1, x2, ··· , xi−1} where xj = [Rj, sj], we attention layers with 0.1 probability. For the add a “OS” token to the end of each sj to selection of margin, we tried 0.1, 0.5, 1.0, and 5.0. denote the end of sentence. At the end of the The choice of 5.0 gives the best result. story, a “ |endoftext| ” token is appended. We then concatenated Xi−1 with Ri as the input to A.3 Model Size conditional generator.

In addition to analyzing the effect of scale on our A.6 Preprocessing conditional generative model, we also performed an ablation study on the model size of our GPT-2- We follow the implementation of Guan et al.(2020), based keyword predictor and BERT-based ranker to replace “ .” with “. ”. More details have already models. The results in Table8 show that increasing been provided in Section 3.1. the keyword model size from 774M to 2B reduces the repetition while keeping the other scores sim- A.6.1 Computation Resources ilar. Increasing the size of our contextual ranker We used up to 20 DGX-1V servers (a total of 160 from 336M to 1.2B reduces the repetition while Tesla V100 32GB GPUs) for our experiments. Our also decreasing the diversity. It is not clear which infrastructure is highly optimized for large scale one is better. We conjecture that as the positive sam- deep learning applications. The servers are con- ples, Ri, we used to train our contextual ranker are nected via NVSwitch with 300 GB/sec bandwidth weakly supervised, and the fact that we used tem- between GPUs inside a server and 100 GB/sec of plates to synthetically convert knowledge triples interconnect bandwidth between servers using 8 In- to knowledge sentences, scaling the model size finiBand adapters per server. As our story dataset is might be overfitting to noise. We, therefore, use small, the training of GPT-2 models usually takes the smaller, more computationally efficient model less than three hours. For the BERT model, it took with 336M parameters for ranker models in all our eight hours as we have ∼ 15 million samples to experiments. train.

2842 A.7 Generation Examples Context: [NEUTRAL] ’s blind date insisted on going A.7.1 Single Sentence Context to the new sushi place. We present five generation examples keeping the Guan et al.(2019) context fixed to one sentence. she accepted without saying a word . later , he picked out a sushi roll . [NEUTRAL] was so happy with Context: [MALE] and his friends are bored on a her new sushi rolls . she had to wait for him to come weekend day. back . Guan et al.(2019) MEGATRON-CNTRL-124M keyword: sure, date they decided to sneak into a movie . they got a bunch she decided to go to the restaurant. she ordered a of popcorn and a movie . then , they made it to a blowfish. they gave her a small grin. she had a great movie . now , they have a really good time . time. MEGATRON-CNTRL-8B keyword: thought, rude MEGATRON-CNTRL-124M keyword: go, decides it is going to be a big night. [MALE] and his friends [NEUTRAL] was nervous because he was worried decide to go out. they get drunk and have a great she would be rude. but at the table , she was sweet time. when they are done they feel happy. and friendly. and she was very nice , even though she ate little sushi. [NEUTRAL] wondered if maybe MEGATRON-CNTRL-8B keyword: go, decides blind dates were actually very wonderful ! they decided to go to the video game store. [MALE] picks out a new video game. [MALE] and his friends MEGATRON-CNTRL-8B-ANT keyword: uncertain have fun playing the new video game. [MALE] is no [NEUTRAL] was uncertain. she was not a sushi longer bored. person but agreed to try it. [NEUTRAL] and her blind date ordered salmon sushi. the salmon sushi MEGATRON-CNTRL-8B-ANT keyword: come was delicious. a new tv show begins to come on the local channel. [MALE] is excited to watch his favorite tv show. Table 11: Example 3 he watches his show and they all laugh together. [MALE] and his friends have a fun time watching their favorite show. Context: [MALE] hiked up the local hill. Table 9: Example 1 Guan et al.(2019) he stopped and picked up his brush . he used that brush to brush his teeth . afterwards , he was ready Context: [MALE] had a heavy duty backpack. to ride back home . he went on his way home . Guan et al.(2019) MEGATRON-CNTRL-124M keyword: top he took it to the airport . he checked his pockets . he got a great view from the top. his legs felt great. [MALE] did not have a bag . [MALE] was sad . he climbed up the mountain and felt much better. he went back to the hill and reached the top. MEGATRON-CNTRL-124M keyword: took he took it to the store one morning. he put it on the MEGATRON-CNTRL-8B keyword: top shelf at the store. a coworker looked at it. then he he was at the top of the hill looking down. the weather told [MALE] he hated lugging around. looked beautiful. the clouds suddenly passed. the sun began to shine over the land again. MEGATRON-CNTRL-8B keyword: took he took it to the army base camp. he packed it with MEGATRON-CNTRL-8B-ANT keyword: bottom items like food , ammo , and tools. he took it to the he saw a huge boulder at the bottom. he could n’t jungle every day. one day , he took it out and his believe its size. he had to climb to the top of the hill. backpack exploded. he was afraid he would fall if he stood on top.

MEGATRON-CNTRL-8B-ANT keyword: abstain Table 12: Example 4 he abstained from buying another backpack. [MALE] used the old backpack for years. his old backpack was a big burden. [MALE] bought a new backpack with greater weight capacity.

Table 10: Example 2

2843 Context: her favorite glasses were ruined. Context: [MALE] liked trading commodities. one day [MALE] was trading several products. [MALE] Guan et al.(2019) noticed that he gained a lot of money from the new she was really upset . she had no choice but to buy trades. new ones . she went to the mall to shop and found a great pair . she bought them and was so happy . MEGATRON-CNTRL-124M keyword: lost, money [MALE] invested the money into stocks. [MALE] MEGATRON-CNTRL-124M keyword: wore lost all of his trading proﬁts. she wore them every day. then she had a horrible accident. her sister had broken them. she had to buy MEGATRON-CNTRL-8B keyword: realize new ones. [MALE] realized that the new trades were his best trading idea. [MALE] quit his old job and went full MEGATRON-CNTRL-8B keyword: wore time on his new trading idea. she wore them daily. she got them at the shoe store. she found them and they were her favorite. after a MEGATRON-CNTRL-8B-ANT keyword: keep while they got broken. [MALE] decided to keep doing this. now [MALE] had a successful trading career. MEGATRON-CNTRL-8B-ANT keyword: refresh she was desperate to refresh them. she found a recipe Table 15: The context contains three sentences. online. she poured bleach in them. she was glad to have functional glasses. Context: we went to a dinner and show one night. Table 13: Example 5 the ﬁrst guy we saw was really nice. after that we asked him out. he agreed.

MEGATRON-CNTRL-124M keyword: got A.7.2 Varying Number of Sentences in the we got into a bad argument. Context MEGATRON-CNTRL-8B keyword: NONE (no key- We present three generation examples varying the words) we had a really good time. number of sentences in the context. MEGATRON-CNTRL-8B-ANT keyword: end we ended up dating exclusively. Context: [MALE] and his friends are bored on a weekend day. it is going to be a big night. Table 16: The context contains four sentences.

MEGATRON-CNTRL-124M keyword: go, decides [MALE] and his friends decide to go out. they get A.8 Human Evaluation Interface for drunk and have a great time. when they are done they Annotators feel happy. Below we provide the interfaces used for human MEGATRON-CNTRL-8B keyword: decides evaluations (coherence, ﬂuency, logical consis- [MALE] decides to watch horror movies. his friends tency, and controllability). join him. they watch horror movies all night.

MEGATRON-CNTRL-8B-ANT keyword: stop they stop by a local arcade after the baseball game. [MALE] turns on his favorite arcade game. [MALE] ’s favorite game makes him feel happy.

Table 14: The context contains two sentences.

2844 2845