Suspense in Short Stories Is Predicted by Uncertainty Reduction Over
Total Page:16
File Type:pdf, Size:1020Kb
Modelling Suspense in Short Stories as Uncertainty Reduction over Neural Representation David Wilmot and Frank Keller Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK [email protected], [email protected] Abstract only sporadically been used in story generation sys- tems (O’Neill and Riedl, 2014; Cheong and Young, Suspense is a crucial ingredient of narrative fic- tion, engaging readers and making stories com- 2014). pelling. While there is a vast theoretical litera- Suspense, intuitively, is a feeling of anticipation ture on suspense, it is computationally not well that something risky or dangerous will occur; this understood. We compare two ways for mod- includes the idea both of uncertainty and jeopardy. elling suspense: surprise, a backward-looking Take the play Romeo and Juliet: Dramatic suspense measure of how unexpected the current state is is created throughout — the initial duel, the meet- given the story so far; and uncertainty reduc- ing at the masquerade ball, the marriage, the fight tion, a forward-looking measure of how unex- pected the continuation of the story is. Both in which Tybalt is killed, and the sleeping potions can be computed either directly over story rep- leading to the death of Romeo and Juliet. At each resentations or over their probability distribu- moment, the audience is invested in something be- tions. We propose a hierarchical language ing at stake and wonders how it will end. model that encodes stories and computes sur- This paper aims to model suspense in computa- prise and uncertainty reduction. Evaluating tional terms, with the ultimate goal of making it against short stories annotated with human sus- deployable in NLP systems that analyze or generate pense judgements, we find that uncertainty re- duction over representations is the best predic- narrative fiction. We start from the assumption that tor, resulting in near human accuracy. We also concepts developed in psycholinguistics to model show that uncertainty reduction can be used to human language processing at the word level (Hale, predict suspenseful events in movie synopses. 2001, 2006) can be generalised to the story level to capture suspense, the Hale model. This assumption 1 Introduction supported by the fact that economists have used is As current NLP research expands to include longer, similar concepts to model suspense in games (Ely fictional texts, it becomes increasingly important et al., 2015; Li et al., 2018), the Ely model. Com- to understand narrative structure. Previous work mon to both approaches is the idea that suspense has analyzed narratives at the level of characters is a form of expectation: In games, we expect to and plot events (e.g., Gorinski and Lapata, 2018; win or lose instead in stories, we expect that the Martin et al., 2018). However, systems that pro- narrative will end a certain way. cess or generate narrative texts also have to take We will therefore compare two ways for mod- into account what makes stories compelling and elling narrative suspense: surprise, a backward- enjoyable. We follow a literary tradition that makes looking measure of how unexpected the current And then? (Forster, 1985; Rabkin, 1973) the pri- state is given the story so far; and uncertainty re- mary question and regards suspense as a crucial duction, a forward-looking and measure of how factor of storytelling. Studies show that suspense is unexpected the continuation of the story is. Both important for keeping readers’ attention (Khrypko measures can be computed either directly over story and Andreae, 2011), promotes readers’ immersion representations, or indirectly over the probability and suspension of disbelief (Hsu et al., 2014), and distributions over such representations. We pro- plays a big part in making stories enjoyable and in- pose a hierarchical language model based on Gen- teresting (Oliver, 1993; Schraw et al., 2001). Com- erative Pre-Training (GPT, Radford et al., 2018) to putationally less well understood, suspense has encode story-level representations and develop an 1763 Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1763–1788 July 5 - 10, 2020. c 2020 Association for Computational Linguistics inference scheme that uses these representations to pense using general language models fine-tuned on compute both surprise and uncertainty reduction. stories, without planning and domain knowledge. For evaluation, we use the WritingPrompt corpus The advantage is that the model can be trained on of short stories (Fan et al., 2018), part of which we large volumes of available narrative text without annotate with human sentence-by-sentence judge- requiring expensive annotations, making it more ments of suspense. We find that surprise over rep- generalisable. resentations and over probability distributions both Other work emphasises the role of characters and predict suspense judgements. However uncertainty their development in story understanding (Bamman reduction over representations is better, resulting et al., 2014, 2013; Chaturvedi et al., 2017; Iyyer in near human-level accuracy. We also show that et al., 2016) or summarisation (Gorinski and Lap- our models can be used to predict turning points, ata, 2018). A further important element of narra- i.e., major narrative events, in movie synopses (Pa- tive structure is plot, i.e., the sequence of events palampidi et al., 2019). in which characters interact. Neural models have explicitly modelled events (Martin et al., 2018; Har- 2 Related Work rison et al., 2017; Rashkin et al., 2018) or the results of actions (Roemmele and Gordon, 2018; Liu et al., In narratology, uncertainty over outcomes is tradi- 2018a,b). On the other hand, some neural genera- tionally seen as suspenseful (e.g., O’Neill, 2013; tion models (Fan et al., 2018) just use a hierarchical Zillmann, 1996; Abbott, 2008). Other authors model on top of a language model; our architecture claim that suspense can exist without uncertainty follows this approach. (e.g., Smuts, 2008; Hoeken and van Vliet, 2000; Gerrig, 1989) and that readers feel suspense even 3 Models of Suspense when they read a story for the second time (Dela- 3.1 Definitions torre et al., 2018), which is unexpected if suspense is uncertainty; this is referred to as the paradox of In order to formalise measures of suspense, we suspense (Prieto-Pablos, 1998; Yanal, 1996). Con- assume that a story consists of a sequence of sen- sidering Romeo and Juliet again, in the first view tences. These sentences are processed one by one, suspense is motivated by primarily by uncertainty and the sentence at the current timepoint t is repre- over what will happen. Who will be hurt or killed in sented by an embedding et (see Section4 for how the fight? What will happen after marriage? How- embeddings are computed). Each embedding is ever, at the beginning of the play we are told “from associated with a probability P(et ). Continuations of the story are represented by a set of possible next forth the fatal loins of these two foes, a pair of star- i crossed lovers take their life”, and so the suspense sentences, whose embeddings are denoted by et+1. is more about being invested in the plot than not The first measure of suspense we consider is knowing the outcome, aligning more with the sec- surprise (Hale, 2001), which in the psycholinguis- ond view: suspense can exist without uncertainty. tic literature has been successfully used to predict We do not address the paradox of suspense directly word-based processing effort (Demberg and Keller, in this paper, but we are guided by the debate to 2008; Roark et al., 2009; Van Schijndel and Linzen, operationalise methods that encompass both views. 2018a,b). Surprise is a backward-looking predic- The Hale model is closer to the traditional model tor: it measures how unexpected the current word of suspense as being about uncertainty. In contrast, is given the words that preceded it (i.e., the left the Ely model is more in line with the second view context). Hale formalises surprise as the negative that uncertainty matters less than consequentially log of the conditional probability of the current different outcomes. word. For stories, we compute surprise over sen- In NLP, suspense is studied most directly in nat- tences. As our sentence embeddings et include ural language generation, with systems such as information about the left context e1;:::;et−1, we Dramatis (O’Neill and Riedl, 2014) and Suspenser can write Hale surprise as: (Cheong and Young, 2014), two planning-based Hale St = −logP(et ) (1) story generators that use the theory of Gerrig and Bernardo(1994) that suspense is created when a An alternative measure for predicting word-by- protagonist faces obstacles that reduce successful word processing effort used in psycholinguistics is outcomes. Our approach, in contrast, models sus- entropy reduction (Hale, 2006). This measure is 1764 forward-looking: it captures how much the current the next state et+1: word changes our expectations about the words we Ely = i 2 will encounter next (i.e., the right context). Again, Ut E[(et − et+1) ] i i 2 (4) we compute entropy at the story level, i.e., over sen- = P e e − e = ( t+1)( t t+1) tences instead of over words. Given a probability i i distribution over possible next sentences P(et+1), This is closely related to Hale entropy reduction, we calculate the entropy of that distribution. En- but again the entropy is computed over states (sen- tropy reduction is the change of that entropy from tence embeddings in our case), rather than over one sentence to the next: probability distributions. Intuitively, this measure captures how much the uncertainty about the rest i i H = − P e logP e of the story is reduced by the current sentence.