Connotation Frames of Power and Agency in Modern Films Maarten Sap Marcella Cindy Prasetio Ari Holtzman Hannah Rashkin Yejin Choi Paul G. Allen School of Computer Science & Engineering University of Washington, Seattle, USA fmsap,mcp21,ahai,hrashkin,[email protected] Abstract power power + agency The framing of an action influences how The man with the roses beckons Irene forward. we perceive its actor. We introduce conno- power power + agency power agency tation frames of and , a prag- Another man steps in behind her, trapping her... matic formalism organized using frame se- + agency mantic representations, to model how dif- She slices upwards with a razor-sharp knife... ferent levels of power and agency are im- The move ends with Irene's finger over her own mouth... plicitly projected on actors through their — agency actions. We use the new power and He , eyes bulging. agency frames to measure the subtle, but obeys prevalent, gender bias in the portrayal of Figure 1: An excerpt from a box-office hit, Sher- modern film characters and provide in- lock Holmes (2009). Bolded words are the predi- sights that deviate from the well-known cates, solid underlined phrases are the agent of the Bechdel test. Our contributions include verb, and dash underlined words are the theme. an extended lexicon of connotation frames The full example with additional nuanced discus- along with a web interface that provides sion is available in Figure6 in the appendix. a comprehensive analysis through the lens of connotation frames. predicate-specific connotative relationships as an extension to Rashkin et al.(2016)’s connotation 1 Introduction frame lexicon. For instance, in Figure1, the verb A viewer’s impression of a movie character is in- “beckoning” implies that its theme (Irene) has less fluenced by how they are written and portrayed, power than its agent (the man). In the third line, which can in turn influence how people form Irene displays strong agency when she “slices” in stereotypes on gender norms (Behm-Morawitz self-defense. In contrast, when the man “obeys”, and Mastro, 2008). A character’s actions can be the man has low implied agency. projected with varying levels of power and agency, Using the new connotation lexicon, we present depending on the specific verbs used. For instance, a quantitative study to reveal the subtle, but preva- 1 somebody who “accepts” things is implied to be a lent gender bias in modern films. Going beyond passive decision-maker (or of lower agency) than the surface level analysis such as screen time or somebody who “assesses” things. While not ex- number of female characters (Google, 2017), our plicitly stated, these connotative meanings pro- study aims for a more focused and precise anal- jected by different verbs can influence the assump- ysis of power differentials between fictional men tions the audience makes about the people being and women. described. These assumptions can have negative In summary, our major contributions include the consequences if they reinforce negative stereo- creation and release of a lexicon with two new types (Walton and Spencer, 2009). connotative dimensions: power and agency and an To formalize this implicit information about 1We acknowledge that gender lies on a spectrum, and re- people projected by actions, we introduce power ducing it to a male-female binary is simplistic, however our and agency connotation frames, two new types of data limits a more complex understanding of gender. He implored the tribunal to show mercy. power(AG<TH) power(AG>TH) VERB AGENT THEME implore power(AG < TH) The princess waited for her prince. VERB AGENT THEME wait agency(AG)=− agency(AG)=+ agency(AG) = - Figure 2: The formal notation of the connotation frames of power and agency. The first example shows the relative power differential implied by the verb “implored”, i.e., the agent (“he”) is in a position of less power than the theme (“the tri- Figure 3: Sample verbs in the connotation frames bunal”). In contrast, “He demanded the tribunal with high annotator agreement. Size is indicative show mercy” implies that the agent has authority of verb frequency in our corpus (bigger = more over the theme. The second example shows the frequent), color differences are only for legibility. low level of agency implied by the verb “waited”. one another. For example, if the agent “dom- interactive demo website of our findings (see Fig- inates” the theme (denoted as power(AG>TH)), ure5 in the appendix for a screenshot). 2 Further- then the agent is implied to have a level of control more, as will be seen in Section 4.1, connotation over the theme. Alternatively, if the agent “hon- frames offer new insights that complement and de- ors” the theme (denoted as power(AG<TH)), the viate from the well-known Bechdel test (Bechdel, writer implies that the theme is more important or 1986). In particular, we find that high-agency authoritative. We used AMT crowdsourcing to la- women through the lens of connotation frames are bel 1700 transitive verbs for power differentials. rare in modern films. It is, in part, because some With three annotators per verb, the inter-annotator movies (e.g., Snow White) accidentally pass the agreement is 0.34 (Krippendorff’s α). Bechdel test and also because even movies with Agency The agency attributed to the agent of the strong female characters are not entirely free from verb denotes whether the action being described the deeply ingrained biases in social norms. implies that the agent is powerful, decisive, and capable of pushing forward their own storyline. 2 Connotation Frames of Power and For example, a person who is described as “ex- Agency periencing” things does not seem as active and de- We create two new connotation relations, power cisive as someone who is described as “determin- and agency (examples in Figure3), as an expan- ing” things. AMT workers labeled 2000 transi- sion of the existing connotation frame lexicons.3 tive verbs for implying high/moderate/low agency Three AMT crowdworkers annotated the verbs (inter-annotator agreement of 0.27). We denote with placeholders to avoid gender bias in the con- high agency as agency(AG)=+, and low agency text (e.g., X rescued Y; an example task is shown as agency(AG)=−. in the appendix in Figure7). We define the anno- Pairwise agreements on a hard constraint are tated constructs as follows: 56% and 51% for power and agency, respec- Power Differentials Many verbs imply the au- tively. Despite this, agreements reach 96% and thority levels of the agent and theme relative to 94% when moderate labels are counted as agree- ing with either high or low labels, showing that an- 2 http://homes:cs:washington:edu/˜msap/ notators rarely strongly disagree with one another. movie-bias/. 3The lexicons and a demo are available at http:// Some contributing factors in the lower KA scores homes:cs:washington:edu/˜msap/movie-bias/. include the subtlety of choosing between neutral Power label distribution Agency label distribution Frame β gender 80% 80% ∗∗ 78.1% agency(AG)=+ −0:951 M 72.8% 60% 60% power(AG>TH) −0:468 M∗∗ ∗∗ 40% 40% agency(AG)=− 0:277 F % verbs % verbs 20% 20% power(AG<TH) not sig. 13.5% 13.7% 10.6% 11.3% 0% 0% Theme Equal Agent Negative Neutral Positive Table 1: Power and agency connotation frames for male and female narratives, controlled for length Figure 4: Label distributions for power and of narrative text. β represents the change in log- agency based on the crowdsourced annotations. odds of a character being male/female were the corresponding frame to change by one unit. Sig- nificant results (∗∗ : p<:001) are in bold. “Male” and positive/negative as well as the skews in the was coded as 0, “Female” as 1. distributions of labels (i.e. more positive than neg- ative labels, see Figure4). Note that a similar dif- 3.1 Bias in Narratives ference between KA scores and soft percent agree- ment was found in our previous connotation frame Narratives describe what characters are doing. We work (Rashkin et al., 2016). investigate how they vary in terms of power and agency, using our connotation frames. We mea- 3 Bias in Movie Scripts sure how each standardized frame metric is asso- ciated with the gender of the character through a We use 772 movie scripts from (Gorinski and La- logistic regression, controlling for the total num- pata, 2015) as a test bed to validate our new con- ber of words that the character said, and correcting notation frames. Scripts have distinct structure, for multiple comparisons using Holm’s correction which allows us to easily parse narrations, dia- (Holm, 1979). logues and character names. Listed in Table1, our results show that male We automatically extract 21K male/female characters are portrayed with higher level of characters, using a name-gender list4 along with agency compared to women. Men are also por- gender specific lexicons (e.g., “actor”/“actresses”, trayed to have more authority than women as they “duke”/“duchess”) to automatically assign gender are more often the agent of powerful verbs. based on their first three narrations. To iden- This suggests that screenwriters tend to have fe- tify verbs with characters as their agent, we de- male characters contribute more to the aesthetic of pendency parse the narratives using the SpaCy5 the movie through low-agency verbs, rather than parser. Power and agency label distributions in our the plot, which is reminiscent of existing gender corpus are consistent with the annotation distribu- bias tests for movies (Yehl, 2013). tion (Figure4), and there is little variance across movies (see Figure8 in the appendix). 3.2 Bias in Character Expression In our dataset, there are nearly twice as many To further our validation of the new connota- men as there are women (34:6% women), in line tive dimensions, we look at how characters ex- with previous findings by Smith et al.(2015) press themselves in movies.