Gender Bias in Coreference Resolution Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme Johns Hopkins University Abstract We present an empirical study of gender bias in coreference resolution systems. We first in- troduce a novel, Winograd schema-style set of minimal pair sentences that differ only by pro- noun gender. With these Winogender schemas, we evaluate and confirm systematic gender bias in three publicly-available coreference Figure 1: Stanford CoreNLP rule-based coreference resolution systems, and correlate this bias with system resolves a male and neutral pronoun as coref- real-world and textual gender statistics. erent with “The surgeon,” but does not for the corre- sponding female pronoun. 1 Introduction the style of Winograd schemas, wherein a pro- There is a classic riddle: A man and his son get noun must be resolved to one of two previously- into a terrible car crash. The father dies, and the mentioned entities in a sentence designed to be boy is badly injured. In the hospital, the surgeon easy for humans to interpret, but challenging for looks at the patient and exclaims, “I can’t operate data-driven systems (Levesque et al., 2011). In on this boy, he’s my son!” How can this be? our setting, one of these mentions is a person That a majority of people are reportedly unable referred to by their occupation; by varying only 1 to solve this riddle is taken as evidence of un- the pronoun’s gender, we are able to test the im- derlying implicit gender bias (Wapman and Belle, pact of gender on resolution. With these “Wino- 2014): many first-time listeners have difficulty as- gender schemas,” we demonstrate the presence signing both the role of “mother” and “surgeon” to of systematic gender bias in multiple publicly- the same entity. available coreference resolution systems, and that As the riddle reveals, the task of coreference occupation-specific bias is correlated with em- resolution in English is tightly bound with ques- ployment statistics. We release these test sen- tions of gender, for humans and automated sys- tences to the public.2 tems alike (see Figure 1). As awareness grows In our experiments, we represent gender as a of the ways in which data-driven AI technolo- categorical variable with either two or three possi- gies may acquire and amplify human-like biases ble values: female, male, and (in some cases) neu- (Caliskan et al., 2017; Barocas and Selbst, 2016; tral. These choices reflect limitations of the textual Hovy and Spruit, 2016), this work investigates and real-world datasets we use. how gender biases manifest in coreference reso- lution systems. 2 Coreference Systems There are many ways one could approach this In this work, we evaluate three publicly- question; here we focus on gender bias with re- available off-the-shelf coreference resolution sys- spect to occupations, for which we have corre- tems, representing three different machine learn- sponding U.S. employment statistics. Our ap- ing paradigms: rule-based systems, feature-driven proach is to construct a challenge dataset in 2https://github.com/rudinger/ 1The surgeon is the boy’s mother. winogender-schemas 8 Proceedings of NAACL-HLT 2018, pages 8–14 New Orleans, Louisiana, June 1 - 6, 2018. c 2018 Association for Computational Linguistics statistical systems, and neural systems. number counts over 85GB of web news. For ex- ample, according to the resource, 9.2% of men- Rule-based In the absence of large-scale data tions of the noun “doctor” are female. The re- for training coreference models, early systems re- source was compiled by bootstrapping coreference lied heavily on expert knowledge. A frequently information from the dependency paths between used example of this is the Stanford multi-pass pairs of pronouns. We employ this data in our sieve system (Lee et al., 2011). A deterministic analysis. system, the sieve consists of multiple rule-based models which are applied in succession, from 3 Winogender Schemas highest-precision to lowest. Gender is among the set of mention attributes identified in the very first Our intent is to reveal cases where coreference stage of the sieve, making this information avail- systems may be more or less likely to recognize a able throughout the system. pronoun as coreferent with a particular occupation based on pronoun gender, as observed in Figure 1. Statistical Statistical methods, often with mil- To this end, we create a specialized evaluation set lions of parameters, ultimately surpassed the per- consisting of 120 hand-written sentence templates, formance of rule-based systems on shared task in the style of the Winograd Schemas (Levesque data (Durrett and Klein, 2013; Bjorkelund¨ and et al., 2011). Each sentence contains three refer- Kuhn, 2014). The system of Durrett and Klein ring expressions of interest: (2013) replaced hand-written rules with simple feature templates. Combinations of these features 1. OCCUPATION , a person referred to by their implicitly capture linguistic phenomena useful for occupation and a definite article, e.g., “the resolving antecedents, but they may also uninten- paramedic.” tionally capture bias in the data. For instance, for occupations which are not frequently found in the 2. PARTICIPANT , a secondary (human) partic- data, an occupation+pronoun feature can be highly ipant, e.g., “the passenger.” informative, and the overly confident model can exhibit strong bias when applied to a new domain. 3. PRONOUN , a pronoun that is coreferent with either OCCUPATION or PARTICIPANT. Neural The move to deep neural models led to more powerful antecedent scoring functions, and We use a list of 60 one-word occupations ob- the subsequent learned feature combinations re- tained from Caliskan et al. (2017) (see supple- sulted in new state-of-the-art performance (Wise- ment), with corresponding gender percentages man et al., 2015; Clark and Manning, 2016b). available from the U.S. Bureau of Labor Statis- Global inference over these models further im- tics.4 For each occupation, we wrote two simi- proved performance (Wiseman et al., 2016; Clark lar sentence templates: one in which PRONOUN is and Manning, 2016a), but from the perspective coreferent with OCCUPATION, and one in which of potential bias, the information available to the it is coreferent with PARTICIPANT (see Figure 2). model is largely the same as in the statistical mod- For each sentence template, there are three PRO- els. A notable exception is in the case of sys- NOUN instantiations (female, male, or neutral), tems which make use of pre-trained word embed- and two PARTICIPANT instantiations (a specific dings (Clark and Manning, 2016b), which have participant, e.g., “the passenger,” and a generic been shown to contain bias and have the potential paricipant, “someone.”) With the templates fully to introduce bias into the system. instantiated, the evaluation set contains 720 sen- tences: 60 occupations 2 sentence templates per × Noun Gender and Number Many coreference occupation 2 participants 3 pronoun genders. resolution systems, including those described × × here, make use of a common resource released by Validation Like Winograd schemas, each sen- Bergsma and Lin (2006)3 (“B&L”): a large list of tence template is written with one intended cor- English nouns and noun phrases with gender and rect answer (here, either OCCUPATION or PAR- 3This data was distributed in the CoNLL 2011 and 2012 450 are from the supplement of Caliskan et al. (2017), an shared tasks on coreference resolution. (Pradhan et al., 2011, additional 7 from personal communication with the authors, 2012) and three that we added: doctor, firefighter, and secretary. 9 (1a) The paramedic performed CPR on the passenger 100 even though she/he/they knew it was too late. (2a) The paramedic performed CPR on the passenger even though she/he/they was/were already dead. 80 (1b) The paramedic performed CPR on someone even though she/he/they knew it was too late. 60 (2b) The paramedic performed CPR on someone even though she/he/they was/were already dead. 40 Figure 2: A “Winogender” schema for the occupation paramedic. Correct answers in bold. In general, OC- CUPATION and PARTICIPANT may appear in either or- 20 der in the sentence. % Female by Occupation in Text (Bergsma and Lin, 2006) 0 TICIPANT).5 We aimed to write sentences where 0 20 40 60 80 100 % Female by Occupation (Bureau of Labor Stats, 2015-16) (1) pronoun resolution was as unambiguous for humans as possible (in the absence of additional Figure 3: Gender statistics from Bergsma and Lin context), and (2) the resolution would not be af- (2006) correlate with Bureau of Labor Statistics 2015. fected by changing pronoun gender. (See Figure However, the former has systematically lower female percentages; most points lie well below the 45-degree 2.) Nonetheless, to ensure that our own judgments line (dotted). Regression line and 95% confidence in- are shared by other English speakers, we vali- terval in blue. Pearson r = 0.67. dated all 720 sentences on Mechanical Turk, with 10-way redundancy. Each MTurk task included 5 sentences from our dataset, and 5 sentences 4 Results and Discussion from the Winograd Schema Challenge (Levesque We evaluate examples of each of the three coref- et al., 2011)6, though this additional validation erence system architectures described in 2: the step turned out to be unnecessary.7 Out of 7200 Lee et al. (2011) sieve system from the rule- binary-choice worker annotations (720 sentences based paradigm (referred to as RULE), Durrett 10-way redundancy), 94.9% of responses agree × and Klein (2013) from the statistical paradigm with our intended answers. With simple major- (STAT), and the Clark and Manning (2016a) deep ity voting on each sentence, worker responses reinforcement system from the neural paradigm agree with our intended answers for 718 of 720 (NEURAL).