Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
Po-Sen Huang♠♦ Huan Zhang♥♥♦ Ray Jiang♠ Robert Stanforth♠ Johannes Welbl♠♣♥ Jack W. Rae♠♣ Vishal Maini♠ Dani Yogatama♠ Pushmeet Kohli♠
♠DeepMind ♥University of California, Los Angeles ♣University College London
Abstract Conditioning Text Generated Sentiment with Attribute Continuations Distribution Advances in language modeling architectures had a grand time organising...(0.97) and the availability of large text corpora have ... driven progress in automatic text generation. 're working on a My friend is a/ While this results in models capable of gener- prototype for her an _, and we... banana bread ating coherent texts, it also prompts models to recipe...(0.51) internalize social biases present in the training ... hear from her all corpus. This paper aims to quantify and reduce the time all the a particular type of bias exhibited by language problems...(0.17) models: bias in the sentiment of generated text. Given a conditioning context (e.g., a writing Figure 1: Conditioning text “My friend is a/an prompt) and a language model, we analyze if
65 Findings of the Association for Computational Linguistics: EMNLP 2020, pages 65–83 November 16 - 20, 2020. c 2020 Association for Computational Linguistics erate continuations with more positive sentiment • We evaluate the proposed methods using both for “baker”, and more negative sentiment with automatic metrics and human evaluations of sen- “accountant” as the occupation. When systemati- timent and semantic relevance, and find a strong cally evaluating this phenomenon by manipulating correlation between automatic metrics and hu- different sensitive attributes values (e.g., country man evaluations (§5). names, occupations, or person names) in the condi- tioning context – that is, performing counterfactual 2 Background & Related Work evaluation – we find that sentiment scores for the Bias in natural language processing systems. generated texts can vary substantially, suggesting Besides learning to favor the language of the au- the existence of sentiment bias. Such a sentiment thors’ demographic group (Hovy and Søgaard, bias can pose a concern for using the text generated 2015), NLP models can pick up on a variety of by language models in downstream applications cultural associations and undesirable social bi- (e.g., dialogue agents (Zhang et al., 2019)) from a ases (Caliskan et al., 2017). Systematic imbalances fairness perspective. were observed across NLP tasks, such as gender To quantify sentiment bias, we propose the use bias in coreference resolution (Zhao et al., 2018; of individual and group fairness metrics from the Rudinger et al., 2018), visual semantic role labeling fair machine learning literature (Dwork et al., 2012; (Zhao et al., 2017), image captioning (Hendricks Jiang et al., 2019; Hardt et al., 2016). We further- et al., 2018), and demographic biases in language more propose a general framework to reduce sen- generation (Sheng et al., 2019), text classification timent bias given a fairness specification based on (Dixon et al., 2018; Garg et al., 2019). Concretely sensitive attributes (e.g., fairness w.r.t. a predefined in sentiment analysis, Kiritchenko and Mohammad set of occupation names). Using this framework, (2018) found systematic biases with respect to race we propose embedding and sentiment prediction- and gender across more than 200 systems. derived regularization on the language model’s la- tent representations. Experiments demonstrate Mitigating bias in language models. Rather that both proposed methods reduce sentiment bias than debiasing word embeddings, Lu et al.(2018) while retaining a comparable level of perplexity proposed counterfactual data augmentation as a and semantic similarity, and show a trade-off be- remedy to occupation-specific gender biases, and tween fairness and semantic relevance. found that it can much better retain model perfor- While specifying concretely what optimal model mance than debiasing word embeddings, especially fairness behavior should be is difficult – it might be in language modeling. Zhao et al.(2019) and Basta defined by law or regulators – we provide a general et al.(2019) demonstrated gender bias in pretrained framework to address given fairness specifications language modeling representations (ELMo), which on sensitive attributes. Our main contributions are: translates into downstream tasks, but did not con- sider the language generated by the ELMo lan- • We demonstrate the existence of systematic guage model. Bordia and Bowman(2019), as well counterfactual sentiment bias in texts generated as Qian et al.(2019) identified biases in a language by large-scale language models (§3). modeling context and propose regularization strate- gies of generating certain words (e.g., “doctor”) • We propose two novel metrics: individual and with differently gendered inputs. group fairness metrics to quantify counterfactual In contrast to these prior works on mitigating sentiment bias in language generation (§3). gender biases of language models based on the probabilities of generating certain words (such as • To the best of our knowledge, this paper is the occupation ratios), we probe texts generated by lan- first to introduce a general framework to reduce guage models using a sentiment analysis system, bias under a specification measure (e.g., senti- similar to Sheng et al.(2019). We further propose ment) for texts generated by language models a general framework to mitigate bias for a given given sensitive attributes. While we focus on specification (e.g., fairness w.r.t. predefined coun- sentiment biases on a few common sensitive try names, occupations, gendered names) under a attributes (country, occupation and name), the specification measure (e.g., sentiment, regard, etc.). framework can be generalized to other specifica- Prior work mostly considers comparatively small tions (§4). language modeling training sets. In contrast, we
66 investigate bias in Transformer-based models with of the sensitive attribute from the set A\{a}, and a similar number of parameters (708 million pa- define the counterfactual input x˜ = cf(x, a, a˜) rameters) to GPT-2 (Solaiman et al., 2019) trained by replacing all occurrences of each sensitive to- on English news articles from WMT-19 (40GB of ken in φ(a) with the corresponding token in φ(˜a), text) and WikiText-103 (Merity et al., 2016). and leaving all other non-sensitive tokens of x un- changed. Given a predefined sentiment classifier Fairness. Popular statistical fairness criteria of- f with sentiment outputs in [0, 1], and a pretrained ten aim at achieving individual fairness (Dwork s language model LM, so that the random variable et al., 2012) or group fairness (Hardt et al., 2016) LM(x) is a sentence sampled from the language goals. In recent years, causal inference tools are model conditioned on x, we define the random vari- also used in fairness research to extend beyond sta- able S(x) = f (LM(x)) to be the sentiment score tistical fairness criteria making use of causal graphs. s in [0, 1] of the generated sentence, and denote its Similar to individual fairness, which requires simi- distribution by P (x). lar individuals to be treated similarly (Dwork et al., S Next, for counterfactual evaluation, we measure 2012), counterfactual fairness requires the same the difference between P (x) and P (x˜) as fol- model predictions before and after intervention on S S lows. When quantifying the difference between sensitive attributes in data-generating causal graphs two output distributions for a binary classifica- (Kusner et al., 2017; Kilbertus et al., 2017; Chiappa, tion problem – such as sentiment prediction – we 2019; Chiappa and Isaac, 2019). typically consider predictions formulated as yˆ = In our problem setting, we deviate from the 1(S > τ), given a decision threshold τ. One fun- counterfactual fairness works above by considering damental fairness concept is “demographic parity” counterfactual fairness (Garg et al., 2019) based for binary classification problems, which requires on a simple causal graph representing the language equal positive classification rates across subgroups, model instead of the data-generating process. We i.e., p(ˆy = 1 | A = a) = p(ˆy = 1 | A =a ˜) for aim towards counterfactual fairness by debiasing any sensitive attribute values a, a˜ ∈ A. We can the latent representation of inputs in the language measure deviation from it, i.e. “demographic dis- models, contributing to a family of methods to learn parity” using the differences between the subgroup fair representations (Beutel et al., 2017; Zemel positive rates: et al., 2013; Creager et al., 2019; Edwards and Storkey, 2016; Louizos et al., 2016) and enforcing p(ˆy = 1 | A = a) − p(ˆy = 1 | A =a ˜) independence between sensitive attributes and pre- (cf. Prop. 3.1 in Dwork et al.(2012)). However, diction outputs (Calders et al., 2009; Zhang et al., often we do not want our fairness goal to be de- 2018; Jiang et al., 2019; Chiappa et al., 2020). pendent on a predetermined decision threshold τ, since τ may be user-defined or simply not known at 3 Counterfactual Evaluation of training time. This consideration leads us to match Sentiment Bias output distributions, which is called “Strong De- Fairness specification. Our goal is to reduce the mographic Parity” (Jiang et al., 2019). Concretely counterfactual sentiment bias in a language model, applied in our LM context, these distributions are given a fairness specification. In our specification, PS(x|A = a) and PS(x˜|A =a ˜). we consider a set of sensitive attribute values (e.g., Extending this definition to measure unfairness country names, occupations, and person names) between counterfactual pairs of subgroups, demo- of a sensitive attribute (e.g., Country, Occupation, graphic disparity is the difference between posi- Name) that we want generated texts to be fair to tive sentiment rates of S(x) and S(x˜): |p(S(x) > under counterfactual evaluation. Formally, con- τ)−p(S(x˜) > τ)|. We can then measure the devia- sidering for example the sensitive attribute Gender, tion by computing the statistical disparity averaged we use A = {female, male} to denote the set of over uniformly random choices of τ ∈ [0, 1], that values considered, and use A = a to denote a ran- is, Eτ∼U[0,1]|p(S(x) > τ) − p(S(x˜) > τ)| where dom variable A that takes the sensitive attribute U denotes the random uniform distribution. This value a ∈ A. For each input sequence x contain- quantity is equal to the Wasserstein-1 distance be- ing sensitive tokens φ(a) (which are given in the tween PS(x) and PS(x˜) (Jiang et al., 2019): specification, e.g., φ(a)={he, his, him, husband, W1(PS(x),PS(x˜)) = (1) Paul} for a = male), we choose another value a˜ Eτ∼U[0,1]|p(S(x) > τ) − p(S(x˜) > τ)| 67 0.445 0.555 0.494 0.505 |A|(|A|−1) 2.00 2.00 where the inner sum is over all unordered 1.75 1.75 2 1.50 1.50 pairs of distinct a, a˜ ∈ A, and a, a˜ are values of the 1.25 1.25 m m 1.00 1.00 sensitive attribute in x and x˜ respectively. 0.75 0.75 0.50 0.50 Group Fairness Metric. This metric measures 0.25 0.25 0.00 0.00 fairness for particular subgroups. Concretely, the 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 evaluation sentences are separated into |A| = K (a) W (·, ·) =0.1 (b) W (·, ·) =0.01 1 1 disjoint subgroups, assigning a sentence to a sub- Figure 2: Illustration of the Wasserstein-1 distance- group a if it contains sensitive tokens from φ(a). based fairness metrics on two Gaussian distributions Taking for example the sensitive attribute Name and truncated to [0,1], simulating sentiment scores. For selecting A = {male, female}, we have K = 2, comparison, the Wasserstein-1 distance for the two sen- and φ(male) = {Jake, Scott, Jacob,...} for a = timent distributions in Figure1 is 0.13. male.2 For each subgroup a ∈ A, we then measure Sentiment bias by counterfactual evaluation, the Wasserstein-1 distance between the sentiment i.e., counterfactual sentiment bias, is then the distributions of all generated sentences of inputs a Wasserstein-1 distance between output sentiment from this subgroup, denoted by PS , and that over ∗ distributions PS of the original input x and its coun- the entire evaluation set, denoted by PS . We report terfactual x˜. Thus, extending Garg et al.(2019), the average of all these subgroup Wasserstein-1 we define a model to be counterfactually fair for distances as the Average Group Fairness metric, sentiment if denoted by G.F.:
1 X a ∗ W1(PS(x),PS(cf(x, a, a˜))) < (2) G.F. := W (P ,P ). (4) |A| 1 S S a∈A for each sensitive attribute value a ∈ A, a˜ ∈ A\{a}, and a chosen threshold > 0. This fair- 4 Language Models with Fair Sentiment ness formulation also expresses individual fairness Distribution which requires similar individuals to be treated sim- In this section, we introduce two approaches for ilarly (Dwork et al., 2012), where similar individu- reducing counterfactual sentiment bias in language als share similar non-sensitive words in a sentence. models, which will be subsequently evaluated with Note that using Wasserstein-1 distance to compare the above described fairness metrics. two distributions does not require assumptions on Given an input prefix x1:i with i tokens, x1:i = their shape (e.g., symmetry). (x1, ··· , xi), where the last token xi ∈ φ(a) is Fairness evaluation. For each sensitive attribute, associated with a subgroup with value a of the we measure the individual fairness and group fair- sensitive attribute, we construct a perturbed prefix ness metrics from distributions of sentiment scores by replacing xi with a token x˜i ∈ φ(˜a) from a different subgroup a˜, where fairness between the PS on the evaluation set in the following ways. Individual Fairness Metric. Based on the fair- two subgroups should be maintained. We obtain a ness property of the Wasserstein-1 distance (Eq. perturbed prefix x˜1:i = (x1:i−1, x˜i). 1), we compute the Average Individual Fairness To train the language model towards reducing by averaging the Wasserstein-1 distance between counterfactual sentiment bias, we want to ensure the sentiment score distribution of every evaluation that the language model produces similar senti- ment distributions for the two prefixes. Specifically, sentence PS(x) and each of its counterfactual sen- 1 we would like the Wasserstein-1 distance between tence PS(x˜) across all M templates. Formally, we define individual fairness metric (denoted by the sentiment distributions of generated sentences, I.F.) as: PS(x1:i) and PS(x˜1:i), to be small, as shown in Eq.2. But in practice, it is prohibitively expensive M 2 X X to sample a distribution of generated sequences for W (P (xm),P (x˜m)) M|A|(|A| − 1) 1 S S every x1:i and x˜1:i during training. Instead, we m=1 a,a˜∈A use hidden features from the language model as a (3) proxy to represent the distribution of future gener- 1 During inference, for each sensitive variable A we de- ated sequences, since p(xi+1, xi+2, · · · |x1:i) and sign a set of sentence templates to evaluate the counterfactual sentiment bias. See §5 for details. 2Here gender is treated as a binary variable.
68 p(xi+1, xi+2, · · · |x˜1:i) depend on the hidden states Fairness through sentiment regularization. of the language model conditioned on x1:i and x˜1:i, To overcome the above-mentioned drawback, respectively. we propose an alternative method for elimi- Concretely, we explore two approaches: Fair- nating sentiment bias using a sentiment classi- ness through embedding regularization and Fair- fier. Instead of measuring d(h(x1:i), h(x˜1:i)) ness through sentiment regularization, which ex- directly, we first apply a sentiment classifier ploit the hidden states of the language model. fsh to both h(x1:i) and h(x˜1:i), and measure
Given an L-layer transformer based language d(fsh (h(x1:i)), fsh (h(x˜1:i))) instead. Note that model with an input x1:i, we let h(x1:i) = the output of fsh can be multi-dimensional (e.g., (1) (L) h (x1:i), ··· , h (x1:i) denote the hidden fea- a hidden layer in the sentiment classifier), and we tures (or contextual embeddings) obtained by its can again measure the distance via cosine similar- hidden layers. ity. Applying the classifier fsh can be seen as a pro- Fairness through embedding regularization. jection from h(x) to a subspace that ideally only In this approach, we desire that the embed- contains sentiment-related information. If such a (j) (j) dings h (x1:i) and h (x˜1:i) are close, since perfect projection exists, we can regularize the sen- the joint distributions p(xi+1, xi+2, · · · |x1:i) and timent difference between the two inputs without p(xi+1, xi+2, · · · |x˜1:i) are determined by these em- losing other information of the sensitive tokens. On beddings. We call it the “embedding regulariza- the one hand, this classifier-based sentiment regu- tion” approach, and define the fairness loss as larization approach avoids the strong regularization a distance between the embeddings, denoted as of enforcing embedding similarity. On the other d(h(x1:i), h(x˜1:i)). We use the cosine distance: hand, the effectiveness of this method is correlated h¯(x )T h¯(x˜ ) with the quality of the sentiment classifier (or senti- 1:i 1:i 3 d(h(x1:i), h(x˜1:i)) := 1 − ment “projection”). The detailed implementation kh¯(x1:i)kkh¯(x˜1:i)k of fsh is introduced in AppendixB. This method where h¯(x) is set as the average of the last two can be extended to specifications with other spec- embedding vectors h(L−1)(x) and h(L)(x) based ification measures beyond sentiment by using a on the following two reasons: First, we want to corresponding classifier fsh . capture high-level semantics (e.g., sentiments) and Implementation: Three-step curriculum embedding in later layers represents higher level training. We use a three-step curriculum train- semantics (Tenney et al., 2019). Second, we ing schema. First, we train a language model using find that averaging too many layers can make the a regular cross-entropy loss for predicting the next ¯ ¯ difference between h(x1:i) and h(x˜1:i) very small, token given all the previous tokens, as done in a reducing the effectiveness of regularization. An typical language model training setting; a good val- advantage of this method is that it can directly idation perplexity ensures a relatively good hidden be applied to fairness specifications beyond senti- feature space has been learned. Second, using this ment, as it encourages p(xi+1, xi+2, · · · |x1:i) and language model, we train a sentiment classifier fsh p(xi+1, xi+2, · · · |x˜1:i) to be close regardless of the (e.g., a simple multilayer perceptron (MLP)) us- specification measure (e.g., sentiment). ing the extracted features from the language model. Since the embedding regularization method en- Since sentiment labels are generally unavailable for forces the model’s predictions to be similar for a large-scale corpus, we label the training data with the original input x1:i and the perturbed input x˜1:i the Google Cloud sentiment API4 and train a sen- without specification measure information, a po- timent classifier on the data with high magnitude. tential drawback of this method is that the regu- Third, with the fixed fsh from the previous step, larization can be too strong. As we require the we continue training on the subset of the original hidden representations (and thus the joint probabil- language model training set that contains any of the ities) to be as close as possible, this can lead to the sensitive tokens, with an additional fairness loss model learning to ignore the sensitive tokens, and Lfairness based on our “embedding regularization” thus generally a reduced dependence on them, as shown in Appendix C.6. Despite being completely 3We use a sentiment classifier as a proxy to measure sen- fair in this extreme case, model performance may timent scores/biases in this paper. The classifier itself might not be perfect and might exhibit some biases; for this reason suffer since the generated texts should ideally be we compare several alternatives. 4 contextually conditioned on xi or x˜i. https://cloud.google.com/natural-language/
69 Prediction Loss for Many tourists visit France for .... Extracted Documents Original sentence embeddings Classifier
Language Model Fairness Loss (e.g., sentiment classifier) (cosine similarity) Extracted Perturbed sentence embeddings Many tourists visit Italy for .... (optional)
Figure 3: Proposed language model debiasing pipeline (the third step in curriculum training). or “sentiment regularization” methods with a reg- Occupation, and Name) to measure the counter- ularization parameter λ. Meanwhile the language factual sentiment bias in language models. Coun- model is also trained on the regular cross-entropy try contains 10 country names and Occupation in- loss (LLM) on predicting the next token of the un- cludes 29 common occupations. For Name, we perturbed input x. Concretely, the loss function for have 17 female and 17 male common names. We an input sequence x during the third step is: list all sensitive attribute values used in our experi- ments in AppendixA. To compute the group fair- L(x) = L (x) + λ ·L (h(x ), h(x˜ )) LM fairness 1:i 1:i ness metric, we treat each country name and each We refer to this third step as the “debiasing step”, occupation as its own subgroup. For Name, we as illustrated in Figure3. Note that we do not use consider all female (male) names as one subgroup. any template at any step of training. 5 Experiments Sentence templates. For each sensitive attribute, We now evaluate our proposed sentiment regular- we design a set of M = 10 templates to evaluate ization and embedding regularization methods via counterfactual sentiment bias. Each m-th template both automatic scores and human evaluations. is a sentence prefix with length im, m = 1,...,M, containing a placeholder that will be replaced by a 5.1 Training details sensitive token in φ(a) for each sensitive attribute Model and datasets. We train two Trans- value a ∈ A. In other words, for each template formerXL (Dai et al., 2019) language models sim- we complete it by inputting the appropriate sensi- ilar in scale to GPT-2 (Radford et al., 2019) on tive token for every a ∈ A, forming a prefix x1:im a medium-scale corpus of Wikipedia articles (i.e., which is used as input to the language model to WikiText-103) and a large-scale corpus of English condition its generation on. We sample 1000 sen- news articles from the WMT-19 document-level tences conditioned on each input prefix, and we 5 translation task (WMT-19). We present dataset apply an external sentiment classifier fs on the gen- statistics, model architectures, and training details erated sentences. All templates are described in in AppendixB. AppendixA. Model selection. We train language models us- Employing specific templates for model evalua- ing both embedding-regularization and sentiment- tion is a commonly used practice (Zhao et al., 2018; regularization losses with different regularization Qian et al., 2019; Sheng et al., 2019), but we ac- strengths. Based on the losses in the validation knowledge that they can lack context-sensitivity, set, we report λ ∈ {1, 10, 100} for embedding- and that such evaluation is necessarily limited and regularization and λ ∈ {10, 100, 1000} for not comprehensive. Indeed, we see the advance- sentiment-regularization on WMT-19, and λ ∈ ment of model evaluation beyond specific tem- {1, 10, 100} for both embedding-regularization plates as an important open research problem. Note and sentiment-regularization on WikiText-103. that during the training process (see Figure3), we do not add any of the templates to the training set; 5.2 Fairness Specifications it is thus unlikely that our models overfit to them. Sensitive attributes and subgroups. We con- Importantly, the templates are used during evalua- sider three common sensitive attributes (Country, tion only and our models need to generalize to the 5http://data.statmt.org/news-crawl/ templates to be effective.
70 5.3 Evaluation Metrics articles with at least one sensitive token (PPLs). Sentiment analysis and fairness metrics. Cal- The perplexity on the whole test set reflects the culating the individual fairness (I.F.) and group language model’s overall performance. Since the sensitive tokens only exist in a small fraction of test fairness (G.F.) scores using Eq.3 and Eq.4 re- s quires sentiment scores from a sentiment classifier data, the subset perplexity PPL examines the lan- guage model performance specifically in contexts fs. We evaluate the generated sentences using three 6 sentiment classifiers: i) the Google Cloud senti- containing sensitive tokens. ment API ii) a BERT (Devlin et al., 2019)-based Semantic Similarity (“S.S.” and “S.S.c”). sentiment classifier fine-tuned on the SST dataset We compute the cosine similarity between the em- (Socher et al., 2013) resulting in 92.7% validation bedding of both the prefix and the generated contin- accuracy, and iii) a simple opinion-word-based sen- uations using the universal sentence encoder (Cer timent classifier, which counts the number of pos- et al., 2018). A generated continuation is consid- itive opinion words p and the number of negative ered semantically similar if the cosine similarity is opinion words n (Hu and Liu, 2004) and derives above a given threshold (set to 0.4; see Appendix its sentiment score as p/(p + n), and 0.5 if no C.7 for further details). The fraction of gener- opinion words exist. We include this simple clas- ated continuations with above-threshold similarity sifier as the Google Cloud sentiment API and the among all generated continuations then defines the BERT-based classifier may themselves contain bias, semantic similarity metric (denoted as “S.S.”). We which has been shown for many sentiment analysis report this S.S. as a proxy for whether the gener- systems (Kiritchenko and Mohammad, 2018). The ated sentences capture the original semantics. In opinion-word-based method, while being less ac- addition, we report the fraction of generated con- curate (69.6% accuracy on the SST validation set), tinuations mentioning the sensitive attribute tokens is less prone to giving biased judgments, as it does as a second proxy for semantic relevance (denoted not contain sensitive tokens or learned associations: as “S.S.c”). We also conduct a human evaluation it only relies on opinion words. Furthermore, since of semantic similarity, and find a strong correlation we also use the Google Cloud sentiment API to between semantic relevance and human judgments create the sentiment labels of the training data for (see §5.5). learning fsh , the BERT-based and opinion-word- based sentiment classifiers provide additional mea- 5.4 Evaluation Results sures of sentiment, helping to avoid findings spe- Fairness Improvements. In Figure4, we report cific to one sentiment classification system in par- the fairness metrics of the sensitive attribute Oc- ticular. We also conduct a human evaluation on cupation for models trained on the WMT-19 and the correlation between automatic sentiment scores WikiText-103 datasets. We evaluate the individ- and human judgments (see §5.5). ual fairness and group fairness metrics using a Language model performance One special set of sentences generated from the templates and case of a fair language model is to generate the prefixes given in AppendixA. Importantly, dur- same continuations regardless of the sensitive at- ing training we never explicitly train the model tribute tokens or prefixes (e.g., Appendix C.6). on these templates. The baseline model repre- However this deteriorates the original language sents the model after the first step of the curricu- model’s performance, and we expect the model to lum training, before any debiasing steps are per- still capture semantics related to the given sensitive formed. Each fairness metric is evaluated using tokens. Thus, in addition to the fairness metrics, three different sentiment classifiers: the BERT- it is important to examine the performance of lan- based and opinion-word-based classifier in Fig- guage models. Here, we evaluate perplexity and ures4 and5, and Google Cloud sentiment API semantic similarity for assessing language model in Appendix C.1. For embedding-regularization performance and generation relevance. 6We train all models to convergence. To rule out the differ- ent numbers of total training iterations as a potential confound- Perplexity (PPL) and subset perplexity ing factor between the fine-tuned and standard model, we also (PPLs). We report the perplexity (PPL) on the trained baseline models with this same additional number of iterations on standard training data. We found performance whole test set of WMT-19/WikiText-103, and the differences to be insignificant, both in terms of perplexity as perplexity on a subset of the test set that includes well as fairness metrics.
71 Baseline Sent- = 10 0.05 Baseline Sent- = 10 Baseline Sent- = 1 Baseline Sent- = 1 0.04 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 10 Embed- = 1 Sent- = 10 0.06 0.06 Embed- = 10 Sent- = 1000 0.04 Embed- = 10 Sent- = 1000 Embed- = 10 Sent- = 100 Embed- = 10 Sent- = 100 Embed- = 100 Embed- = 100 Embed- = 100 0.03 Embed- = 100 0.03 0.04 0.04
I.F. I.F. 0.02 G.F. G.F. 0.02
0.02 0.02 0.01 0.01
0.00 0.00 0.00 0.00 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) WMT-19, I.F. (b) WMT-19, G.F. (c) WikiText-103, I.F. (d) WikiText-103, G.F.
Figure 4: I.F. and G.F improvements on WMT-19 and WikiText-103 datasets for the Occupation attribute using a BERT-based sentiment classifier, for both embedding regularization (“Embed-λ”) and sentiment regularization (“Sent-λ”) methods under different regularization strengths λ. Note a lower I.F./G.F. is better.
0.05 Baseline Sent- = 10 0.030 Baseline Sent- = 10 0.06 Baseline Sent- = 1 Baseline Sent- = 1 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 10 Embed- = 1 Sent- = 10 0.025 0.03 0.04 Embed- = 10 Sent- = 1000 Embed- = 10 Sent- = 1000 0.05 Embed- = 10 Sent- = 100 Embed- = 10 Sent- = 100 Embed- = 100 Embed- = 100 Embed- = 100 Embed- = 100 0.020 0.04 0.03 0.02 0.015 0.03 I.F. I.F. 0.02 G.F. G.F. 0.010 0.02 0.01 0.01 0.005 0.01
0.00 0.000 0.00 0.00 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) WMT-19, I.F. (b) WMT-19, G.F. (c) WikiText-103, I.F. (d) WikiText-103, G.F.
Figure 5: Individual fairness score (I.F.) and group fairness score (G.F.) improvements on WMT-19 and WikiText- 103 datasets for the Occupation attribute, with the opinion-word-based classifier. Note a lower I.F./G.F. is better. and sentiment-regularization methods, we report WMT-19 Occupation WikiText-103 Occupation Model PPL PPLs S.S. S.S.c PPL PPLs S.S. S.Sc the performance of two methods with different reg- Baseline 17.9 18.0 17.9 9.9 18.9 21.4 40.3 24.3 λ = 1 17.6 17.6 12.8 5.6 18.4 20.9 24.4 3.7 Emb. ularization parameters for the fairness loss. Overall, 10 17.8 17.9 7.3 2.2 18.5 20.8 24.0 3.1 Reg. we observe that both proposed approaches achieve 100 18.5 18.5 5.9 1.8 18.4 20.8 23.7 3.9 λ = 1 - - - - 18.4 21.0 32.4 11.9 reduced bias in both individual fairness and group Sent. 10 17.6 17.7 14.5 6.4 18.4 20.9 28.2 8.9 fairness metrics compared to the baseline model. A Reg. 100 17.7 17.7 10.8 4.5 18.4 21.0 22.6 3.4 larger regularization parameter λ typically reduces 1000 17.9 17.9 8.4 2.4 18.4 21.0 22.8 2.0 the bias further. The results of sensitive attributes Table 1: Perplexity and semantic similarity scores of Country and Name can be found in Appendices C.2 WMT19 and WikiText-103 models for the Occupation and C.3, and the overall findings are similar to the attribute. A lower perplexity is better; higher semantic sensitive attribute Occupation discussed here. similarity scores (S.S. and S.S.c) are better. Trade-off between generation quality and fair- ness. In Table1, we present the perplexity 7 and better individual fairness scores than embedding semantic similarity of models in Figure4. Over- regularization based models. Both proposed ap- all, we observe a trade-off between fairness and proaches improve the individual fairness scores semantic similarity. significantly compared to the baseline models. The To further illustrate the trade-off between fair- sentiment regularization based models further im- ness and relevance of generated texts, in Figure6 prove the individual fairness score by a large mar- we show both semantic similarity (S.S.) and indi- gin while maintaining similar semantic similarity. vidual fairness scores (I.F.) under different regular- ization strengths for WMT-19 models in sensitive 5.5 Human Evaluation attributes Country, Occupation, and Name. We We perform a human evaluation to justify the use can observe that the sentiment regularization based of automatic measurements of both semantic rele- models achieve higher semantic similarity scores vance and sentiment, and the effectiveness of the than embedding regularization based models at a proposed fairness scores. We have 19 human anno- similar level of individual fairness score. On the tators, each annotator labels 50–100 sentences, and other hand, with similar semantic similarity scores, each sentence is rated by 2 human annotators. We the sentiment regularization based models achieve measure the Spearman’s correlations with human 7Since we do not further train our baseline model with the predictions in three settings, demonstrating that the additional epochs of the debiasing step, both PPL and PPLs can sometimes slightly improve, while improving fairness automatic metrics are positively correlated with measures. human judgment. Sentiment Scores: we evalu-
72 0.010 0.015 0.0150
0.015 0.020 0.0175
0.020 0.025 0.0200
0.0225 0.025 0.030 I.F. I.F. I.F. 0.0250 0.030 0.035 0.0275 Baseline Baseline Baseline 0.035 0.040 Embedding Reg. Embedding Reg. 0.0300 Embedding Reg. 0.040 Sentiment Reg. 0.045 Sentiment Reg. 0.0325 Sentiment Reg.
18 20 22 24 26 28 30 32 34 6 8 10 12 14 16 18 6 8 10 12 14 Semantic Simiarlity Semantic Simiarlity Semantic Simiarlity (a) WMT-19 Country (b) WMT-19 Occupation (c) WMT-19 Name
Figure 6: Trade-off between I.F. and S.S. using a BERT-based sentiment classifier. A lower I.F. is better (note that the y-axis is reversed); a higher S.S. is better. Each point represents a model trained using a certain λ. Overall, both embedding and sentiment regularization help reduce I.F., and sentiment regularization works better than embedding regularization. ate 100 sentences randomly generated by our lan- We have quantified the presence of sentiment bias guage model. The human annotation is then com- using our proposed novel fairness metrics based pared against the Google Cloud sentiment API, on Wasserstein distance, and demonstrated two BERT-based sentiment classifier, and the opinion- flexible methods to reduce counterfactual senti- word-based classifier. The Spearman’s correlations ment bias, while maintaining similar perplexity with human predictions are 0.75, 0.79, and 0.67, and generation semantics. For future work, the pro- respectively. Semantic Similarity: we present a posed framework could be extended to study coun- sensitive token, the associated prefix, and the 100 terfactual biases given other specifications (e.g., generated sentences from the language models to religion, ethnicity, age, or multiple-attribute cross- human annotators. We ask the annotators to label subgroups) that require fairness guarantees, and the relevance, and then compare human annotations could be used with other specification measures with “S.S.” and “S.S.c” scores; the Spearman’s cor- beyond sentiment. relations with human evaluation are 0.72 and 0.63, respectively. Fairness Scores: We compute the Acknowledgments I.F. score using sentiment scores from human eval- The authors thank the anonymous reviewers, Gabor´ uation for baseline and sentiment-regularization Melis, Stephen Clark, Chris Dyer, Jonathan Uesato, models in the following two settings. Firstly, we Martin Szummer, Silvia Chiappa, Andrew Strait, randomly sample 50 sentences from each of the pre- Emily Sheng, Sumanth Dathathri, and Cyprien de fixes “You are a designer, and you” and “You are Masson d’Autume for helpful feedback and com- an accountant, and you”. The I.F. scores are 0.333 ments for the paper. and 0.056 for baseline and sentiment-regularization models, respectively. Secondly, we use instead the prefixes “Libya is” and “Iceland is”, again sam- References pling 50 sentences from each. The I.F. score is reduced from 0.291 (baseline) to 0.155 (sentiment- Christine Basta, Marta R. Costa-jussa,` and Noe Casas. regularization). Both evaluations demonstrate that 2019. Evaluating the underlying gender bias in con- textualized word embeddings. In Proceedings of the our proposed method does indeed reduce sentiment First Workshop on Gender Bias in Natural Language bias – also under human evaluation. The annotation Processing, pages 33–39, Florence, Italy. Associa- instructions and details are shown in AppendixD. tion for Computational Linguistics. 6 Conclusion A. Beutel, J. Chen, Z. Zhao, and E. H. Chi. 2017. Data decisions and theoretical implications when As large-scale language models are increasingly adversarially learning fair representations. CoRR, deployed for real-world applications, developing abs/1707.00075. methods for assessing and mitigating bias with re- spect to sensitive attributes is an important area of Shikha Bordia and Samuel R. Bowman. 2019. Identify- ing and reducing gender bias in word-level language inquiry to enable pro-social outcomes. In this pa- models. In Proceedings of the 2019 Conference of per, we have studied counterfactual sentiment bias the North American Chapter of the Association for in texts generated by large-scale language models. Computational Linguistics: Student Research Work-
73 shop, pages 7–15, Minneapolis, Minnesota. Associ- H. Edwards and A. Storkey. 2016. Censoring repre- ation for Computational Linguistics. sentations with an adversary. In 4th International Conference on Learning Representations. T. Calders, F. Kamiran, and M. Pechenizkiy. 2009. Building classifiers with independency constraints. Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur In International Conference on Data Mining Work- Taly, Ed H. Chi, and Alex Beutel. 2019. Counterfac- shops, pages 13–18. tual fairness in text classification through robustness. In AIES, pages 219–226. ACM. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically M. Hardt, E. Price, and N. Srebro. 2016. Equality of from language corpora contain human-like biases. opportunity in supervised learning. In Advances in Science, 356(6334):183–186. Neural Information Processing Systems 29, pages 3315–3323. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, and Anna Rohrbach. 2018. Women also snowboard: et al. 2018. Universal sentence encoder. arXiv Overcoming bias in captioning models. In Proceed- preprint arXiv:1803.11175. ings of the European Conference on Computer Vi- sion (ECCV), pages 771–787. S. Chiappa. 2019. Path-specific counterfactual fairness. In Thirty-Third AAAI Conference on Artificial Intel- Dirk Hovy and Anders Søgaard. 2015. Tagging perfor- ligence, pages 7801–7808. mance correlates with author age. In Proceedings of the 53rd Annual Meeting of the Association for S. Chiappa and William S. Isaac. 2019. A Causal Computational Linguistics and the 7th International Bayesian Networks Viewpoint on Fairness, volume Joint Conference on Natural Language Processing 547 of IFIP AICT, pages 3–20. Springer Nature (Volume 2: Short Papers), pages 483–488, Beijing, Switzerland. China. Association for Computational Linguistics. Minqing Hu and Bing Liu. 2004. Mining and summa- S. Chiappa, R. Jiang, T. Stepleton, A. Pacchiano, rizing customer reviews. In Proceedings of the tenth H. Jiang, and J. Aslanides. 2020. A general ap- ACM SIGKDD International Conference on Knowl- proach to fairness with optimal transport. In Thirty- edge Discovery and Data Mining, pages 168–177. Fourth AAAI Conference on Artificial Intelligence. ACM.
Elliot Creager, David Madras, Jorn-Henrik¨ Jacobsen, Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Marissa A. Weis, Kevin Swersky, Toniann Pitassi, Jiang, and Silvia Chiappa. 2019. Wasserstein fair and Richard S. Zemel. 2019. Flexibly fair rep- classification. In Proceedings of the Thirty-Fifth resentation learning by disentanglement. CoRR, Conference on Uncertainty in Artificial Intelligence. abs/1906.02589. Niki Kilbertus, Mateo Rojas Carulla, Giambattista Zihang Dai, Zhilin Yang, Yiming Yang, William W Parascandolo, Moritz Hardt, Dominik Janzing, and Cohen, Jaime Carbonell, Quoc V Le, and Rus- Bernhard Scholkopf.¨ 2017. Avoiding discrimina- lan Salakhutdinov. 2019. Transformer-XL: Atten- tion through causal reasoning. In I. Guyon, U. V. tive language models beyond a fixed-length context. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- arXiv preprint arXiv:1901.02860. wanathan, and R. Garnett, editors, Advances in Neu- ral Information Processing Systems, pages 656–666. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Curran Associates, Inc. Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- Svetlana Kiritchenko and Saif Mohammad. 2018. Ex- standing. In Proceedings of the 2019 Conference amining gender and race bias in two hundred sen- of the North American Chapter of the Association timent analysis systems. In Proceedings of the for Computational Linguistics: Human Language Seventh Joint Conference on Lexical and Compu- Technologies, Volume 1 (Long and Short Papers), tational Semantics, pages 43–53, New Orleans, pages 4171–4186, Minneapolis, Minnesota. Associ- Louisiana. Association for Computational Linguis- ation for Computational Linguistics. tics.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, M. J. Kusner, J. R. Loftus, C. Russell, and R. Silva. and Lucy Vasserman. 2018. Measuring and mitigat- 2017. Counterfactual fairness. In Advances in Neu- ing unintended bias in text classification. In AIES, ral Information Processing Systems 30, pages 4069– pages 67–73. ACM. 4079.
C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and C. Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel. 2012. Fairness through awareness. In R. Zemel. 2016. The variational fair autoencoder. Proceedings of the 3rd Innovations in Theoretical In 4th International Conference on Learning Repre- Computer Science Conference, pages 214–226. sentations.
74 Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Aman- Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, charla, and Anupam Datta. 2018. Gender bias and Sameer Singh. 2019. Universal adversarial trig- in neural natural language processing. CoRR, gers for attacking and analyzing NLP. Empirical abs/1807.11714. Methods in Natural Language Processing.
Stephen Merity, Caiming Xiong, James Bradbury, and Rowan Zellers, Ari Holtzman, Hannah Rashkin, Richard Socher. 2016. Pointer sentinel mixture mod- Yonatan Bisk, Ali Farhadi, Franziska Roesner, and els. arXiv preprint arXiv:1609.07843. Yejin Choi. 2019. Defending against neural fake news. In Advances in Neural Information Process- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt ing Systems, pages 9051–9062. Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork. resentations. In Proceedings of the 2018 Confer- 2013. Learning fair representations. In Proceedings ence of the North American Chapter of the Associ- of the 30th International Conference on Machine ation for Computational Linguistics: Human Lan- Learning, pages 325–333. guage Technologies, Volume 1 (Long Papers), pages Brian Hu Zhang, Blake Lemoine, and Margaret 2227–2237, New Orleans, Louisiana. Association Mitchell. 2018. Mitigating unwanted biases with ad- for Computational Linguistics. versarial learning. In AAAI/ACM Conference on AI, Ethics, and Society, pages 335–340. ACM. Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. 2019. Reducing gender bias in word-level Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, language models with a gender-equalizing loss func- Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing tion. In Proceedings of the 57th Annual Meeting of Liu, and Bill Dolan. 2019. DialoGPT: Large-scale the Association for Computational Linguistics: Stu- generative pre-training for conversational response dent Research Workshop, pages 223–228, Florence, generation. arXiv preprint arXiv:1911.00536. Italy. Association for Computational Linguistics. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cot- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, terell, Vicente Ordonez, and Kai-Wei Chang. 2019. Dario Amodei, and Ilya Sutskever. 2019. Language Gender bias in contextualized word embeddings. In models are unsupervised multitask learners. OpenAI Proceedings of the 2019 Conference of the North Blog, 1(8). American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Rachel Rudinger, Jason Naradowsky, Brian Leonard, Volume 1 (Long and Short Papers), pages 629–634, and Benjamin Van Durme. 2018. Gender bias in Minneapolis, Minnesota. Association for Computa- coreference resolution. In Proceedings of the An- tional Linguistics. nual Meeting of the North American Association of Computational Linguistics (NAACL). Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or- donez, and Kai-Wei Chang. 2017. Men also like Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, shopping: Reducing gender bias amplification using and Nanyun Peng. 2019. The woman worked as corpus-level constraints. In Proceedings of the 2017 a babysitter: On biases in language generation. In Conference on Empirical Methods in Natural Lan- EMNLP-IJCNLP, pages 3405–3410. guage Processing, pages 2979–2989, Copenhagen, Denmark. Association for Computational Linguis- Richard Socher, Alex Perelygin, Jean Wu, Jason tics. Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or- for semantic compositionality over a sentiment tree- donez, and Kai-Wei Chang. 2018. Gender bias in bank. In EMNLP, pages 1631–1642. coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference Irene Solaiman, Miles Brundage, Jack Clark, Amanda of the North American Chapter of the Association Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, for Computational Linguistics: Human Language and Jasmine Wang. 2019. Release strategies and the Technologies, Volume 2 (Short Papers), pages 15–20, social impacts of language models. arXiv e-prints, New Orleans, Louisiana. Association for Computa- page arXiv:1908.09203. tional Linguistics.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In As- sociation for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, pages 5998–6008.
75 A Sensitive Attributes and Templates in 1. I got a new job as a/an
76 perplexity on the validation set. On the WMT-19 WMT-19 Country WikiText-103 Country Model PPL PPLs S.S. S.S.c PPL PPLs S.S. S.Sc dataset, we train a language model composed of 48 Baseline 17.9 18.7 33.9 23.0 18.9 18.0 49.5 31.1 λ = 1 18.0 18.7 29.7 20.9 19.4 18.4 36.4 8.0 Emb. layer transformers with an embedding size of 1024, 10 18.1 18.8 25.7 16.7 19.5 18.5 35.1 6.4 Reg. comprising 708 million parameters. The model 100 18.1 18.9 24.2 15.1 19.6 18.5 26.9 4.3 λ = 1 - - - - 19.5 18.5 36.8 18.4 achieved 17.46 perplexity on the validation set. Sent. 10 17.9 18.7 33.7 21.7 19.4 18.5 34.4 10.9 Reg. 100 18.0 18.8 29.0 19.6 19.4 18.4 29.7 5.2 Language model training (step 1 of curriculum 1000 18.1 18.9 23.7 12.8 19.5 18.6 24.2 2.1 training). For WMT-19, we train our model on Table 5: Perplexity and semantic similarity scores of 128 Google Cloud TPUv3 cores using the Adam WMT19 and WikiText-103 models for the Country at- −4 optimizer with a learning rate of 2.5 × 10 , a tribute. A lower perplexity is better; higher semantic batch size of 256 and a total of 5 × 105 training similarity scores (S.S. and S.S.c) are better. steps; for WikiText-103, we train our model on 128 Google Cloud TPUv3 cores using the Adam optimizer with a learning rate of 2.5×10−4, a batch WikiText-103, respectively. Due to the decrease size of 512, and a total of 2.5 × 105 training steps. of step size in this step, we find that sometimes For both datasets, we use a sequence length of 512 language model perplexity improves after step 3, per batch, and we keep the states (embeddings) despite adding the additional fairness loss. The for the latest 512 tokens in the transformer-based training time of this step is between 3–15 hrs, de- language models. pending on the amount of data that contains any of the sensitive tokens. Note our proposed approach Sentiment projection training (step 2 of cur- only requires an additional sentiment projection riculum training). We train a 3-layer MLP net- from hidden states and minimizing the regulariza- work with a hidden layer size 128 as the sentiment tion loss, which is scalable to large language mod- classifier fsh for the sentiment projection. To train els. the sentiment classifier, we create a training set by selecting a subset of the WMT-19 and WikiText- Sample generation. Using the sensitive at- 103 training set that are with absolute sentiment tributes and templates in AppendixA, we sample scores greater than 0.7 using the Google Cloud 1,000 sentences per template for a given sensitive sentiment API, which provides sentiment scores attribute value. We have 10 templates per sensitive between -1 and 1. There are 28,957,245 sentences attribute. In each sensitive attribute, we have tens for WMT-19 and 369,594 sentences for WikiText- of sensitive tokens. Throughout the sampling ex- 103. Note we train the sentiment classifier on the periments, we sample sentences with a maximum positive and negative sentiment classification task of 50 tokens. We sample with a temperature of 1.0. only, since we empirically found that training only C Additional Experimental Results on positive and negative sentiment data works bet- ter than training also with neutral sentiment data. C.1 Results on the Occupation attribute with We train the model on a single NVIDIA V100 GPU, the Google Cloud sentiment API and the training process takes around 14–21 hrs. In Section5, we present the results with the BERT- The accuracy of the sentiment classifier is 98.8% based and the opinion-word-based sentiment clas- and 98.7% for WikiText-103 and WMT-19, respec- sifier. In Figure7, we present individual fairness tively, on the subset of the validation set selected scores and group fairness scores under the same using the same procedure as the training set. setting of Occupation attributes on WMT-19 and Language model debiasing (step 3 of curricu- WikiText-103 datasets using the sentiment scores lum training). Since the language model has from Google Cloud sentiment API. We find that the achieved good validation perplexity in step 1, we trends are similar as observed in Section5, where decrease the learning rate and use a smaller number our two proposed methods can effectively improve of training steps in this step. For both datasets, we fairness metrics. reduce the learning rate to 2.5 × 10−5; we train WMT-19 for 5 × 104 steps, and train WikiText103 C.2 Results on the Country attribute for 2.5 × 104 steps for debiasing. For this step, In Figures8 and9 we report the individual fairness we only use 16 Google Cloud TPUv3 cores and and group fairness scores for the WMT-19 models reduce the batch size to 16 and 32 for WMT-19 and trained using our proposed embedding regulariza-
77 Baseline Sent- = 10 0.020 Baseline Sent- = 10 Baseline Sent- = 1 Baseline Sent- = 1 0.030 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 100 0.025 Embed- = 1 Sent- = 10 0.0125 Embed- = 1 Sent- = 10 Embed- = 10 Sent- = 1000 Embed- = 10 Sent- = 1000 Embed- = 10 Sent- = 100 Embed- = 10 Sent- = 100 0.025 0.015 Embed- = 100 Embed- = 100 0.020 Embed- = 100 0.0100 Embed- = 100 0.020 0.015 0.0075 0.015 0.010 I.F. I.F. G.F. G.F. 0.010 0.010 0.0050 0.005 0.005 0.005 0.0025
0.000 0.000 0.000 0.0000 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) I.F. (WMT-19) (b) G.F. (WMT-19) (c) I.F. (WikiText-103) (d) G.F. (WikiText-103)
Figure 7: Individual fairness score (I.F.) and group fairness score (G.F.) improvements on WMT-19 and WikiText- 103 datasets for the Occupation attribute, with the Google Cloud sentiment API. Note a lower I.F./G.F. is better.
Baseline Sent- = 10 Baseline Sent- = 10 0.035 Baseline Sent- = 10 0.06 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 100 0.030 Embed- = 1 Sent- = 100 0.06 0.05 Embed- = 10 Sent- = 1000 Embed- = 10 Sent- = 1000 Embed- = 10 Sent- = 1000 Embed- = 100 Embed- = 100 0.025 Embed- = 100 0.04 0.04 0.020
I.F. 0.03 I.F. I.F. 0.015 0.02 0.02 0.010
0.01 0.005
0.00 0.00 0.000 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) BERT, I.F. (b) Opinion-word, I.F. (c) Google-API, I.F.
Figure 8: Individual fairness score (I.F.) improvements on WMT-19 dataset for the Country attribute, evaluated with three sentiment classifiers. Note a lower I.F. is better.
Baseline Sent- = 10 Baseline Sent- = 10 Baseline Sent- = 10 0.04 0.05 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 100 0.020 Embed- = 1 Sent- = 100 Embed- = 10 Sent- = 1000 0.04 Embed- = 10 Sent- = 1000 Embed- = 10 Sent- = 1000 0.03 Embed- = 100 Embed- = 100 Embed- = 100 0.015 0.03 0.02
G.F. G.F. G.F. 0.010 0.02
0.01 0.01 0.005
0.00 0.00 0.000 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) BERT, G.F. (b) Opinion-word, G.F. (c) Google-API, G.F.
Figure 9: Group fairness score (G.F.) improvements on WMT-19 dataset for the Country attribute, evaluated with three sentiment classifiers. Note a lower G.F. is better.
Baseline Sent- = 1 Baseline Sent- = 1 Baseline Sent- = 1 0.04 0.030 Embed- = 1 Sent- = 10 Embed- = 1 Sent- = 10 Embed- = 1 Sent- = 10 0.015 Embed- = 10 Sent- = 100 0.025 Embed- = 10 Sent- = 100 Embed- = 10 Sent- = 100 0.03 Embed- = 100 Embed- = 100 Embed- = 100 0.020 0.010 0.02 0.015 I.F. I.F. I.F.
0.010 0.005 0.01 0.005
0.00 0.000 0.000 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) BERT, I.F. (b) Opinion-word, I.F. (c) Google-API, I.F.
Figure 10: Individual fairness score (I.F.) improvements on WikiText-103 dataset for the Country attribute, evalu- ated with three sentiment classifiers. Note a lower I.F. is better.
0.014 0.020 Baseline Sent- = 1 Baseline Sent- = 1 Baseline Sent- = 1 0.005 Embed- = 1 Sent- = 10 0.012 Embed- = 1 Sent- = 10 Embed- = 1 Sent- = 10 Embed- = 10 Sent- = 100 Embed- = 10 Sent- = 100 Embed- = 10 Sent- = 100 0.015 Embed- = 100 0.010 Embed- = 100 0.004 Embed- = 100
0.008 0.003 0.010
G.F. G.F. 0.006 G.F. 0.002 0.004 0.005 0.001 0.002
0.000 0.000 0.000 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) BERT, G.F. (b) Opinion-word, G.F. (c) Google-API, G.F.
Figure 11: Group fairness score (G.F.) improvements on WikiText-103 dataset for the Country attribute, evaluated with three sentiment classifiers. Note a lower G.F. is better.
78 Baseline Sent- = 10 Baseline Sent- = 10 Baseline Sent- = 10 0.04 0.025 0.05 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 100 Embed- = 10 Sent- = 1000 Embed- = 10 Sent- = 1000 Embed- = 10 Sent- = 1000 0.020 0.04 Embed- = 100 0.03 Embed- = 100 Embed- = 100
0.03 0.015 0.02 I.F. I.F. I.F. 0.02 0.010
0.01 0.01 0.005
0.00 0.00 0.000 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) BERT, I.F. (b) Opinion-word, I.F. (c) Google-API, I.F.
Figure 12: Individual fairness score (I.F.) improvements on WMT-19 dataset for the Name attribute, evaluated with three sentiment classifiers. Note a lower I.F. is better.
0.0150 Baseline Sent- = 10 0.008 Baseline Sent- = 10 Baseline Sent- = 10 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 100 Embed- = 1 Sent- = 100 0.0125 Embed- = 10 Sent- = 1000 Embed- = 10 Sent- = 1000 0.006 Embed- = 10 Sent- = 1000 0.006 Embed- = 100 Embed- = 100 Embed- = 100 0.0100 0.004 0.0075 0.004 G.F. G.F. G.F. 0.0050 0.002 0.002 0.0025
0.0000 0.000 0.000 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) BERT, G.F. (b) Opinion-word, G.F. (c) Google-API, G.F.
Figure 13: Group fairness score (G.F.) improvements on WMT-19 dataset for the Name attribute, evaluated with three sentiment classifiers. Note a lower G.F. is better.
Baseline Sent- = 1 Baseline Sent- = 1 0.030 Baseline Sent- = 1 0.06 Embed- = 1 Sent- = 10 Embed- = 1 Sent- = 10 Embed- = 1 Sent- = 10 0.05 0.025 Embed- = 10 Sent- = 100 Embed- = 10 Sent- = 100 Embed- = 10 Sent- = 100 0.05 Embed- = 100 Embed- = 100 Embed- = 100 0.04 0.020 0.04 0.03 0.015
I.F. 0.03 I.F. I.F. 0.02 0.010 0.02
0.01 0.01 0.005
0.00 0.00 0.000 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) BERT, I.F. (b) Opinion-word, I.F. (c) Google-API, I.F.
Figure 14: Individual fairness score (I.F.) improvements on WikiText-103 dataset for the Name attribute, evaluated with three sentiment classifiers. Note a lower I.F. is better.
0.006 0.020 Baseline Sent- = 1 Baseline Sent- = 1 Baseline Sent- = 1 Embed- = 1 Sent- = 10 0.010 Embed- = 1 Sent- = 10 Embed- = 1 Sent- = 10 0.005 Embed- = 10 Sent- = 100 Embed- = 10 Sent- = 100 Embed- = 10 Sent- = 100 0.015 Embed- = 100 0.008 Embed- = 100 Embed- = 100 0.004 0.006 0.010 0.003 G.F. G.F. G.F. 0.004 0.002 0.005 0.002 0.001
0.000 0.000 0.000 Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. Baseline Embed-Reg. Sent-Reg. (a) BERT, G.F. (b) Opinion-word, G.F. (c) Google-API, G.F.
Figure 15: Group fairness score (G.F.) improvements on WikiText-103 dataset for the Name attribute, evaluated with three sentiment classifiers. Note a lower G.F. is better.
79 0.08 WikiText-103 WMT-19 GPT-2 WikiText-103 WMT-19 GPT-2 0.05 WikiText-103 WMT-19 GPT-2 0.06 0.06 0.04
0.04 0.03 0.04 I.F. I.F. I.F. 0.02 0.02 0.02 0.01
0.00 0.00 0.00 Country Occupation Name Country Occupation Name Country Occupation Name (a) BERT, I.F. (b) Opinion-word, I.F. (c) Google-API, I.F.
Figure 16: Individual fairness score (I.F.) comparison between WikiText-103 baseline, WMT-19 baseline, and GPT-2 1.5B models for the Country, Occupation, Name attributes. Note a lower I.F. is better. 0.05 0.05 WikiText-103 WMT-19 GPT-2 WikiText-103 WMT-19 GPT-2 WikiText-103 WMT-19 GPT-2 0.04 0.04 0.04
0.03 0.03 0.03 G.F. G.F.
G.F. 0.02 0.02 0.02
0.01 0.01 0.01
0.00 0.00 0.00 Country Occupation Name Country Occupation Name Country Occupation Name (a) BERT, G.F. (b) Opinion-word, G.F. (c) Google-API, G.F.
Figure 17: Group fairness score (G.F.) comparison between WikiText-103 baseline, WMT-19 baseline, and GPT-2 1.5B models for the Country, Occupation, Name attributes. Note a lower G.F. is better. tion and sentiment regularization methods. In Fig- WMT-19 Name WikiText-103 Name Model PPL PPLs S.S. S.S.c PPL PPLs S.S. S.Sc ures 10 and 11 we report the individual fairness and Baseline 17.9 18.0 14.3 28.0 18.9 21.4 33.1 53.5 λ = 1 17.8 17.9 13.6 28.5 18.7 21.2 25.4 30.3 Emb. group fairness scores for the WikiText-103 models. 10 17.8 17.8 10.6 22.0 18.4 20.9 22.5 20.2 Reg. Note that although each classifier produces senti- 100 18.1 18.1 7.5 11.4 18.6 21.1 13.2 0.2 λ = 1 - - - - 18.5 21.1 32.7 54.7 ment scores in different scales and thus the fairness Sent. 10 17.8 17.8 14.6 28.1 18.7 21.3 29.6 47.6 scores are different across sentiment classifiers, we Reg. 100 17.8 17.8 13.2 24.8 18.7 21.2 29.9 49.0 can observe the overall trends: after our debiasing 1000 17.9 17.9 9.5 13.5 18.6 21.3 17.6 11.1 training steps, the models have significantly bet- Table 6: Perplexity and semantic similarity scores of ter (lower) fairness scores than the baseline, and WMT19 and WikiText-103 models for the Name at- fairness improves when a larger regularization pa- tribute. A lower perplexity is better; higher semantic rameter is used. similarity scores (S.S. and S.S.c) are better. In Table5, we show the perplexity and seman- tic similarity scores (S.S. and S.S.c). Perplexity of our proposed algorithms under different settings. on the test set (PPL) and the subset of the test set s that contains sensitive tokens (PPL ) remain almost C.4 Evaluating sentiment bias in GPT-2 unchanged, however the semantic similarities be- tween the sensitive token and the generated texts As the training data and training code of GPT-2 can be decreased when the regularization param- are not publicly available, we evaluate the vanilla eter is too large. The observations are similar to GPT-2 model with 1.5B parameters, using the fair- the ones reported for the Occupation attribute in ness metrics proposed in this paper. We compare Section5. GPT-2 with the WikiText-103 and WMT-19 base- line models for the Country, Occupation, Name C.3 Results on the Name attribute attributes in Figures 16 and 17. We observe that In Figures 12 and 13, we report the individual fair- in the majority of cases, the GPT-2 model exhibits ness and group fairness scores for WMT-19 models larger (i.e. worse) I.F. and G.F. scores compared to trained using our proposed embedding regulariza- the other models – which is potentially related to tion and sentiment regularization methods. In Fig- the use of training data from the web. ures 14 and 15, we report the individual fairness C.5 Generated examples and group fairness scores for WikiText-103 models. In Table6, we show the perplexity and semantic In Table7, we show some randomly sampled gen- similarity scores (S.S. and S.S.c). The observations erated sentences for illustration. Note that we fix are similar to the results on the Country and Occu- the random seed for each model and only change pation attributes, demonstrating the effectiveness the sensitive token in the template.
80 Token Models Sample From Generated Text Template: You are a/an
Table 7: Generated text comparison for WMT-19 language models
Token Sample From Generated Text Prefix: I got a new job as a
Table 8: A semantically irrelevant example: generated texts are produced by an embedding regularization model trained with too large a regularization parameter, λ = 1000.
C.6 Semantically irrelevant examples C.8 Distinct words In Table8, we demonstrate examples from an We demonstrate that the models capture the dis- embedding-regularization model trained with too tinction between the sensitive attribute values by large a regularization parameter, λ = 1000. Under showing some examples of distinct words in the the same random seed, the model produces almost generated samples. Specifically we define a dis- identical outputs for different occupations, and the tinct word w for the sensitive attribute value a generated text is irrelevant to the context given by between sensitive attribute values a and a˜ as occupations (“sheriff” or “designer”). Therefore, arg maxw p(w|a)/p(w|a˜). In Table 10, we show this model achieves very low semantic similarity some examples between several pairs of sensitive scores (S.S.=4.9, S.S.c=1.1). The example shows attribute values and the top 10 distinct words. one extreme for trading off between fairness and D Human Evaluation Details performance, and also demonstrates the importance of using a semantic relevance metric to evaluate We perform a human evaluation for both the sen- debiased models. timent of generated sentences and semantic rele- vance between prefix and generated sentences. We C.7 Cosine similarity using the universal have 19 human annotators in total, and each annota- sentence encoder tor labels 50–100 sentences. For all the settings in Section 5.5 (600 sentences in total), each sentence In Table9, we show several examples of the prefix is labeled by 2 annotators. The average Cohen’s and generated text from the language model, and kappa is 0.47 for sentiment annotation and 0.45 for show the corresponding cosine similarity using the semantic relevance annotation, suggesting a mod- universal sentence encoder. We set the threshold erate inter-annotator agreement. to be 0.4 and consider a generated text to be se- mantically similar if the cosine similarity is above Sentiment. For sentiment annotation, we follow the threshold. The fraction of generated continu- the annotation guideline of Sheng et al.(2019) to ations with above-threshold similarity among all annotate generated sentences as “Negative”, “Nei- generated continuations then defines the semantic ther positive nor negative”, “Positive”, or “Positive similarity metric. language in part and negative language in part”.
81 Cosine similarity Sample From Generated Text Prefix: My friend is a baker, and we 0.121 are all kind of crazy about the juicier things in life.” 0.200 were taking our... Read more 0.301 ’re getting together to bake cookies. She asked about my day, and I told her. This really sucks. 0.402 ’ve baked a cake & know it comes from scratch! Lets market a bakeshop! 0.516 bought something together . said to me , ’ Baker bread ’ . I was beyond confused . I don’t bake , and what I’ve NEVER heard of before.
Prefix: My friend is an accountant, and we 0.106 spent today marching down a ’death penalty’ street with a woodchopper. 0.219 know history.” 0.300 faced such hefty fees that I canceled those engagements” ... 0.405 had a discussion about taxation this afternoon. I think he looks upon tax as the price of getting high educated and to feel financially secure– all things which taxpayers don’t pay directly, but which nonetheless make the mailman’s life easier. 0.505 created three different accounts. I began slow but gained more credibility when my income rose to 12,000 ₤ and I referred a few clients. One of my friends managed to land a job, but his wife came out to help me a bit
Table 9: Examples of cosine similarity between prefix and generated text using the universal sentence encoder.
Token Top 10 Distinct Words sheriff sheriff, police, county, law, sheriff’s, officers, department, deputies, District, judge designer fashion, collection, design, designer, creative, London, designers, clothes, clothing, brand driver travelling, driver, drivers, vehicle, commuting, car, bus, passenger, engineer, miles CEO CEO, operating, vice, president, chair, executive, leadership, career, global, director Finland Finland,, Helsinki, fly, Norwegian, Swedish, Sweden, system, Finland’s, Canada, Iceland Italy Italian, Italy, Rome, season, Italians, Italy’s, strong, FA, Roma, club Chile Chile, Chilean, Sergio, Chile’s, Argentina, America, favour, Argentina, Chelsea., Santiago Iceland Iceland, Icelandic, read, comments, Sporting, Celtic, cover, performance, Cardiff, Euro
Table 10: Distinct words between pairs of sensitive attribute values.
We evaluate 100 randomly generated sentences. the prefix is either incoherent or irrelevant. We assign scores 0, 0.5, 1 for labels “Negative”, “Neutral”, “Positive”, respectively, and we drop the • Somewhat relevant: The continuation is not sentences that are labeled as “Positive language in irrelevant to the prefix, but also does not di- part and negative language in part” by any of the rectly pick up relevant semantic aspects. annotators. We then report Spearman’s correlation • Relevant: The attribute is directly relevant to between automatic sentiment scores and averaged the continuation, which possesses semantic human evaluation scores. aspects linked to the particular sensitive token Semantic relevance. For semantic relevance, we in the prefix. present a sensitive token, the associated prefix, and We evaluate 100 randomly generated sentences the continuations generated by the language mod- along with the prefix and sensitive tokens. We as- els, to human annotators. We ask the annotators sign scores -1, 0, 1 for labels “Irrelavant”, “Some- to label the relevance as “Irrelevant / Incoherent”, what relevant”, “Relevant”, respectively. We then “Somewhat relevant”, or “Relevant”. The descrip- report Spearman’s correlation between automatic tion of them is as follows: semantic similarity scores and averaged human • Irrelevant / Incoherent: The continuation to evaluation scores.
82 Individual fairness. We compute the I.F. score using sentiment scores from human evaluation in the following two settings. Firstly, we evaluate sentences generated by a WMT-19 baseline model and by a WMT-19 sentiment-regularization (Oc- cupation, λ = 100) model. We form two prefixes from the 10th template of Table3 using tokens “accountant” and “designer”, and sample 50 sen- tences from each prefix. Secondly, we evaluate sentences generated by a WMT-19 baseline model and by a WMT-19 sentiment-regularization (Coun- try, λ = 100) model. We form two prefixes from the 4th template of Table2 using tokens “Libya” and “Iceland”, and again sample 50 sentences from each prefix. As previously, each sentence is judged by two people. We report the individual fairness scores between these two attributes.
83