TrustNLP TrustNLP: First Workshop on Trustworthy Natural Language Processing Proceedings of the Workshop June 10, 2021 ©2021 The Association for Computational Linguistics Order copies of this and other ACL proceedings from: Association for Computational Linguistics (ACL) 209 N. Eighth Street Stroudsburg, PA 18360 USA Tel: +1-570-476-8006 Fax: +1-570-476-0860 [email protected] ISBN 978-1-954085-33-6 ii Introduction Recent progress in Artificial Intelligence (AI) and Natural Language Processing (NLP) has greatly increased their presence in everyday consumer products in the last decade. Common examples include virtual assistants, recommendation systems, and personal healthcare management systems, among others. Advancements in these fields have historically been driven by the goal of improving model performance as measured by accuracy, but recently the NLP research community has started incorporating additional constraints to make sure models are fair and privacy-preserving. However, these constraints are not often considered together, which is important since there are critical questions at the intersection of these constraints such as the tension between simultaneously meeting privacy objectives and fairness objectives, which requires knowledge about the demographics a user belongs to. In this workshop, we aim to bring together these distinct yet closely related topics. We invited papers which focus on developing models that are “explainable, fair, privacy-preserving, causal, and robust” (Trustworthy ML Initiative). Topics of interest include: • Differential Privacy • Fairness and Bias: Evaluation and Treatments • Model Explainability and Interpretability • Accountability • Ethics • Industry applications of Trustworthy NLP • Causal Inference • Secure and trustworthy data generation In total, we accepted 11 papers, including 2 non-archival papers. We hope all the attendants enjoy this workshop. iii Organizing Committee • Yada Pruksachatkun - Alexa AI • Anil Ramakrishna - Alexa AI • Kai-Wei Chang - UCLA, Amazon Visiting Academic • Satyapriya Krishna - Alexa AI • Jwala Dhamala - Alexa AI • Tanaya Guha - University of Warwick • Xiang Ren - USC Speakers • Mandy Korpusik - Assistant professor, Loyola Marymount University • Richard Zemel - Industrial Research Chair in Machine Learning, University of Toronto • Robert Monarch - Author, Human-in-the-Loop Machine Learning Program committee • Rahul Gupta - Alexa AI • Willie Boag - Massachusetts Institute of Technology • Naveen Kumar - Disney Research • Nikita Nangia - New York University • He He - New York University • Jieyu Zhao - University of California Los Angeles • Nanyun Peng - University of California Los Angeles • Spandana Gella - Alexa AI • Moin Nadeem - Massachusetts Institute of Technology • Maarten Sap - University of Washington • Tianlu Wang - University of Virginia • William Wang - University of Santa Barbara • Joe Near - University of Vermont • David Darais - Galois • Pratik Gajane - Department of Computer Science, Montanuniversitat Leoben, Austria • Paul Pu Liang - Carnegie Mellon University v • Hila Gonen - Bar-Ilan University • Patricia Thaine - University of Toronto • Jamie Hayes - Google DeepMind, University College London, UK • Emily Sheng - University of California Los Angeles • Isar Nejadgholi - National Research Council Canada • Anthony Rios - University of Texas at San Antonio vi Table of Contents Interpretability Rules: Jointly Bootstrapping a Neural Relation Extractorwith an Explanation Decoder Zheng Tang and Mihai Surdeanu . .1 Measuring Biases of Word Embeddings: What Similarity Measures and Descriptive Statistics to Use? Hossein Azarpanah and Mohsen Farhadloo . .8 Private Release of Text Embedding Vectors Oluwaseyi Feyisetan and Shiva Kasiviswanathan. .15 Accountable Error Characterization Amita Misra, Zhe Liu and Jalal Mahmud . 28 xER: An Explainable Model for Entity Resolution using an Efficient Solution for the Clique Partitioning Problem Samhita Vadrevu, Rakesh Nagi, JinJun Xiong and Wen-mei Hwu . 34 Gender Bias in Natural Language Processing Across Human Languages Abigail Matthews, Isabella Grasso, Christopher Mahoney, Yan Chen, Esma Wali, Thomas Middle- ton, Mariama Njie and Jeanna Matthews . 45 Interpreting Text Classifiers by Learning Context-sensitive Influence of Words Sawan Kumar, Kalpit Dixit and Kashif Shah. .55 Towards Benchmarking the Utility of Explanations for Model Debugging Maximilian Idahl, Lijun Lyu, Ujwal Gadiraju and Avishek Anand . 68 vii Conference Program June 10, 2021 9:00–9:10 Opening Organizers 9:10–10:00 Keynote 1 Richard Zemel 10:00–11:00 Paper Presentations Interpretability Rules: Jointly Bootstrapping a Neural Relation Extractorwith an Explanation Decoder Zheng Tang and Mihai Surdeanu Measuring Biases of Word Embeddings: What Similarity Measures and Descriptive Statistics to Use? Hossein Azarpanah and Mohsen Farhadloo Private Release of Text Embedding Vectors Oluwaseyi Feyisetan and Shiva Kasiviswanathan Accountable Error Characterization Amita Misra, Zhe Liu and Jalal Mahmud 11:00–11:15 Break ix June 10, 2021 (continued) 11:15–12:15 Paper Presentations xER: An Explainable Model for Entity Resolution using an Efficient Solution for the Clique Partitioning Problem Samhita Vadrevu, Rakesh Nagi, JinJun Xiong and Wen-mei Hwu Gender Bias in Natural Language Processing Across Human Languages Abigail Matthews, Isabella Grasso, Christopher Mahoney, Yan Chen, Esma Wali, Thomas Middleton, Mariama Njie and Jeanna Matthews Interpreting Text Classifiers by Learning Context-sensitive Influence of Words Sawan Kumar, Kalpit Dixit and Kashif Shah Towards Benchmarking the Utility of Explanations for Model Debugging Maximilian Idahl, Lijun Lyu, Ujwal Gadiraju and Avishek Anand 12:15–1:30 Lunch Break 13:00–14:00 Mentorship Meeting 14:00–14:50 Keynote 2 Mandy Korpusik 14:50–15:00 Break 15:00–16:00 Poster Session 16:15–17:05 Keynote 3 Robert Munro 17:05–17:15 Closing Address x Interpretability Rules: Jointly Bootstrapping a Neural Relation Extractor with an Explanation Decoder Zheng Tang, Mihai Surdeanu Department of Computer Science University of Arizona, Tucson, Arizona, USA {zhengtang, msurdeanu}@email.arizona.edu Abstract traction (RE) system (Angeli et al., 2015) and boot- strap a neural RE approach that is trained jointly We introduce a method that transforms a rule- with a decoder that learns to generate the rules that based relation extraction (RE) classifier into a neural one such that both interpretability best explain each particular extraction. The contri- and performance are achieved. Our approach butions of our idea are the following: jointly trains a RE classifier with a decoder (1) We introduce a strategy that jointly learns a RE that generates explanations for these extrac- classifier between pairs of entity mentions with a tions, using as sole supervision a set of rules decoder that generates explanations for these ex- that match these relations. Our evaluation on the TACRED dataset shows that our neural RE tractions in the form of Tokensregex (Chang and classifier outperforms the rule-based one we Manning, 2014) or Semregex (Chambers et al., started from by 9 F1 points; our decoder gen- 2007) patterns. The only supervision for our erates explanations with a high BLEU score of method is a set of input rules (or patterns) in these over 90%; and, the joint learning improves the two frameworks (Angeli et al., 2015), which we performance of both the classifier and decoder. use to generate positive examples for both the clas- sifier and the decoder. We generate negative exam- 1 Introduction ples automatically from the sentences that contain Information extraction (IE) is one of the key chal- positives examples. lenges in the natural language processing (NLP) (2) We evaluate our approach on the TACRED field. With the explosion of unstructured informa- dataset (Zhang et al., 2017) and demonstrate that: tion on the Internet, the demand for high-quality (a) our neural RE classifier outperforms consider- tools that convert free text to structured information ably the rule-based one we started from; (b) our continues to grow (Chang et al., 2010; Lee et al., decoder generates explanations with high accuracy, 2013; Valenzuela-Escarcega et al., 2018). i.e., a BLEU overlap score between the generated The past decades have seen a steady transition rules and the gold, hand-written rules of over 90%; from rule-based IE systems (Appelt et al., 1993) to and, (c) joint learning improves the performance of methods that rely on machine learning (ML) (see both the classifier and decoder. Related Work). While this transition has generally yielded considerable performance improvements, it (3) We demonstrate that our approach generalizes was not without a cost. For example, in contrast to to the situation where a vast amount of labeled modern deep learning methods, the predictions of training data is combined with a few rules. We com- rule-based approaches are easily explainable, as a bined the TACRED training data with the above small number of rules tends to apply to each extrac- rules and showed that when our method is trained tion. Further, in many situations, rule-based meth- on this combined data, the classifier obtains near ods can be developed by domain experts with mini- state-of-art performance at 67.0% F1, while the de- mal training data. For these reasons, rule-based IE coder generates accurate explanations with a BLEU methods remain widely used in industry (Chiticariu score of 92.4%. et al., 2013). 2 Related Work In this work we demonstrate that this transition