Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments Ben Green Yiling Chen Harvard University Harvard University [email protected] [email protected] ABSTRACT 1 INTRODUCTION Despite vigorous debates about the technical characteristics of risk Across the United States, courts are increasingly using risk as- assessments being deployed in the U.S. criminal justice system, sessments to estimate the likelihood that criminal defendants will remarkably little research has studied how these tools aect actual engage in unlawful behavior in the future.1 These tools are being decision-making processes. After all, risk assessments do not make deployed during several stages of criminal justice adjudication, in- denitive decisions—they inform judges, who are the nal arbiters. cluding at bail hearings (to predict the risk that the defendant, if It is therefore essential that considerations of risk assessments be released, will be rearrested before trial or not appear for trial) and informed by rigorous studies of how judges actually interpret and at sentencing (to predict the risk that the defendant will recidivate). use them. This paper takes a rst step toward such research on Because risk assessments rely on data and a standardized process, human interactions with risk assessments through a controlled many proponents believe that they can mitigate judicial biases and experimental study on Amazon Mechanical Turk. We found several make “objective” decisions about defendants [9, 12, 34]. Risk assess- behaviors that call into question the supposed ecacy and fair- ments have therefore gained widespread support as a tool to reduce ness of risk assessments: our study participants 1) underperformed incarceration rates and spur criminal justice reform [9, 27, 34]. the risk assessment even when presented with its predictions, 2) Yet many are concerned that risk assessments make biased de- could not eectively evaluate the accuracy of their own or the risk cisions due to the historical discrimination embedded in training assessment’s predictions, and 3) exhibited behaviors fraught with data. For example, the widely-used COMPAS risk assessment tool “disparate interactions,” whereby the use of risk assessments led wrongly labels black defendants as future criminals at twice the to higher risk predictions about black defendants and lower risk rate it does for white defendants [3]. Prompted by these concerns, predictions about white defendants. These results suggest the need machine learning researchers have developed a rapidly-growing for a new “algorithm-in-the-loop” framework that places machine body of technical work focused on topics such as characterizing the learning decision-making aids into the sociotechnical context of incompatibility of dierent fairness metrics [6, 44] and developing improving human decisions rather than the technical context of new algorithms to reduce bias [24, 33]. generating the best prediction in the abstract. If risk assessments Despite these eorts, current research into fair machine learning are to be used at all, they must be grounded in rigorous evaluations fails to capture an essential aspect of how risk assessments impact of their real-world impacts instead of in their theoretical potential. the criminal justice system: their inuence on judges. After all, risk assessments do not make denitive decisions about pretrial CCS CONCEPTS release and sentencing—they merely aid judges, who must decide whom to release before trial and how to sentence defendants after • Human-centered computing → Human computer interac- trial. In other words, algorithmic outputs act as decision-making tion (HCI); • Applied computing → Law. aids rather than nal arbiters. Thus, whether a risk assessment itself is accurate and fair is of only indirect concern—the primary KEYWORDS considerations are how it aects decision-making processes and fairness, risk assessment, behavioral experiment, Mechanical Turk whether it makes judges more accurate and fair. No matter how well we characterize the technical specications of risk assessments, we ACM Reference Format: will not fully understand their impacts unless we also study how Ben Green and Yiling Chen. 2019. Disparate Interactions: An Algorithm-in- judges interpret and use them. the-Loop Analysis of Fairness in Risk Assessments. In FAT* ’19: Conference This study sheds new light on how risk assessments inuence on Fairness, Accountability, and Transparency (FAT* ’19), January 29–31, 2019, human decisions in the context of criminal justice adjudication. Atlanta, GA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10. We ran experiments using Amazon Mechanical Turk to study how 1145/3287560.3287563 people make predictions about risk, both with and without the aid of a risk assessment. We focus on pretrial release, which in 2 Permission to make digital or hard copies of part or all of this work for personal or many respects resembles a typical prediction problem. By studying classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation 1 Although there have been several generations of criminal justice risk assessments on the rst page. Copyrights for third-party components of this work must be honored. over the past century [41], throughout this paper we use risk assessments to refer to For all other uses, contact the owner/author(s). machine learning algorithms that provide statistical predictions. FAT* ’19, January 29–31, 2019, Atlanta, GA, USA 2After someone is arrested, courts must decide whether to release that person until © 2019 Copyright held by the owner/author(s). their trial. This is typically done by setting an amount of “bail,” or money that the ACM ISBN 978-1-4503-6125-5/19/01. defendant must pay as collateral for release. The broad goal of this process is to protect https://doi.org/10.1145/3287560.3287563 individual liberty while also ensuring that the defendant appears in court for trial FAT* ’19, January 29–31, 2019, Atlanta, GA, USA Ben Green and Yiling Chen behavior in this controlled environment, we discerned important 2.1 People are bad at incorporating patterns in how risk assessments inuence human judgments of quantitative predictions risk. Although these experiments involved laypeople rather than The phenomenon of “automation bias” suggests that automated judges—limiting the extent to which our results can be assumed tools inuence human decisions in signicant, and often detrimen- to directly implicate real-world risk assessments—they highlight tal, ways. Two types of errors are particularly common: omission several types of interactions that should be studied further before errors, in which people do not recognize when automated systems risk assessments can be responsibly deployed in the courtroom. err, and commission errors, in which people follow automated sys- Our results suggest several ways in which the interactions be- tems without considering contradictory information [51]. Heavy tween people and risk assessments can generate errors and biases reliance on automated systems can alter people’s relationship to a in the administration of criminal justice, thus calling into question task by creating a “moral buer” between their decisions and the the supposed ecacy and fairness of risk assessments. First, even impacts of those decisions [11]. Thus, although “[a]utomated deci- when presented with the risk assessment’s predictions, participants sion support tools are designed to improve decision eectiveness made decisions that were less accurate than the advice provided. and reduce human error, [...] they can cause operators to relinquish Second, people could not eectively evaluate the accuracy of their a sense of responsibility and subsequently accountability because own or the risk assessment’s predictions: participants’ condence of a perception that the automation is in charge” [11]. in their performance was associated with their actual negatively Even when algorithms are more accurate, people do not appro- performance and their judgments of the risk assessment’s accuracy priately incorporate algorithmic recommendations to improve their and fairness had no association with the risk assessment’s actual decisions, instead preferring to rely on their own or other people’s accuracy and fairness. Finally, participant interactions with the risk judgment [47, 66]. One study found that people could not distin- assessment introduced two new forms of bias (which we collec- guish between reliable and unreliable predictions [30], and another tively term “disparate interactions”) into decision-making: when found that people often deviate incorrectly from algorithmic fore- evaluating black defendants, participants were 25.9% more strongly casts [18]. Compounding this bias is the phenomenon of “algorithm inuenced to increase their risk prediction at the suggestion of the aversion,” through which people are less tolerant of errors made by risk assessment and were 36.4% more likely to deviate from the algorithms than errors made by other people [17]. risk assessment toward higher levels of risk. Further research is necessary to ascertain whether judges exhibit similar behaviors. The chain from algorithm to person to decision has become 2.2 Information lters through existing biases vitally important as algorithms inform increasing numbers of high- Previous research suggests that information presumed to help peo- stakes decisions.