A Report Commissioned by the NACDL Task Force on Risk Assessment Tools
Total Page:16
File Type:pdf, Size:1020Kb
Risk Assessment Tools in the Criminal Legal System – Theory and Practice: A RESOURCE GUIDE A Report Commissioned by the NACDL Task Force on Risk Assessment Tools November 2020 Melissa Hamilton Reader in Law & Criminal Justice University of Surrey School of Law J.D., Ph.D., The University of Texas at Austin i | Page Copyright © 2020 National Association of Criminal Defense Lawyers This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. It may be reproduced, provided that no charge is imposed, and the National Association of Criminal Defense Lawyers is acknowledged as the original publisher and the copyright holder. For any other form of reproduction, please contact NACDL for permission. For more information contact: NATIONAL ASSOCIATION OF CRIMINAL DEFENSE LAWYERS© 1660 L Street NW, 12th Floor, Washington, DC 20036, Phone 202-872-8600 www.NACDL.org This publication is available online at www.NACDL.org/RiskAssessmentReport ii | Page Risk Assessment Tools in the Criminal Legal System – Theory and Practice: A RESOURCE GUIDE A Report Commissioned by the NACDL Task Force on Risk Assessment Tools Christopher W. Adams Lisa M. Wayne President, NACDL President, NFCJ Charleston, South Carolina Denver, Colorado Norman L. Reimer Executive Director, NACDL & NFCJ Washington, DC Task Force on Risk Assessment Tools Vicki H. Young Marvin Schechter Co-Chair Co-Chair Palo Alto, California New York, New York Franklin Draper Matthew Knecht Colette Tvedt Washington, DC New York, New York Denver, Colorado Nina J. Ginsberg John Philipsborn CeCelia Valentine-Andrews Alexandria, Virginia San Francisco, California Vancouver, Washington Stephen Ross Johnson Gail Shifman AUTHOR Knoxville, Tennessee San Francisco, California Dr. Melissa Hamilton iii | Page Table of Contents I. About NACDL and the NACDL FCJ ............................................................................................................ vi II. Acknowledgements ................................................................................................................................ vii III. Foreword .............................................................................................................................................. viii IV. Executive Summary .................................................................................................................................. x V. Introduction ............................................................................................................................................. 1 VI. Overview .................................................................................................................................................. 2 VII. A Primer on Risk Assessment Practices .................................................................................................... 5 A. The Evolution of Risk Tools ......................................................................................................................... 6 B. Proclaimed Benefits of Risk Assessment .................................................................................................. 11 C. Decision Points ......................................................................................................................................... 15 VIII. The Science Underlying Algorithmic Risk Tools ...................................................................................... 17 A. Tool Development .................................................................................................................................... 17 1. Risk Tool Developers ............................................................................................................................ 17 2. Stages of Development ........................................................................................................................ 18 3. Types of Assessments ........................................................................................................................... 21 4. Training ................................................................................................................................................ 22 5. Collecting Information to Score ........................................................................................................... 23 B. Accuracy and Validation ........................................................................................................................... 24 1. Validity Measures ................................................................................................................................ 24 2. Accuracy (and Error) Rates Are Changeable ........................................................................................ 32 C. Cross-Validations ...................................................................................................................................... 34 D. Reliability .................................................................................................................................................. 38 1. Inter-rater Reliability ............................................................................................................................ 38 2. Overrides .............................................................................................................................................. 39 E. Communicating Risk Tool Results ............................................................................................................. 41 1. Group-to-Individual Challenge ............................................................................................................. 42 2. Forms of Risk Communication ............................................................................................................. 44 3. Risk Rankings Are Relative ................................................................................................................... 48 IX. Issues of Fairness and Bias ..................................................................................................................... 53 A. Systematic Bias Overall (i.e., Poor Accuracy) ............................................................................................ 53 B. Measures of Fairness or Unfairness ......................................................................................................... 54 1. Individual Fairness ............................................................................................................................... 54 2. Algorithmic Group Fairness .................................................................................................................. 54 3. Test Bias ............................................................................................................................................... 62 C. How Biases May Enter Algorithms ........................................................................................................... 63 1. Label Bias ............................................................................................................................................. 63 iv | Page 2. Feature Selection ................................................................................................................................. 65 3. Specification Error ................................................................................................................................ 65 4. Sample Bias .......................................................................................................................................... 67 5. Biases Preexisting in Training Data ...................................................................................................... 70 6. Conflicts of Interest .............................................................................................................................. 71 7. Feedback Loop ..................................................................................................................................... 73 D. Efforts to Remediate Bias ......................................................................................................................... 73 X. Legal and Ethical Issues .......................................................................................................................... 75 A. Risk Factors ............................................................................................................................................... 76 1. Sociodemographic Variables ................................................................................................................ 76 2. Immutable Characteristics ................................................................................................................... 78 3. Dehumanization ................................................................................................................................... 79 B. Due Process and Equitable Considerations .............................................................................................. 80 1. The Adversarial Process ....................................................................................................................... 80 2. Fit for Purpose .....................................................................................................................................