November 2019 New York City Automated Decision Systems Task Force Report Understanding this report This report is a product of the New York City Automated Decision Systems (ADS) Task Force. In accordance with the mandates of Local Law 49 of 2018, the Task Force includes representatives from both the City and external organizations, and has prepared this report and recommendations contained herein for the Mayor of New York City and the Speaker of the New York City Council. Inclusion of any material in the report does not represent an endorsement of such material by the City of New York or any individual City agencies. The ADS Task Force’s work and recommendations are advisory in nature and, as set forth in Local Law 49, nothing “shall require [the City or its agencies’] compliance with the task force’s recommendations or disclosure of any information where such disclosure would violate local, state, or federal law, interfere with a law enforcement investigation or operations, compromise public health or safety, or that would result in the disclosure of proprietary information.” Table of Contents 2 | Letter from the Chairs 3 | Executive Summary 4 | Chapter 1: Acknowledgments 6 | Chapter 2: Members of the Automated Decision Systems Task Force 12 | Chapter 3: Background 15 | Chapter 4: Process 17 | Chapter 5: Recommendations 26 | Chapter 6: Challenges 30 | Chapter 7: Unresolved and Related Topics for Consideration 31 | Appendix I: Summary of Recommendations 32 | Appendix II: Text of Local Law 49 of 2018 NOVEMBER 2019 1 Letter from the Chairs It has been a great honor to serve as chairs of the Automated Decision Systems Task Force. We went into this process expecting to have smart, in-depth conversations about a whole new frontier of government work. We didn’t have models to look to or a guidebook to follow, but we expected the members to raise new questions, delve into tough topics, and approach this issue with creativity, expertise, and rigor. And we couldn’t be more proud of the Task Force for rising to—and exceeding—those expectations. It has been an incredible privilege to sit down with this group of experienced, thoughtful experts and talk about a topic that hasn’t been talked about in government enough yet, and to look at issues new and old through a lens that many New Yorkers have never had to consider: What do the values of equity, transparency, and accountability that are already embedded in our work mean in this context? How do we make sure that the technologies that can help improve the lives of those who rely on local government services are being used in an ethical manner and do not have unintended consequences that are unfair or harmful? There aren’t easy answers to these questions, but in this report you’ll find an immense amount of deliberative thinking that advances the conversation on how government can better protect the people it serves when using these automated tools. We are bringing forward a set of consensus-based recommendations to guide the Administration, which we are proud to say took a great deal of time, effort, energy, and deliberation to develop. An issue this complicated and new deserves nothing less. Jeff Thamkittikasem Chair, ADS Task Force Director, Mayor’s Office of Operations Brittny Saunders Co-Chair, ADS Task Force Deputy Commissioner for Strategic Initiatives, NYC Commission on Human Rights Kelly Jin Co-Chair, ADS Task Force NYC Chief Analytics Officer Director, Mayor’s Office of Data Analytics 2 NEW YORK CITY Automated Decision Systems Task Force Report Executive Summary Automated Decision Systems (ADS) are tools or systems that New York City agencies and offices may use to help them make decisions about services and resources provided to New Yorkers. These tools use data, algorithms, and, in some cases, machine learning or other artificial intelligence (AI) to aid in decision-making. In May of last year, Mayor Bill de Blasio convened the Automated Decision Systems Task Force, as required by Local Law 49 of 2018. Since then, the members and chairs of the Task Force have worked together to develop recommendations that will provide a framework for the use of and policy around ADS. Local Law 49 required that a mayoral Task Force provide recommendations on six different topics related to the use of ADS by City agencies. During the past 18 months, we have worked diligently toward the common goal of providing meaningful recommendations to the Mayor and City Council. In this report, we outline how we met the requirements of Local Law 49 through rigorous research, a months-long public engagement process, and collaborative deliberation. Additionally, our report explains how we plan to continue pushing forward the work of managing ADS in New York City beyond our tenure as a Task Force. This report is the culmination of the ADS Task Force’s work together, in which we provide our recommendations to the Mayor and City Council, and important background about our process, the challenges we faced, and more. The recommendations we present in this document center around three key themes: 1. Building capacity for an equitable, effective, and responsible approach to the City’s ADS. 2. Broadening public discussion on ADS. 3. Formalizing ADS management functions. While our recommendations reflect our agreement about the need for ADS management in New York City, we did not reach consensus on every potentially relevant issue, and there were certain topics beyond the scope of our mandate and timeframe as a Task Force. We outline these topics following our recommendations. It is our hope that this report will help drive meaningful work around the local government’s use of ADS and related tools and systems in the future, and will support the Administration as it delivers services equitably and effectively for millions of New Yorkers. NOVEMBER 2019 3 Chapter 1: Acknowledgments This report would not have been possible without the dedicated work of many people across New York City and beyond the five boroughs. First, the Task Force would like to thank the members of the public who provided their insights throughout this process. Without the written feedback via email or our online form, and the insights gleaned from our public forums and community sessions, the Task Force would not have had the knowledge necessary to provide meaningful recommendations that could positively impact the lives of all New Yorkers. The Task Force would also like to acknowledge the many experts who came forward and provided us with their insights. Understanding how ADS are used to support government decision-making is a complex undertaking, and the insights of policymakers, nonprofit organizations, advocacy groups, community leaders, technology experts, and others are necessary to help create clarity on the best ways to use and develop these tools. In particular, we wish to acknowledge the expert panelists who spoke at our public forums1: Aaron Pallas, Professor of Sociology and Education, Teachers College, Columbia University Andrew Nicklin, Futurist-At-Large, Johns Hopkins University Center for Government Excellence Chancey Fleet, Fellow, Data & Society and Assistive Technology Coordinator, Andrew Heiskell Braille and Talking Book Library, New York Public Library Ginger Zielinskie, President & CEO, Benefits Data Trust Janai Nelson, Associate Director-Counsel, NAACP Legal Defense and Educational Fund, Inc. Natalie Evans Harris, Co-Founder and Head of Strategic Initiatives, BrightHive Rumman Chowdhury, Senior Principal, Global Lead of Ethical Artificial Intelligence, Accenture Sarah Kaufman, Associate Director, New York University Rudin Center for Transportation The Task Force would also like to thank the many organizations that submitted comments to our public forums: AI Now Institute Brennan Center for Justice Legal Aid Society Luminosity NAACP Legal Defense and Educational Fund, Inc. National Domestic Worker Alliance Surveillance Technology Oversight Project 1 Titles for the individuals listed here reflect positions held in April and May 2019, when the public forums were held. Michael Appleton/Mayoral Photography Office,NYC 4 NEW YORK CITY Automated Decision Systems Task Force Report The Task Force would also like to thank the organizations that provided event space or otherwise supported our public engagement sessions: New York Law School, the Queens Public Library, Hostos Community College, the NYC Department of Youth & Community Development, the Atlantic Plaza Towers Tenant Association, and the Arden Heights Boulevard Jewish Center. And we would like to thank representatives of President Barack Obama’s Task Force on 21st Century Policing, who provided us with a model that helped inform the Task Force’s public engagement. We would especially like to thank Mayor Bill de Blasio for convening and appointing our Task Force, and for providing us with the first-in-the-nation opportunity to tackle this challenging and timely topic. We are excited to have New York City lead the way on innovative solutions in public policy. The Task Force would like to thank the New York City Council for introducing and passing Local Law 49 of 2018, which catalyzed this group’s important work. In particular, we would like to thank the Local Law’s original lead sponsor former Council Member James Vacca, Council Member and current Chair of the Committee on Technology Robert Holden, and Council Member and former Chair of the Committee on Technology Peter Koo. As technologies continue to develop and we find new and better ways to provide New Yorkers with the services they need, the continued partnership between the Mayor and the City Council will be important for driving the work forward. The Task Force is grateful to the many City staff at the Mayor’s Office of Operations (Operations), the Mayor’s Office of Data Analytics (MODA), the City Commission on Human Rights (CCHR), and the Mayor’s Office of Information Privacy (MOIP) for their collective advising, research, and administrative support that included coordinating meeting logistics, conducting interviews, and planning our public forums and community sessions.