Accountable Algorithms
Total Page:16
File Type:pdf, Size:1020Kb
Accountable Algorithms Joshua Alexander Kroll ADissertation Presented to the Faculty of Princeton University in Candidacy for the Degree of Doctor of Philosophy Recommended for Acceptance by the Department of Computer Science Adviser: Professor Edward W. Felten September 2015 c Copyright by Joshua Alexander Kroll, 2015. All rights reserved. Abstract Important decisions about people are increasingly made by algorithms: Votes are counted; voter rolls are purged; financial aid decisions are made; taxpayers are chosen for audits; air travelers are selected for search; credit eligibility decisions are made. Citizens, and society as a whole, have an interest in making these processes more transparent. Yet the full basis for these decisions is rarely available to a↵ected people: the algorithm or some inputs may be secret; or the implementation may be secret; or the process may not be precisely described. A person who suspects the process went wrong has little recourse. And an oversight authority who wants to ensure that decisions are made according to an acceptable policy has little assurance that pro↵ered decision rules match decisions for actual users. Traditionally, Computer Science addresses these problems by demanding a spec- ification of the desired behavior, which can be enforced or verified. But this model is poorly suited to real-world oversight tasks, where specifications are complicated or might not be known in advance; laws are often ambiguous precisely because it would be politically (and practically) infeasible to give a precise description of their meaning. People do their best to approximate what they believe the law will allow and disputes about what is acceptable happen after-the-fact via expensive investiga- tion and adjudication (e.g., in a court or legislature). Actual oversight, in which real decisions are reviewed for their correctness, fairness, or faithfulness to a rule happens only rarely, if at all. Further, Computer Science often sees rules as self-enforcing: the mere fact that an automated check fails is sufficient to demonstrate that some choice was invalid. However, like all rules, automated rules are just the intentions of a system designer and only bear relevance if people will actually follow them, either due to internalized incentives or the external threat of punishment. iii This dissertation relates the tools of technology to the problem of overseeing de- cision making processes. These methods use the tools of computer science to cryp- tographically ensure the technical properties that can be proven, while providing information necessary for a political, legal, or social oversight process to operate ef- fectively. First, we present an example of the current state-of-the-art in technical systems for accountability: a well-defined policy, specified in advance, is operational- ized with technical tools, and those same tools are used to convince outsiders or auditors. Our system enables the accountable execution of legal orders by a judge allowing an investigator compelled access to private records, so that the investigator’s access to sensitive information is limited to only that information which the judge has explicitly allowed (and this can be confirmed by a disinterested third party). Moving beyond these methods, we present a general framework for accountable algorithms, unifying a suite of cryptographic tools to design processes that enable meaningful after-the-fact oversight, consistent with the norm in law and policy. Accountable al- gorithms can attest to the valid operation of a decision policy even when all or part of that policy is kept secret. iv Acknowledgements Iamextremelygratefultomyadvisor,EdFelten,forhisunflaggingsupport,his generosity, and his willingness to go out of his way to be available and helpful to me and all of his students. I’ve learned much from Ed about how to spot interesting research problems, how to execute substantial research, and how to communicate the results of that research for maximum impact. Ihavebeenfortunatetoworkwithawiderangeofcollaboratorsonanumberof projects, and would like to acknowledge the contributions to my graduate career of (in alphabetical order) Andrew Appel, Dan Boneh, Mitchell Berger, Joseph Bonneau, Nicholas Butowski, Joe Calandrino, Jeremy Clark, Will Clarkson, Ian Davey, Anne Edmundson, Steve Englehardt, Ariel Feldman, Steven Goldfeder, J. Alex Halder- mann, Sean Hervey-Jumper, Timothy B. Lee, Peter Johnsen, Ben Jones, Seth Josef- fer, Harry Kalodner, Anna Kornfeld-Simpson, Elliott Krauss, Andrew Miller, Arvind Narayanan, Valeria Nikolaenko, Laura Roberts, Cole Schlesinger, Gordon Stewart, David Wu, Harlan Yu, William Zeller, and Joe Zimmerman. While I did not always get to work with each of you as much as I would have wanted, you certainly all had a strong impact on my thinking about research. I am especially grateful to my fellow students, from whom I learned the most during my time at Princeton. Academically, I have been strongly influenced by Andrew Appel, who taught me the value of thinking clearly about a problem before diving into a solution while remaining efficient, and by Arvind Narayanan, who taught me the value of looking where others are not and the value of eschewing short-term wins when one believes in a long-term plan. Arvind has also taught me the value of real-world adoption as a measure of research output. I would like to thank my entire thesis committee: Andrew Appel, Nick Feamster, Edward W. Felten, Matthew D. Green, and Arvind Narayanan for their candid, careful, and supportive feedback throughout this process. v I have been very fortunate to have the unflagging support of my wife, Ann Frey Kroll, who gave me the strength to keep believing in my work even when it was difficult or seemed not to be going anywhere. Ann has shown immense patience and has taught me to have courage in the face of even great difficulty. Iamalsothankfulforthesupportofmyfamily—myparents,RonandDarcie Kroll; my sister, Amy Kroll; and the many extended family and in-laws who have inspired me—all of them have enriched my experience in pursuing my research career in their own way. This dissertation is dedicated to the memory of my grandparents, Arvel Henning Mattson, who helped stir my childhood interest in engineering, and Verna Trom Mattson, who taught me much about the world and how to live in it e↵ectively. Without them, I would not have had the opportunity to pursue the education I have received, and I am truly grateful. I also acknowledge the memory of my friend, Vice Admiral Thomas J. Hughes, U.S.N., who is responsible for much more of my career than he ever realized. Additionally, I must take the opportunity to thank Princeton for giving me the opportunity and resources to give so much of my time to research in the years since 2009. I am especially grateful to Joy Montero from the office of the Dean of the Graduate School and her successor Lisa Schreyer for their support through my years in New Jersey. Finally, the work in this dissertation was supported in part by a National Science Foundation Graduate Research Fellowship under Grant No. DGE-1148900 and by a gift by Norm B. Tomlinson Jr., ’48 to the Center for Information Technology Policy at Princeton University. vi To my wife, Ann, whose courage and strength impel me to keep trying. vii Contents Abstract..................................... iii Acknowledgements ............................... v 1 Introduction 1 2 Background 21 2.1 Related Concepts from Computer Science . 21 2.1.1 Software Testing . 22 2.1.2 Type Systems and Software Verification . 24 2.1.3 Verified Computation . 26 2.1.4 Fairness in Computer Science . 27 2.1.5 SystemsforCryptographicAssurance . 40 2.1.6 AuditingSystems......................... 43 2.1.7 Measurement of Automated Decision Systems . 46 2.1.8 Accountability........................... 47 2.1.9 OurDefinitionofAccountability. 56 2.2 Related Concepts from Fields Outside Computer Science . 58 2.2.1 Philosophy of Law, Rule of Law, and Software as Law . 59 2.2.2 Due Process . 62 2.2.3 Transparency . 63 2.2.4 ReverseEngineering . 67 viii 2.2.5 Oversight and Enforcement . 69 2.2.6 Fairness . 71 2.2.7 Accountability........................... 76 3 Required Cryptographic Primitives 79 3.1 Common Primitives . 79 3.1.1 Signature Schemes . 80 3.1.2 Zero-Knowledge Proofs . 82 3.2 Primitives Required for Accountable Warrant Execution . 88 3.2.1 Identity-BasedEncryption . 88 3.2.2 Oblivious Transfer . 91 3.2.3 Secret Sharing . 94 3.2.4 Threshold Cryptography . 96 3.3 Primitives Required for General-Purpose Accountable Algorithms . 98 3.3.1 CryptographicCommitments . 98 3.3.2 Verified Computation . 101 3.3.3 Pseudorandomness . 108 3.3.4 Fair Randomness . 113 4 Accountable Warrant Execution 117 4.1 Problem and Setting . 117 4.1.1 Applications............................ 121 4.1.2 Our Results . 123 4.2 Security Model . 124 4.2.1 Setting . 124 4.2.2 Security Goals . 126 4.2.3 A simple, insufficient approach . 129 4.2.4 A complete approach . 131 ix 4.3 Protocol-Specific Primitives . 134 4.3.1 AuditableObliviousTransfer . 135 4.3.2 Sharing the IBE master secret . 136 4.4 ProtocolforAccountableCompelledAccess . 138 4.5 Prototype Implementation . 141 4.5.1 Deployment Concerns . 142 4.6 Evaluation . 145 4.6.1 Encryption Benchmarks . 146 4.6.2 Investigation Benchmarks . 147 4.7 Related Literature . 150 4.8 Discussion and Extensions . 152 4.8.1 Extensions . 153 4.8.2 OnselectingDecryptionAuthorities. 154 5 Constructing Accountable Algorithms 156 5.1 Setting and Model . 157 5.2 AccountableAlgorithms: AGeneralProtocol. 159 5.2.1 Protocol . 161 5.2.2 Analysis .............................. 165 5.2.3 Extensions . 169 5.3 Realizing Accountable Algorithms with Concrete Primitives . 171 6 Example Accountable Algorithms 173 6.1 Implementation . 173 6.1.1 Constraint System Generation . 175 6.2 Evaluation . 177 6.3 Examples . 178 6.3.1 Linear Classification: Scoring and Risk Assessment . 178 x 6.3.2 Fair Classification .