DOCSLIB.ORG

Nber Working Paper Series Human Decisions And

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Nber Working Paper Series Human Decisions And NBER WORKING PAPER SERIES HUMAN DECISIONS AND MACHINE PREDICTIONS Jon Kleinberg Himabindu Lakkaraju Jure Leskovec Jens Ludwig Sendhil Mullainathan Working Paper 23180 http://www.nber.org/papers/w23180 NATIONAL BUREAU OF ECONOMIC RESEARCH 1050 Massachusetts Avenue Cambridge, MA 02138 February 2017 We are immensely grateful to Mike Riley for meticulously and tirelessly spearheading the data analytics, with effort well above and beyond the call of duty. Thanks to David Abrams, Matt Alsdorf, Molly Cohen, Alexander Crohn, Gretchen Ruth Cusick, Tim Dierks, John Donohue, Mark DuPont, Meg Egan, Elizabeth Glazer, Judge Joan Gottschall, Nathan Hess, Karen Kane, Leslie Kellam, Angela LaScala-Gruenewald, Charles Loeffler, Anne Milgram, Lauren Raphael, Chris Rohlfs, Dan Rosenbaum, Terry Salo, Andrei Shleifer, Aaron Sojourner, James Sowerby, Cass Sunstein, Michele Sviridoff, Emily Turner, and Judge John Wasilewski for valuable assistance and comments, to Binta Diop, Nathan Hess, and Robert Webberfor help with the data, to David Welgus and Rebecca Wei for outstanding work on the data analysis, to seminar participants at Berkeley, Carnegie Mellon, Harvard, Michigan, the National Bureau of Economic Research, New York University, Northwestern, Stanford and the University of Chicago for helpful comments, to the Simons Foundation for its support of Jon Kleinberg's research, to the Stanford Data Science Initiative for its support of Jure Leskovec’s research, to the Robert Bosch Stanford Graduate Fellowship for its support of Himabindu Lakkaraju and to Tom Dunn, Ira Handler, and the MacArthur, McCormick and Pritzker foundations for their support of the University of Chicago Crime Lab and Urban Labs. The main data we analyze are provided by the New York State Division of Criminal Justice Services (DCJS), and the Office of Court Administration. The opinions, findings, and conclusions expressed in this publication are those of the authors and not those of DCJS. Neither New York State nor DCJS assumes liability for its contents or use thereof. The paper also includes analysis of data obtained from the Inter-University Consortium for Political and Social Research at the University of Michigan. Any errors and all opinions are our own. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research. NBER working papers are circulated for discussion and comment purposes. They have not been peer-reviewed or been subject to the review by the NBER Board of Directors that accompanies official NBER publications. © 2017 by Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. All rights reserved. Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission provided that full credit, including © notice, is given to the source. Human Decisions and Machine Predictions Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan NBER Working Paper No. 23180 February 2017 JEL No. C01,C54,C55,D8,H0,K0 ABSTRACT We examine how machine learning can be used to improve and understand human decision- making. In particular, we focus on a decision that has important policy consequences. Millions of times each year, judges must decide where defendants will await trial—at home or in jail. By law, this decision hinges on the judge’s prediction of what the defendant would do if released. This is a promising machine learning application because it is a concrete prediction task for which there is a large volume of data available. Yet comparing the algorithm to the judge proves complicated. First, the data are themselves generated by prior judge decisions. We only observe crime outcomes for released defendants, not for those judges detained. This makes it hard to evaluate counterfactual decision rules based on algorithmic predictions. Second, judges may have a broader set of preferences than the single variable that the algorithm focuses on; for instance, judges may care about racial inequities or about specific crimes (such as violent crimes) rather than just overall crime risk. We deal with these problems using different econometric strategies, such as quasi-random assignment of cases to judges. Even accounting for these concerns, our results suggest potentially large welfare gains: a policy simulation shows crime can be reduced by up to 24.8% with no change in jailing rates, or jail populations can be reduced by 42.0%with no increase in crime rates. Moreover, we see reductions in all categories of crime, including violent ones. Importantly, such gains can be had while also significantly reducing the percentage of African-Americans and Hispanics in jail. We find similar results in a national dataset as well. In addition, by focusing the algorithm on predicting judges’ decisions, rather than defendant behavior, we gain some insight into decision-making: a key problem appears to be that judges to respond to ‘noise’ as if it were signal. These results suggest that while machine learning can be valuable, realizing this value requires integrating these tools into an economic framework: being clear about the link between predictions and decisions; specifying the scope of payoff functions; and constructing unbiased decision counterfactuals. Jon Kleinberg Jens Ludwig Department of Computer Science University of Chicago Department of Information Science 1155 East 60th Street Cornell University Chicago, IL 60637 Ithaca, NY 14853 and NBER [email protected] [email protected] Himabindu Lakkaraju Sendhil Mullainathan Stanford University Department of Economics Littauer M-18 Gates Computer Science Building Harvard University 353 Serra Mall Cambridge, MA 02138 Stanford, CA 94305 and NBER [email protected] [email protected] Jure Leskovec Stanford University Gates Computer Science Building 353 Serra Mall Stanford, CA 94305 [email protected] R AMi`Q/m+iBQM J+?BM2 H2`MBM; Bb bm`T`BbBM;Hv 2z2+iBp2 i rB/2 p`B2iv Q7 ibFb i`/BiBQMHHv bbQ+Bi2/ rBi? ?mKM BMi2HHB;2M+2- 7`QK `2+Q;MBxBM; 7+2b BM T?QiQb iQ b+Q`BM; r`Bii2M 2bbvbX A7 i?2b2 iQQHb +M #2 TTHB2/ bm++2bb7mHHv iQ #bB+ ibFb BMpQHpBM; ?mKM pBbBQM M/ HM;m;2- r?v MQi iQ KQ`2 +QKTH2t ?mKM /2+BbBQMb b r2HH\ q2 2tKBM2 i?2 T`QKBb2 M/ i?2 TBi7HHb Q7 bm+? iQQHb rBi?BM i?2 +QMi2ti Q7 M BKTQ`iMi Dm/B+BH /2+BbBQM, 1p2`v v2` BM i?2 lMBi2/ aii2b- i?2 TQHB+2 ``2bi Qp2` Ry KBHHBQM T2QTH2 U6"A- kyReVX aQQM 7i2` ``2bi- Dm/;2b /2+B/2 r?2i?2` /272M/Mib Kmbi rBi BM DBH r?BH2 i?2B` H2;H 7i2 Bb #2BM; /2+B/2/X aBM+2 +b2b +M iF2 b2p2`H KQMi?b QM p2`;2 iQ `2bQHp2- i?Bb Bb +QMb2[m2MiBH /2+BbBQM 7Q` #Qi? /272M/Mib M/ bQ+B2iv b r?QH2XR Cm/;2b `2 #v Hr bmTTQb2/ iQ #b2 i?2B` #BH /2+BbBQM bQH2Hv QM T`2/B+iBQM, r?i rBHH i?2 /272M/Mi /Q B7 `2H2b2/\ qBHH ?2 ~22\ P` +QKKBi MQi?2` +`BK2\ Pi?2` 7+iQ`b- bm+? b r?2i?2` i?2 /272M/Mi Bb ;mBHiv- /Q MQi 2Mi2` i?Bb /2+BbBQMX h?2 `2HBM+2 Q7 i?2 #BH /2+BbBQM QM T`2/B+iBQM KF2b i?Bb M B/2H TTHB+iBQM i?i THvb iQ i?2 bi`2M;i?b Q7 K+?BM2 H2`MBM;Xk 7+2 `2+Q;MBiBQM H;Q`Bi?K- 7Q` 2tKTH2- Bb i`BM2/ QM /ib2i Q7 T?QiQb rBi? M/ rBi?Qmi 7+2bc Bi T`Q/m+2b 7mM+iBQM i?i ǵT`2/B+ibǶ i?2 T`2b2M+2 Q7 7+2 BM ;Bp2M T?QiQX MHQ;QmbHv- r2 TTHv K+?BM2 H2`MBM; H;Q`Bi?Kě bT2+B}+HHv- ;`/B2Mi@#QQbi2/ /2+BbBQM i`22běi`BM2/ QM /272M/Mi +?`+i2`BbiB+b iQ T`2/B+i +`BK2 `BbFX Pm` H;Q`Bi?K KF2b mb2 Q7 /ib2i Q7 758, 027 /272M/Mib r?Q r2`2 ``2bi2/ BM L2r uQ`F *Biv #2ir22M kyy3 M/ kyRjXj q2 ?p2 /2iBH2/ BM7Q`KiBQM #Qmi /272M/Mib- r?2i?2` i?2v r2`2 `2H2b2/ T`2@i`BH- M/ B7 bQ- r?2i?2` i?2v r2Mi QM iQ +QKKBi M2r +`BK2X9 AM i?Bb +b2- i?2 BMTmib Ui?2 MHQ;m2 Q7 i?2 T?QiQ BM 7+2 `2+Q;MBiBQM TTHB+iBQMV `2 i?2 /272M/Mi ii`B#mi2b pBH#H2 iQ Dm/;2b r?2M i?2v /2+B/2, T`BQ` `T b?22i- +m``2Mi Qz2Mb2- M/ bQ QMX h?2 H;Q`Bi?K QmiTmib T`2/B+iBQM Q7 +`BK2 `BbFX8 h?2 ;QH Q7 Qm` TT2` Bb MQi iQ B/2MiB7v M2r rvb Q7 QTiBKBxBM; i?2 K+?BM2 H2`MBM; R6Q` 2tKTH2- i?2 p2`;2 H2M;i? Q7 biv Bb #Qmi irQ KQMi?b BM L2r uQ`F *Bivc b22 `2TQ`iX MMmHBx2/ +Qbib Q7 DBHBM; T2`bQM `2 QM i?2 Q`/2` Q7 0jy-yyyX CBH bT2HHb HbQ BKTQb2 +Qbib QM /272M/Mib BM i2`Kb Q7 HQbi 7`22/QK U#`Kb M/ _Q?H7b- kyRRV- BKT+ib QM 7KBHB2b- ?B;?2` +?M+2 Q7 }M/BM; Q7 ;mBHi- M/ /2+HBM2b BM 7mim`2 2KTHQvK2Mi U.Q##B2 2i HX- kyReVX kh?Bb TTHB+iBQM +QMM2+ib iQ i?2 TBQM22`BM; rQ`F #v _B+?`/ "2`F QM mbBM; K+?BM2 H2`MBM; iQ T`2/B+i +`BK2 `BbFc b22- 7Q` 2tKTH2- i?2 bmKK`v BM "2`F UkyRkVX /Bz2`2Mi bi`M/ Q7 rQ`F BM 2+QMQKB+b 7Q+mb2b QM mbBM; K+?BM2 H2`MBM; iQQHb 7Q` +mbH BM72`2M+2bc b22- 7Q` 2tKTH2- "2HHQMB- *?2`MQx?mFQp M/ >Mb2M UkyR9V M/ i?2v M/ AK#2Mb UkyReVX jb Bb biM/`/- r2 `M/QKHv T`iBiBQM Qm` /i BMiQ i`BMBM; M/ ?QH/@Qmi b2i iQ T`Qi2+i ;BMbi Qp2`@}iiBM;X HH `2bmHib 7Q` i?2 T2`7Q`KM+2 Q7 i?2 H;Q`Bi?K `2 K2bm`2/ BM i?2 ?QH/@Qmi b2iX 9AM KQbi Dm`Bb/B+iBQMb Dm/;2b `2 bF2/ iQ 7Q+mb QM #Qi? i?2 `BbF Q7 7BHBM; iQ TT2` i `2[mB`2/ 7mim`2 +Qm`i /i2 Ur?B+? +M H2/ iQ M2r +`BKBMH +?`;2V M/ `2@``2biX >Qr2p2` mM/2` L2r uQ`F bii2 Hr- Dm/;2b BM #BH ?2`BM;b `2 QMHv bmTTQb2/ iQ +QMbB/2` ~B;?i `BbF U7BHm`2 iQ TT2`- Q` 6hVX lMH2bb Qi?2`rBb2 BM/B+i2/- BM r?i 7QHHQrb- 7Q` +QMp2MB2M+2- r2 mb2 i?2 i2`K Ǵ+`BK2Ǵ iQ `272` iQ 7BHm`2 iQ TT2` BM i?2 L2r uQ`F /i- M/ iQ 2Bi?2` 7BHm`2 iQ TT2` Q` `2@``2bi BM i?2 MiBQMH /iX q2 b?Qr BM a2+iBQM e i?i Qm` `2bmHib 7Q` i?2 `2HiBp2 T2`7Q`KM+2 Q7 i?2 H;Q`Bi?K p2`bmb Dm/;2 /2+BbBQMb +``v i?`Qm;? B``2bT2+iBp2 Q7 i?2 bT2+B}+ /2}MBiBQM Q7 Ǵ+`BK2Ǵ i?i r2 mb2X 8AM r?i 7QHHQrb- #v dz T`2/B+iBQMǴ r2 rBHH Hrvb K2M i?2 QmiTmi Q7 T`Q##BHBiv `i?2` i?M #BM`v Qmi+QK2 b bQK2iBK2b b22M BM +HbbB}+iBQM KQ/2HbX j H;Q`Bi?KX AMbi2/ Qm` ;QH Bb iQ
Load more

Recommended publications
  • Himabindu Lakkaraju
    Himabindu Lakkaraju Contact Information 428 Morgan Hall Harvard Business School Soldiers Field Road Boston, MA 02163 E-mail: [email protected] Webpage: http://web.stanford.edu/∼himalv Research Interests Transparency, Fairness, and Safety in Artificial Intelligence (AI); Applications of AI to Criminal Justice, Healthcare, Public Policy, and Education; AI for Decision-Making. Academic & Harvard University 11/2018 - Present Professional Postdoctoral Fellow with appointments in Business School and Experience Department of Computer Science Microsoft Research, Redmond 5/2017 - 6/2017 Visiting Researcher Microsoft Research, Redmond 6/2016 - 9/2016 Research Intern University of Chicago 6/2014 - 8/2014 Data Science for Social Good Fellow IBM Research - India, Bangalore 7/2010 - 7/2012 Technical Staff Member SAP Research, Bangalore 7/2009 - 3/2010 Visiting Researcher Adobe Systems Pvt. Ltd., Bangalore 7/2007 - 7/2008 Software Engineer Education Stanford University 9/2012 - 9/2018 Ph.D. in Computer Science Thesis: Enabling Machine Learning for High-Stakes Decision-Making Advisor: Prof. Jure Leskovec Thesis Committee: Prof. Emma Brunskill, Dr. Eric Horvitz, Prof. Jon Kleinberg, Prof. Percy Liang, Prof. Cynthia Rudin Stanford University 9/2012 - 9/2015 Master of Science (MS) in Computer Science Advisor: Prof. Jure Leskovec Indian Institute of Science (IISc) 8/2008 - 7/2010 Master of Engineering (MEng) in Computer Science & Automation Thesis: Exploring Topic Models for Understanding Sentiments Expressed in Customer Reviews Advisor: Prof. Chiranjib
    [Show full text]
  • Overview on Explainable AI Methods Global‐Local‐Ante‐Hoc‐Post‐Hoc
    00‐FRONTMATTER Seminar Explainable AI Module 4 Overview on Explainable AI Methods Birds‐Eye View Global‐Local‐Ante‐hoc‐Post‐hoc Andreas Holzinger Human‐Centered AI Lab (Holzinger Group) Institute for Medical Informatics/Statistics, Medical University Graz, Austria and Explainable AI‐Lab, Alberta Machine Intelligence Institute, Edmonton, Canada a.holzinger@human‐centered.ai 1 Last update: 11‐10‐2019 Agenda . 00 Reflection – follow‐up from last lecture . 01 Basics, Definitions, … . 02 Please note: xAI is not new! . 03 Examples for Ante‐hoc models (explainable models, interpretable machine learning) . 04 Examples for Post‐hoc models (making the “black‐box” model interpretable . 05 Explanation Interfaces: Future human‐AI interaction . 06 A few words on metrics of xAI (measuring causability) a.holzinger@human‐centered.ai 2 Last update: 11‐10‐2019 01 Basics, Definitions, … a.holzinger@human‐centered.ai 3 Last update: 11‐10‐2019 Explainability/ Interpretability – contrasted to performance Zachary C. Lipton 2016. The mythos of model interpretability. arXiv:1606.03490. Inconsistent Definitions: What is the difference between explainable, interpretable, verifiable, intelligible, transparent, understandable … ? Zachary C. Lipton 2018. The mythos of model interpretability. ACM Queue, 16, (3), 31‐57, doi:10.1145/3236386.3241340 a.holzinger@human‐centered.ai 4 Last update: 11‐10‐2019 The expectations of xAI are extremely high . Trust – interpretability as prerequisite for trust (as propagated by Ribeiro et al (2016)); how is trust defined? Confidence? . Causality ‐ inferring causal relationships from pure observational data has been extensively studied (Pearl, 2009), however it relies strongly on prior knowledge . Transferability – humans have a much higher capacity to generalize, and can transfer learned skills to completely new situations; compare this with e.g.
    [Show full text]
  • 2018 Annual Report Alfred P
    2018 Annual Report Alfred P. Sloan Foundation $ 2018 Annual Report Contents Preface II Mission Statement III From the President IV The Year in Discovery VI About the Grants Listing 1 2018 Grants by Program 2 2018 Financial Review 101 Audited Financial Statements and Schedules 103 Board of Trustees 133 Officers and Staff 134 Index of 2018 Grant Recipients 135 Cover: The Sloan Foundation Telescope at Apache Point Observatory, New Mexico as it appeared in May 1998, when it achieved first light as the primary instrument of the Sloan Digital Sky Survey. An early set of images is shown superimposed on the sky behind it. (CREDIT: DAN LONG, APACHE POINT OBSERVATORY) I Alfred P. Sloan Foundation $ 2018 Annual Report Preface The ALFRED P. SLOAN FOUNDATION administers a private fund for the benefit of the public. It accordingly recognizes the responsibility of making periodic reports to the public on the management of this fund. The Foundation therefore submits this public report for the year 2018. II Alfred P. Sloan Foundation $ 2018 Annual Report Mission Statement The ALFRED P. SLOAN FOUNDATION makes grants primarily to support original research and education related to science, technology, engineering, mathematics, and economics. The Foundation believes that these fields—and the scholars and practitioners who work in them—are chief drivers of the nation’s health and prosperity. The Foundation also believes that a reasoned, systematic understanding of the forces of nature and society, when applied inventively and wisely, can lead to a better world for all. III Alfred P. Sloan Foundation $ 2018 Annual Report From the President ADAM F.
    [Show full text]
  • Algorithms, Platforms, and Ethnic Bias: an Integrative Essay
    BERKELEY ROUNDTABLE ON THE INTERNATIONAL ECONOMY BRIE Working Paper 2018-3 Algorithms, Platforms, and Ethnic Bias: An Integrative Essay Selena Silva and Martin Kenney Algorithms, Platforms, and Ethnic Bias: An Integrative Essay In Phylon: The Clark Atlanta University Review of Race and Culture (Summer/Winter 2018) Vol. 55, No. 1 & 2: 9-37 Selena Silva Research Assistant and Martin Kenney* Distinguished Professor Community and Regional Development Program University of California, Davis Davis & Co-Director Berkeley Roundtable on the International Economy & Affiliated Professor Scuola Superiore Sant’Anna * Corresponding Author The authors wish to thank Obie Clayton for his encouragement and John Zysman for incisive and valuable comments on an earlier draft. Keywords: Digital bias, digital discrimination, algorithms, platform economy, racism 1 Abstract Racially biased outcomes have increasingly been recognized as a problem that can infect software algorithms and datasets of all types. Digital platforms, in particular, are organizing ever greater portions of social, political, and economic life. This essay examines and organizes current academic and popular press discussions on how digital tools, despite appearing to be objective and unbiased, may, in fact, only reproduce or, perhaps, even reinforce current racial inequities. However, digital tools may also be powerful instruments of objectivity and standardization. Based on a review of the literature, we have modified and extended a “value chain–like” model introduced by Danks and London, depicting the potential location of ethnic bias in algorithmic decision-making.1 The model has five phases: input, algorithmic operations, output, users, and feedback. With this model, we identified nine unique types of bias that might occur within these five phases in an algorithmic model: (1) training data bias, (2) algorithmic focus bias, (3) algorithmic processing bias, (4) transfer context bias, (5) misinterpretation bias, (6) automation bias, (7) non-transparency bias, (8) consumer bias, and (9) feedback loop bias.
    [Show full text]
  • Bayesian Sensitivity Analysis for Offline Policy Evaluation
    Bayesian Sensitivity Analysis for Offline Policy Evaluation Jongbin Jung Ravi Shroff [email protected] [email protected] Stanford University New York University Avi Feller Sharad Goel UC-Berkeley [email protected] [email protected] Stanford University ABSTRACT 1 INTRODUCTION On a variety of complex decision-making tasks, from doctors pre- Machine learning algorithms are increasingly used by employ- scribing treatment to judges setting bail, machine learning algo- ers, judges, lenders, and other experts to guide high-stakes de- rithms have been shown to outperform expert human judgments. cisions [2, 3, 5, 6, 22]. These algorithms, for example, can be used to One complication, however, is that it is often difficult to anticipate help determine which job candidates are interviewed, which defen- the effects of algorithmic policies prior to deployment, as onegen- dants are required to pay bail, and which loan applicants are granted erally cannot use historical data to directly observe what would credit. When determining whether to implement a proposed policy, have happened had the actions recommended by the algorithm it is important to anticipate its likely effects using only historical been taken. A common strategy is to model potential outcomes data, as ex-post evaluation may be expensive or otherwise imprac- for alternative decisions assuming that there are no unmeasured tical. This general problem is known as offline policy evaluation. confounders (i.e., to assume ignorability). But if this ignorability One approach to this problem is to first assume that treatment as- assumption is violated, the predicted and actual effects of an al- signment is ignorable given the observed covariates, after which a gorithmic policy can diverge sharply.
    [Show full text]
  • Designing Alternative Representations of Confusion Matrices to Support Non-Expert Public Understanding of Algorithm Performance
    153 Designing Alternative Representations of Confusion Matrices to Support Non-Expert Public Understanding of Algorithm Performance HONG SHEN, Carnegie Mellon University, USA HAOJIAN JIN, Carnegie Mellon University, USA ÁNGEL ALEXANDER CABRERA, Carnegie Mellon University, USA ADAM PERER, Carnegie Mellon University, USA HAIYI ZHU, Carnegie Mellon University, USA JASON I. HONG, Carnegie Mellon University, USA Ensuring effective public understanding of algorithmic decisions that are powered by machine learning techniques has become an urgent task with the increasing deployment of AI systems into our society. In this work, we present a concrete step toward this goal by redesigning confusion matrices for binary classification to support non-experts in understanding the performance of machine learning models. Through interviews (n=7) and a survey (n=102), we mapped out two major sets of challenges lay people have in understanding standard confusion matrices: the general terminologies and the matrix design. We further identified three sub-challenges regarding the matrix design, namely, confusion about the direction of reading the data, layered relations and quantities involved. We then conducted an online experiment with 483 participants to evaluate how effective a series of alternative representations target each of those challenges in the context of an algorithm for making recidivism predictions. We developed three levels of questions to evaluate users’ objective understanding. We assessed the effectiveness of our alternatives for accuracy in answering those questions, completion time, and subjective understanding. Our results suggest that (1) only by contextualizing terminologies can we significantly improve users’ understanding and (2) flow charts, which help point out the direction ofreading the data, were most useful in improving objective understanding.
    [Show full text]
  • Human Decisions and Machine Predictions*
    HUMAN DECISIONS AND MACHINE PREDICTIONS* Jon Kleinberg Himabindu Lakkaraju Jure Leskovec Jens Ludwig Sendhil Mullainathan August 11, 2017 Abstract Can machine learning improve human decision making? Bail decisions provide a good test case. Millions of times each year, judges make jail-or-release decisions that hinge on a prediction of what a defendant would do if released. The concreteness of the prediction task combined with the volume of data available makes this a promising machine-learning application. Yet comparing the algorithm to judges proves complicated. First, the available data are generated by prior judge decisions. We only observe crime outcomes for released defendants, not for those judges detained. This makes it hard to evaluate counterfactual decision rules based on algorithmic predictions. Second, judges may have a broader set of preferences than the variable the algorithm predicts; for instance, judges may care specifically about violent crimes or about racial inequities. We deal with these problems using different econometric strategies, such as quasi-random assignment of cases to judges. Even accounting for these concerns, our results suggest potentially large welfare gains: one policy simulation shows crime reductions up to 24.7% with no change in jailing rates, or jailing rate reductions up to 41.9% with no increase in crime rates. Moreover, all categories of crime, including violent crimes, show reductions; and these gains can be achieved while simultaneously reducing racial disparities. These results suggest that while machine learning can be valuable, realizing this value requires integrating these tools into an economic framework: being clear about the link between predictions and decisions; specifying the scope of payoff functions; and constructing unbiased decision counterfactuals.
    [Show full text]
  • Curriculum Vitae February, 2021 JENS LUDWIG
    Curriculum Vitae February, 2021 JENS LUDWIG OFFICE ADDRESS University of Chicago 1307 East 60th Street, Room 2065 Chicago, IL 60637 (773) 834-0811 [email protected] EDUCATION Ph.D. Economics, Duke University, Durham, NC 1994 B.A. Economics, (Minor in Religion), Rutgers College, New Brunswick, NJ, 1990 CURRENT ACADEMIC APPOINTMENTS 2019-present Edwin A. and Betty L. Bergman Distinguished Service Professor, University of Chicago 2019-present Pritzker Co-Director, University of Chicago Urban Education Lab 2019-present Pritzker Director, University of Chicago Crime Lab PREVIOUS ACADEMIC APPOINTMENTS 2011-2019 Co-Director, University of Chicago Urban Education Lab 2008-2019 Director, University of Chicago Crime Lab 2008-2018 McCormick Foundation Professor of Social Service Administration, Law, and Public Policy, University of Chicago 2011-2012 Academic Director for Students, Harris School for Public Policy Studies, University of Chicago 2007-2008 Professor of Social Service Administration, Law, and Public Policy, University of Chicago 2006-2007 Professor of Public Policy, Georgetown University 2006-2007 Associate Dean for Public Policy Admissions, Georgetown University 2001-2002 Andrew W. Mellon Fellow in Economic Studies, The Brookings Institution 2001-2006 Associate Professor of Public Policy, Georgetown University 1997-1998 Visiting Scholar, Northwestern / University of Chicago Joint Center for Poverty Research 1994-2001 Assistant Professor of Public Policy, Georgetown University OTHER AFFILIATIONS 2014-2019 Crime Program Chair, Abdul
    [Show full text]
  • Examining Wikipedia with a Broader Lens
    Examining Wikipedia With a Broader Lens: Quantifying the Value of Wikipedia’s Relationships with Other Large-Scale Online Communities Nicholas Vincent Isaac Johnson Brent Hecht PSA Research Group PSA Research Group PSA Research Group Northwestern University Northwestern University Northwestern University [email protected] [email protected] [email protected] ABSTRACT recruitment and retention strategies (e.g. [17,18,59,62]), and The extensive Wikipedia literature has largely considered even to train many state-of-the-art artificial intelligence Wikipedia in isolation, outside of the context of its broader algorithms [12,13,25,26,36,58]. Indeed, Wikipedia has Internet ecosystem. Very recent research has demonstrated likely become one of the most important datasets and the significance of this limitation, identifying critical research environments in modern computing [25,28,36,38]. relationships between Google and Wikipedia that are highly relevant to many areas of Wikipedia-based research and However, a major limitation of the vast majority of the practice. This paper extends this recent research beyond Wikipedia literature is that it considers Wikipedia in search engines to examine Wikipedia’s relationships with isolation, outside the context of its broader online ecosystem. large-scale online communities, Stack Overflow and Reddit The importance of this limitation was made quite salient in in particular. We find evidence of consequential, albeit recent research that suggested that Wikipedia’s relationships unidirectional relationships. Wikipedia provides substantial with other websites are tremendously significant value to both communities, with Wikipedia content [15,35,40,57]. For instance, this work has shown that increasing visitation, engagement, and revenue, but we find Wikipedia’s relationships with other websites are important little evidence that these websites contribute to Wikipedia in factors in the peer production process and, consequently, have impacts on key variables of interest such as content return.
    [Show full text]
  • Graph Evolution: Densification and Shrinking Diameters
    Graph Evolution: Densification and Shrinking Diameters JURE LESKOVEC Carnegie Mellon University JON KLEINBERG Cornell University and CHRISTOS FALOUTSOS Carnegie Mellon University How do real graphs evolve over time? What are normal growth patterns in social, technological, and information networks? Many studies have discovered patterns in static graphs, identifying properties in a single snapshot of a large network or in a very small number of snapshots; these include heavy tails for in- and out-degree distributions, communities, small-world phenomena, and others. However, given the lack of information about network evolution over long periods, it has been hard to convert these findings into statements about trends over time. Here we study a wide range of real graphs, and we observe some surprising phenomena. First, most of these graphs densify over time with the number of edges growing superlinearly in the number of nodes. Second, the average distance between nodes often shrinks over time in contrast to the conventional wisdom that such distance parameters should increase slowly as a function of the number of nodes (like O(log n)orO(log(log n)). Existing graph generation models do not exhibit these types of behavior even at a qualitative level. We provide a new graph generator, based on a forest fire spreading process that has a simple, intuitive justification, requires very few parameters (like the flammability of nodes), and produces graphs exhibiting the full range of properties observed both in prior work and in the present study. This material is based on work supported by the National Science Foundation under Grants No. IIS-0209107 SENSOR-0329549 EF-0331657IIS-0326322 IIS- 0534205, CCF-0325453, IIS-0329064, CNS-0403340, CCR-0122581; a David and Lucile Packard Foundation Fellowship; and also by the Pennsylvania Infrastructure Technology Alliance (PITA), a partnership of Carnegie Mellon, Lehigh University and the Commonwealth of Pennsylvania’s Department of Community and Economic Development (DCED).
    [Show full text]
  • CURRICULUM Vitae Jure LESKOVEC
    CURRICULUM vitae Jure LESKOVEC Department of Computer Science Phone: (650) 723 - 2273 Stanford University Cell: (412) 478 - 8329 353 Serra Mall E-mail: [email protected] Stanford CA 93405 http://www.cs.stanford.edu/∼jure Positions held Stanford University, Stanford , CA. Assistant Professor. Department of Computer Science. August 2009. Cornell University, Ithaca, NY. Postdoctoral researcher. Department of Computer Science. September 2008 – July 2009. Education Carnegie Mellon University, Pittsburgh, PA. Ph.D. in Computational and Statistical Learning, Machine Learning Department, School of Computer Science. September 2008. University of Ljubljana, Slovenia. Diploma [B.Sc.] in Computer Science (Summa Cum Laude), May 2004. Research Applied machine learning and large-scale data mining, focusing on the analysis and interests modeling of large real-world networks. Academic ACM SIGKDD Dissertation Award 2009 honors Best Research Paper Award at ASCE Journal of Water Resources Planning and Man- agement Engineering, 2009. Best student paper award at 13th ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, KDD 2007. Best solution of Battle of the Water Sensor Networks (BWSN) competition on sensor placement in water distribution networks, 2007. Microsoft Graduate Research Fellowship, 2006 - 2008. Best research paper award at 11th ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining, KDD 2005. Slovenian Academy of Arts and Sciences Fellowship, 2004. University of Ljubljana Preserenˇ Award (highest university award for students given to 10 out of 40,000 sutdents), 2004. Winner of the KDD Cup 2003 on estimating the download rate of scientific articles on Arxiv. KDD Cup is a data mining competition held in conjunction with the Annual ACM SIGKDD Conference.
    [Show full text]
  • Jure Leskovec
    Jure Leskovec Computer Science Department Stanford University Includes joint work with Deepay Chakrabarti, Anirban Dasgupta, Christos Faloutsos, Jon Kleinberg, Kevin Lang and Michael Mahoney Workshop on Complex Networks in Information & Knowledge Management, CIKM 2009 Interaction graph model of networks: . Nodes represent “entities” . Edges represent “interactions” between pairs of entities Thinking of data in a form of a network makes the difference: . Google realized web-pages are connected . Collective classification, targeting Jure Leskovec, CNIKM '09 2 Online social networks Collaboration networks Systems biology networks arXiv DBLP Communication networks Web graph & citation networks Internet Jure Leskovec, CNIKM '09 3 Complex networks as phenomena, not just designed artifacts What are the common patterns that emerge? Flickr social network Scale free networks n= 584,207, m=3,555,115 many more hubs than expected Jure Leskovec, CNIKM '09 4 Network data spans many orders of magnitude: . 436-node network of email exchange over 3-months at corporate research lab [Adamic-Adar, SocNets ‘03] . 43,553-node network of email exchange over 2 years at a large university [Kossinets-Watts, Science ‘06] . 4.4-million-node network of declared friendships on a blogging community [Backstrom et at., KDD ‘06] . 240-million-node network of all IM communication over a month on Microsoft Instant Messenger [Leskovec-Horvitz, WWW ‘08] Jure Leskovec, CNIKM '09 5 [Leskovec et al. KDD 05] How do network properties scale with the size of the network? Citations Citations ) a=1.6 E(t diameter N(t) time Densification Shrinking diameter Average degree increases Path lengths get shorter Jure Leskovec, CNIKM '09 6 What do we hope to achieve by analyzing networks? .
    [Show full text]