Prediction, Judgment, and Complexity: a Theor

Prediction, Judgment, and Complexity: A Theor...

The Economics of Artiﬁ cial Intelligence

National Bureau of Economic Research Conference Report

The Economics of Artiﬁ cial Intelligence: An Agenda

Edited by Ajay Agrawal, Joshua Gans, and Avi Goldfarb

The University of Chicago Press

Chicago and London

The University of Chicago Press, Chicago 60637 The University of Chicago Press, Ltd., London © 2019 by the National Bureau of Economic Research, Inc. All rights reserved. No part of this book may be used or reproduced in any manner whatsoever without written permission, except in the case of brief quotations in critical articles and reviews. For more information, contact the University of Chicago Press, 1427 E. 60th St., Chicago, IL 60637. Published 2019 Printed in the United States of America

28 27 26 25 24 23 22 21 20 19 1 2 3 4 5

ISBN-13: 978-0-226-61333-8 (cloth) ISBN-13: 978-0-226-61347-5 (e-book) DOI: https:// doi .org / 10 .7208 / chicago / 9780226613475 .001 .0001

Library of Congress Cataloging-in-Publication Data Names: Agrawal, Ajay, editor. | Gans, Joshua, 1968– editor. | Goldfarb, Avi, editor. Title: The economics of artifi cial intelligence : an agenda / Ajay Agrawal, Joshua Gans, and Avi Goldfarb, editors. Other titles: National Bureau of Economic Research conference report. Description: Chicago ; London : The University of Chicago Press, 2019. | Series: National Bureau of Economic Research conference report | Includes bibliographical references and index. Identifi ers: LCCN 2018037552 | ISBN 9780226613338 (cloth : alk. paper) | ISBN 9780226613475 (ebook) Subjects: LCSH: Artifi cial intelligence—Economic aspects. Classifi cation: LCC TA347.A78 E365 2019 | DDC 338.4/70063—dc23 LC record available at https:// lccn .loc .gov / 2018037552

♾ This paper meets the requirements of ANSI/NISO Z39.48-1992 (Permanence of Paper).

DOI: 10.7208/chicago/9780226613475.003.0003

National Bureau of Economic Research

Offi cers Karen N. Horn, chair Kelly Horak, controller and assistant John Lipsky, vice chair corporate secretary James M. Poterba, president and chief Alterra Milone, corporate secretary executive offi cer Denis Healy, assistant corporate secretary Robert Mednick, treasurer

Directors at Large Peter C. Aldrich Diana Farrell Michael H. Moskow Elizabeth E. Bailey Jacob A. Frenkel Alicia H. Munnell John H. Biggs Robert S. Hamada Robert T. Parry John S. Clarkeson Peter Blair Henry James M. Poterba Kathleen B. Cooper Karen N. Horn John S. Reed Charles H. Dallara Lisa Jordan Marina v. N. Whitman George C. Eads John Lipsky Martin B. Zimmerman Jessica P. Einhorn Laurence H. Meyer Mohamed El-Erian Karen Mills

Directors by University Appointment Timothy Bresnahan, Stanford George Mailath, Pennsylvania Pierre-André Chiappori, Columbia Marjorie B. McElroy, Duke Alan V. Deardorff , Michigan Joel Mokyr, Northwestern Edward Foster, Minnesota Cecilia Rouse, Princeton John P. Gould, Chicago Richard L. Schmalensee, Massachusetts Mark Grinblatt, California, Los Angeles Institute of Technology Bruce Hansen, Wisconsin–Madison Ingo Walter, New York Benjamin Hermalin, California, Berkeley David B. Yoffi e, Harvard Samuel Kortum, Yale

Directors by Appointment of Other Organizations Jean-Paul Chavas, Agricultural and Applied Robert Mednick, American Institute of Economics Association Certiﬁ ed Public Accountants Martin J. Gruber, American Finance Peter L. Rousseau, American Economic Association Association Philip Hoff man, Economic History Gregor W. Smith, Canadian Economics Association Association Arthur Kennickell, American Statistical William Spriggs, American Federation Association of Labor and Congress of Industrial Jack Kleinhenz, National Association for Organizations Business Economics Bart van Ark, The Conference Board

Directors Emeriti George Akerlof Franklin Fisher John J. Siegfried Jagdish Bhagwati Saul H. Hymans Craig Swan Don R. Conlan Rudolph A. Oswald Ray C. Fair Andrew Postlewaite

Relation of the Directors to the Work and Publications of the National Bureau of Economic Research

1. The object of the NBER is to ascertain and present to the economics profession, and to the public more generally, important economic facts and their interpretation in a scientifi c manner without policy recommendations. The Board of Directors is charged with the responsibility of ensuring that the work of the NBER is carried on in strict conformity with this object. 2. The President shall establish an internal review process to ensure that book manuscripts pro- posed for publication DO NOT contain policy recommendations. This shall apply both to the proceedings of conferences and to manuscripts by a single author or by one or more co-authors but shall not apply to authors of comments at NBER conferences who are not NBER affi liates. 3. No book manuscript reporting research shall be published by the NBER until the President has sent to each member of the Board a notice that a manuscript is recommended for publication and that in the President’s opinion it is suitable for publication in accordance with the above principles of the NBER. Such notifi cation will include a table of contents and an abstract or summary of the manuscript’s content, a list of contributors if applicable, and a response form for use by Directors who desire a copy of the manuscript for review. Each manuscript shall contain a summary drawing attention to the nature and treatment of the problem studied and the main conclusions reached. 4. No volume shall be published until forty-fi ve days have elapsed from the above notifi cation of intention to publish it. During this period a copy shall be sent to any Director requesting it, and if any Director objects to publication on the grounds that the manuscript contains policy recommendations, the objection will be presented to the author(s) or editor(s). In case of dispute, all members of the Board shall be notifi ed, and the President shall appoint an ad hoc committee of the Board to decide the matter; thirty days additional shall be granted for this purpose. 5. The President shall present annually to the Board a report describing the internal manuscript review process, any objections made by Directors before publication or by anyone after publication, any disputes about such matters, and how they were handled. 6. Publications of the NBER issued for informational purposes concerning the work of the Bureau, or issued to inform the public of the activities at the Bureau, including but not limited to the NBER Digest and Reporter, shall be consistent with the object stated in paragraph 1. They shall contain a specifi c disclaimer noting that they have not passed through the review procedures required in this resolution. The Executive Committee of the Board is charged with the review of all such publications from time to time. 7. NBER working papers and manuscripts distributed on the Bureau’s web site are not deemed to be publications for the purpose of this resolution, but they shall be consistent with the object stated in paragraph 1. Working papers shall contain a specifi c disclaimer noting that they have not passed through the review procedures required in this resolution. The NBER’s web site shall contain a similar disclaimer. The President shall establish an internal review process to ensure that the working papers and the web site do not contain policy recommendations, and shall report annually to the Board on this process and any concerns raised in connection with it. 8. Unless otherwise determined by the Board or exempted by the terms of paragraphs 6 and 7, a copy of this resolution shall be printed in each NBER publication as described in paragraph 2 above. Contents

Acknowledgments xi Introduction 1 Ajay Agrawal, Joshua Gans, and Avi Goldfarb

I. AI as a GPT 1. Artifi cial Intelligence and the Modern Productivity Paradox: A Clash of Expectations and Statistics 23 Erik Brynjolfsson, Daniel Rock, and Chad Syverson Comment: Rebecca Henderson 2. The Technological Elements of Artifi cial Intelligence 61 Matt Taddy 3. Prediction, Judgment, and Complexity: A Theory of Decision-Making and Artifi cial Intelligence 89 Ajay Agrawal, Joshua Gans, and Avi Goldfarb Comment: Andrea Prat 4. The Impact of Artifi cial Intelligence on Innovation: An Exploratory Analysis 115 Iain M. Cockburn, Rebecca Henderson, and Scott Stern Comment: Matthew Mitchell

vii viii Contents

5. Finding Needles in Haystacks: Artiﬁ cial Intelligence and Recombinant Growth 149 Ajay Agrawal, John McHale, and Alexander Oettl 6. Artiﬁ cial Intelligence as the Next GPT: A Political-Economy Perspective 175 Manuel Trajtenberg

II. Growth, Jobs, and Inequality 7. Artifi cial Intelligence, Income, Employment, and Meaning 189 Betsey Stevenson 8. Artifi cial Intelligence, Automation, and Work 197 Daron Acemoglu and Pascual Restrepo 9. Artifi cial Intelligence and Economic Growth 237 Philippe Aghion, Benjamin F. Jones, and Charles I. Jones Comment: Patrick Francois 10. Artifi cial Intelligence and Jobs: The Role of Demand 291 James Bessen 11. Public Policy in an AI Economy 309 Austan Goolsbee 12. Should We Be Reassured If Automation in the Future Looks Like Automation in the Past? 317 Jason Furman 13. R&D, Structural Transformation, and the Distribution of Income 329 Jeff rey D. Sachs 14. Artifi cial Intelligence and Its Implications for Income Distribution and Unemployment 349 Anton Korinek and Joseph E. Stiglitz 15. Neglected Open Questions in the Economics of Artifi cial Intelligence 391 Tyler Cowen Contents ix

III. Machine Learning and Regulation 16. Artifi cial Intelligence, Economics, and Industrial Organization 399 Hal Varian Comment: Judith Chevalier 17. Privacy, Algorithms, and Artifi cial Intelligence 423 Catherine Tucker 18. Artifi cial Intelligence and Consumer Privacy 439 Ginger Zhe Jin 19. Artifi cial Intelligence and International Trade 463 Avi Goldfarb and Daniel Trefl er 20. Punishing Robots: Issues in the Economics of Tort Liability and Innovation in Artifi cial Intelligence 493 Alberto Galasso and Hong Luo

IV. Machine Learning and Economics 21. The Impact of Machine Learning on Economics 507 Susan Athey Comment: Mara Lederman 22. Artifi cial Intelligence, Labor, Productivity, and the Need for Firm-Level Data 553 Manav Raj and Robert Seamans 23. How Artifi cial Intelligence and Machine Learning Can Impact Market Design 567 Paul R. Milgrom and Steven Tadelis 24. Artifi cial Intelligence and Behavioral Economics 587 Colin F. Camerer Comment: Daniel Kahneman Contributors 611 Author Index 615 Subject Index 625

Acknowledgments

This volume contains chapters and ideas discussed at the fi rst NBER Con- ference on the Economics of Artifi cial Intelligence, held in September 2017 in Toronto. We thank all the authors and discussants for their contributions. Funds for the conference and book project were provided by the Sloan Foun- dation, the Canadian Institute for Advanced Research, and the Creative Destruction Lab at the University of Toronto. At the Sloan Foundation, Danny Goroff provided guidance that improved the overall agenda. The NBER digitization initiative, under the leadership of Shane Greenstein, was a key early supporter. We thank our dean, Tiff Macklem. In addition, Jim Poterba at the NBER has been generous, giving us the fl exibility needed to bring this project together. Special thanks are due to Rob Shannon, Denis Healy, Carl Beck, and Dawn Bloomfi eld for managing the conference and logistics and to Helena Fitz-Patrick for guiding the book through the edito- rial process. Finally we thank our families, Gina, Natalie, Rachel, Amelia, Andreas, Belanna, Ariel, Annika, Anna, Sam, and Ben.

3 Prediction, Judgment, and Complexity A Theory of Decision-Making and Artiﬁ cial Intelligence

Ajay Agrawal, Joshua Gans, and Avi Goldfarb

3.1 Introduction There is widespread discussion regarding the impact of machines on employment (see Autor 2015). In some sense, the discussion mirrors a long- standing literature on the impact of the accumulation of capital equipment on employment; speciﬁ cally, whether capital and labor are substitutes or complements (Acemoglu 2003). But the recent discussion is motivated by the integration of software with hardware and whether the role of machines goes beyond physical tasks to mental ones as well (Brynjolfsson and McAfee 2014). As mental tasks were seen as always being present and essential, human comparative advantage in these was seen as the main reason why, at least in the long term, capital accumulation would complement employment by enhancing labor productivity in those tasks. The computer revolution has blurred the line between physical and men-

Ajay Agrawal is the Peter Munk Professor of Entrepreneurship at the Rotman School of Management, University of Toronto, and a research associate of the National Bureau of Eco- nomic Research. Joshua Gans is professor of strategic management and holder of the Jeff rey S. Skoll Chair of Technical Innovation and Entrepreneurship at the Rotman School of Manage- ment, University of Toronto (with a cross appointment in the Department of Economics), and a research associate of the National Bureau of Economic Research. Avi Goldfarb holds the Rotman Chair in Artiﬁ cial Intelligence and Healthcare and is professor of marketing at the Rotman School of Management, University of Toronto, and is a research associate of the National Bureau of Economic Research. Our thanks to Andrea Prat, Scott Stern, Hal Varian, and participants at the AEA (Chicago), NBER Summer Institute (2017), NBER Economics of AI Conference (Toronto), Columbia Law School, Harvard Business School, MIT, and University of Toronto for helpful comments. Responsibility for all errors remains our own. The latest version of this chapter is available at joshuagans .com. For acknowledgments, sources of research support, and disclosure of the authors’ material ﬁ nancial relationships, if any, please see http:// www .nber .org/ chapters / c14010.ack.

89 90 Ajay Agrawal, Joshua Gans, and Avi Goldfarb tal tasks. For instance, the invention of the spreadsheet in the late 1970s fundamentally changed the role of bookkeepers. Prior to that invention, there was a time- intensive task involving the recomputation of outcomes in spreadsheets as data or assumptions changed. That human task was substituted by the spreadsheet software that could produce the calculations more quickly, cheaply, and frequently. However, at the same time, the spreadsheet made the jobs of accountants, analysts, and others far more productive. In the accounting books, capital was substituting for labor, but the mental productivity of labor was being changed. Thus, the impact on employment critically depended on whether there were tasks the “computers cannot do.” These assumptions persist in models today. Acemoglu and Restrepo (2017) observe that capital substitutes for labor in certain tasks while at the same time technological progress creates new tasks. They make what they call a “natural assumption” that only labor can perform the new tasks as they are more complex than previous ones.1 Benzell et al. (2015) consider the impact of software more explicitly. Their environment has two types of labor—high-tech (who can, among other things, code) and low-tech (who are empathetic and can handle interpersonal tasks). In this environment, it is the low- tech workers who cannot be replaced by machines while the high- tech ones are employed initially to create the code that will eventually displace their kind. The results of the model depend, therefore, on a class of worker who cannot be substituted directly for capital, but also on the inability of workers themselves to substitute between classes. In this chapter, our approach is to delve into the weeds of what is hap- pening currently in the ﬁ eld of artiﬁ cial intelligence (AI). The recent wave of developments in AI all involve advances in machine learning. Those advances allow for automated and cheap prediction; that is, providing a forecast (or nowcast) of a variable of interest from available data (Agrawal, Gans and Goldfarb 2018b). In some cases, prediction has enabled full automation of tasks—for example, self-driving vehicles where the process of data collection, prediction of behavior and surroundings, and actions are all conducted without a human in the loop. In other cases, prediction is a standalone tool—such as image recognition or fraud detection—that may or may not lead to further substitution of human users of such tools by machines. Thus far, substitution between humans and machines has focused mainly on cost considerations. Are machines cheaper, more reliable, and more scalable (in their software form) than humans? This chapter, however, considers the role of prediction in decision- making explicitly and from that examines the complementary skills that may be matched with prediction within a task.

1. To be sure, their model is designed to examine how automation of tasks causes a change in factor prices that biases innovation toward the creation of new tasks that labor is more suited to. A Theory of Decision- Making and Artiﬁ cial Intelligence 91

Our focus, in this regard, is on what we term judgment. While judgment is a term with broad meaning, here we use it to refer to a very speciﬁ c skill. To see this, consider a decision. That decision involves choosing an action, x, from a set, X. The payoff (or reward) from that action is deﬁ ned by a function, u(x, ) where is a realization of an uncertain state drawn from a distribution, F(). Suppose that, prior to making a decision, a prediction (or signal), s, can be generated that results in a posterior, F(|s). Thus, the decision maker would solve

maxx X ux(), dF()s . In other words, a standard problem of choice under uncertainty. In this standard world, the role of prediction is to improve decision- making. The payoff , or utility function, is known. To create a role for judgment, we depart from this standard set-up in statistical decision theory and ask how a decision maker comes to know the function, u(x, )? We assume that this is not simply given or a primitive of the decision- making model. Instead, it requires a human to undertake a costly process that allows the mapping from (x, ) to a particular payoff value, u, to be discovered. This is a reasonable assumption given that beyond some rudi- mentary experimentation in closed environments, there is no current way for an AI to impute a utility function that resides with humans. Additionally, this process separates the costs of providing the mapping for each pair, (x, ). (Actually, we focus, without loss in generality, on situations where u(x, ) ≠ u(x) for all and presume that if a payoff to an action is state independent that payoff is known.) In other words, while prediction can obtain a signal of the underlying state, judgment is the process by which the payoff s from actions that arise based on that state can be determined. We assume that this process of determining payoff s requires human understanding of the situation: it is not a prediction problem. For intuition on the diff erence between prediction and judgment, consider the example of credit card fraud. A bank observes a credit card transaction. That transaction is either legitimate or fraudulent. The decision is whether to approve the transaction. If the bank knows for sure that the transaction is legitimate, the bank will approve it. If the bank knows for sure that it is fraudulent, the bank will refuse the transaction. Why? Because the bank knows the payoff of approving a legitimate transaction is higher than the payoff of refusing that transaction. Things get more interesting if the bank is uncertain about whether the transaction is legitimate. The uncertainty means that the bank also needs to know the payoff from refusing a legitimate transaction and from approving a fraudulent transaction. In our model, judgment is the process of determining these payoff s. It is a costly activity, in the sense that it requires time and eff ort. As the new developments regarding AI all involve making prediction more readily available, we ask, how does judgment and its endogenous appli- 92 Ajay Agrawal, Joshua Gans, and Avi Goldfarb cation change the value of prediction? Are prediction and judgment substitutes or complements? How does the value of prediction change mono- tonically with the diffi culty of applying judgment? In complex environments (as they relate to automation, contracting, and the boundaries of the fi rm), how do improvements in prediction aff ect the value of judgment? We proceed by fi rst providing supportive evidence for our assumption that recent developments in AI overwhelmingly impact the costs of prediction. We then use the example of radiology to provide a context for understanding the diff erent roles of prediction and judgment. Drawing inspiration from Bolton and Faure-Grimaud (2009), we then build the baseline model with two states of the world and uncertainty about payoff s to actions in each state. We explore the value of judgment in the absence of any prediction technology, and then the value of prediction technology when there is no judgment. We fi nish the discussion of the baseline model with an explora- tion of the interaction between prediction and judgment, demonstrating that prediction and judgment are complements as long as judgment isn’t too diffi cult. We then separate prediction quality into prediction frequency and prediction accuracy. As judgment improves, accuracy becomes more important relative to frequency. Finally, we examine complex environments where the number of potential states is large. Such environments are common in economic models of automation, contracting, and boundaries of the fi rm. We show that the eff ect of improvements in prediction on the importance of judgment depend a great deal on whether the improvements in prediction enable automated decision-making.

3.2 AI and Prediction Costs We argue that the recent advances in artifi cial intelligence are advances in the technology of prediction. Most broadly, we defi ne prediction as the ability to take known information to generate new information. Our model emphasizes prediction about the state of the world. Most contemporary artifi cial intelligence research and applications come from a fi eld now called “machine learning.” Many of the tools of machine learning have a long history in statistics and data analysis, and are likely familiar to economists and applied statisticians as tools for prediction and classifi cation.2 For example, Alpaydin’s (2010) textbook Introduction to Machine Learning covers maximum likelihood estimation, Bayesian estimation, multivariate linear regression, principal components analysis, clustering, and nonparametric regression. In addition, it covers tools that may be less familiar, but also use independent variables to predict outcomes:

2. We defi ne prediction as known information to generate new information. Therefore, classifi cation techniques such as clustering are prediction techniques in which the new information to be predicted is the appropriate category or class. A Theory of Decision- Making and Artifi cial Intelligence 93 regression trees, neural networks, hidden Markov models, and reinforcement learning. Hastie, Tibshirani, and Friedman (2009) cover similar topics. The 2014 Journal of Economic Perspectives symposium on big data covered several of these less familiar prediction techniques in articles by Varian (2014) and Belloni, Chernozhukov, and Hansen (2014). While many of these prediction techniques are not new, recent advances in computer speed, data collection, data storage, and the prediction methods themselves have led to substantial improvements. These improvements have transformed the computer science research fi eld of artifi cial intelligence. The Oxford English Dictionary defi nes artifi cial intelligence as “[t]he theory and development of computer systems able to perform tasks normally requiring human intelligence.” In the 1960s and 1970s, artifi cial intelligence research was primarily rules- based, symbolic logic. It involved human experts generating rules that an algorithm could follow (Domingos 2015, 89). These are not prediction technologies. Such systems became very good chess players and they guided factory robots in highly controlled settings; however, by the 1980s, it became clear that rules- based systems could not deal with the complexity of many nonartifi cial settings. This led to an “AI winter” in which research funding artifi cial intelligence projects largely dried up (Markov 2015). Over the past ten years, a diff erent approach to artifi cial intelligence has taken off . The idea is to program computers to “learn” from example data or experience. In the absence of the ability to predetermine the decision rules, a data-driven prediction approach can conduct many mental tasks. For example, humans are good at recognizing familiar faces, but we would struggle to explain and codify this skill. By connecting data on names to image data on faces, machine learning solves this problem by predicting which image data patterns are associated with which names. As a prominent artifi cial intelligence researcher put it, “Almost all of AI’s recent progress is through one type, in which some input data (A) is used to quickly generate some simple response (B)” (Ng 2016). Thus, the progress is explicitly about improvements in prediction. In other words, the suite of technologies that have given rise to the recent resurgence of interest in artifi cial intelligence use data collected from sensors, images, videos, typed notes, or anything else that can be represented in bits to fi ll in missing information, recognize objects, or forecast what will happen next. To be clear, we do not take a position on whether these prediction technologies really do mimic the core aspects of human intelligence. While Palm Computing founder Jeff Hawkins argues that human intelligence is—in essence—prediction (Hawkins 2004), many neuroscientists, psychologists, and others disagree. Our point is that the technologies that have been given the label artifi cial intelligence are prediction technologies. Therefore, in order to understand the impact of these technologies, it is important to assess the impact of prediction on decisions. 94 Ajay Agrawal, Joshua Gans, and Avi Goldfarb

3.3 Case: Radiology Before proceeding to the model, we provide some intuition of how prediction and judgment apply in a particular context where prediction machines are expected to have a large impact: radiology. In 2016, Geoff Hinton—one of the pioneers of deep learning neural networks—stated that it was no longer worth training radiologists. His strong implication was that radiologists would not have a future. This is something that radiologists have been con- cerned about since 1960 (Lusted 1960). Today, machine- learning techniques are being heavily applied in radiology by IBM using its Watson computer and by a start-up, Enlitic. Enlitic has been able to use deep learning to detect lung nodules (a fairly routine exercise)3 but also fractures (which is more complex). Watson can now identify pulmonary embolism and some other heart issues. These advances are at the heart of Hinton’s forecast, but have also been widely discussed among radiologists and pathologists (Jha and Topol 2016). What does the model in this chapter suggest about the future of radiologists? If we consider a simplifi ed characterization of the job of a radiologist, it would be that they examine an image in order to characterize and clas- sify that image and return an assessment to a physician. While often that assessment is a diagnosis (i.e., “the patient has pneumonia”), in many cases, the assessment is in the negative (i.e., “pneumonia not excluded”). In that regard, this is stated as a predictive task to inform the physician of the likelihood of the state of the world. Using that, the physician can devise a treatment. These predictions are what machines are aiming to provide. In particular, it might provide a diff erential diagnosis of the following kind: Based on Mr Patel’s demographics and imaging, the mass in the liver has a 66.6 percent chance of being benign, 33.3 percent chance of being malignant, and a 0.1 percent of not being real.4 The action is whether some intervention is needed. For instance, if a potential tumor is identifi ed in a noninvasive scan, then this will inform whether an invasive examination will be conducted. In terms of identifying the state of the world, the invasive exam is costly but safe—it can deduce a cancer with certainty and remove it if necessary. The role of a noninvasive exam is to inform whether an invasive exam should be forgone. That is, it is to make physicians more confi dent about abstaining from treatment and further analysis. In this regard, if the machine improves prediction, it will lead to fewer invasive examinations.

3. “You did not go to medical school to measure lung nodules.” http://www .medscape .com / viewarticle/ 863127#vp_2. 4. http:// www .medscape .com/ viewarticle/ 863127#vp_3. A Theory of Decision- Making and Artiﬁ cial Intelligence 95

Judgment involves understanding the payoff s. What is the payoff to con- ducting a biopsy if the mass is benign, malignant, or not real? What is the payoff to not doing anything in those three states? The issue for radiologists in particular is whether a trained specialist radiologist is in the best position to make this judgment or will it occur further along the chain of decision- making or involve new job classes that merge diagnostic information such as a combined radiologist/pathologist (Jha and Topol 2016). Next, we for- malize these ideas.

3.4 Baseline Model Our baseline model is inspired by the “bandit” environment considered by Bolton and Faure- Grimaud (2009), although it departs signiﬁ cantly in the questions addressed and base assumptions made. Like them, in our baseline model, we suppose there are two states of the world, { 1, 2} with prior probabilities of {,1 – }. There are two possible actions: a state independent action with known payoff of S (safe) and a state dependent action with two possible payoff s, R or r, as the case may be (risky). As noted in the introduction, a key departure from the usual assumptions of rational decision-making is that the decision maker does not know the payoff from the risky action in each state and must apply judgment to determine that payoff .5 Moreover, decision makers need to be able to make a judgment for each state that might arise in order to formulate a plan that would be the equivalent of payoff maximization. In the absence of such judgment, the ex ante expectation that the risky action is optimal in any state is v (which is independent between states). To make things more concrete, we assume R > S > r.6 Thus, we assume that v is the probability in any state that the risky payoff is R rather than r. This is not a conditional probability of the state. It is a statement about the payoff , given the state. In the absence of knowledge regarding the speciﬁ c payoff s from the risky action, a decision can only be made on the basis of prior probabilities. Then the safe action will be chosen if μ()vR + (1 v)r + (1 μ)()vR + (1 v)r = vR + (1 v)r S.

5. Bolton and Faure-Grimaud (2009) consider this step to be the equivalent of a thought experiment where thinking takes time. To the extent that our results can be interpreted as a statement about the comparative advantage of humans, we assume that only humans can do judgment. 6. Thus, we assume that the payoff function, u, can only take one of three values, {R, r, S}. The issue is which combinations of state realization and action lead to which payoff s. However, we assume that S is the payoff from the safe action regardless of state and so this is known to the decision maker. As it is the relative payoff s from actions that drive the results, this assumption is without loss in generality. Requiring this property of the safe action to be discovered would just add an extra cost. Implicitly, as the decision maker cannot make a decision in complete ignorance, we are assuming that the safe action’s payoff can be judged at an arbitrarily low cost. 96 Ajay Agrawal, Joshua Gans, and Avi Goldfarb

So that the payoff is: V0 = max{vR + (1 – v)r, S}. To make things simpler, we will focus our attention on the case where the safe action is—in the absence of prediction or judgment—the default. That is, we assume that (A1) (Safe Default) vR + (1 – v)r ≤ S. This assumption is made for simplicity only and will not change the qualitative conclusions.7 Under (A1), in the absence of knowledge of the payoff function or a signal of the state, the decision maker would choose S. 3.4.1 Judgment in the Absence of Prediction Prediction provides knowledge of the state. The process of judgment provides knowledge of the payoff function. Judgment therefore allows the decision maker to understand which action is optimal for a given state should it arise. Suppose that this knowledge is gained without cost (as it would be assumed to do under the usual assumptions of economic rationality). In other words, the decision maker has knowledge of optimal action in a given state. Then the risky action will be chosen (a) if it is the preferred action in both states (which arises with probability v2); (b) if it is the preferred action in 1 but not 2 and R + (1 – )r > S (with probability v(1 – v)); or (c) if it is the preferred action in 2 but not 1 and r + (1 – )R > S (with probability v(1 – v)). Thus, the expected payoff is v2 R + v(1 v)max{}μR + (1 μ)r,S + v(1 v)max{}μr + (1 μ)R,S + (1 v)2 S.

Note that this is greater than V0. The reason for this is that, when there is uncertainty, judgment is valuable because it can identify actions that are dominant or dominated—that is, that might be optimal across states. In this situation, any resolution of uncertainty does not matter as it will not change the decision made. A key insight is that judgment itself can be consequential. Result 1: If max{R + (1 – )r, r + (1 – )R} > S, it is possible that judgment alone can cause the decision to switch from the default action (safe) to the alternative action (risky). As we are motivated by understanding the interplay between prediction and judgment, we want to make these consequential. Therefore, we make the following assumption to ensure prediction always has some value: (A2) (Judgment Insuffi cient) max{R + (1 – )r, r + (1 – )R} ≤ S. Under this assumption, if diff erent actions are optimal in each state and this is known, the decision maker will not change to the risky action. This, of course, implies that the expected payoff is

7. Bolton and Faure-Grimaud (2009) make the opposite assumption. Here, as our focus is on the impact of prediction, it is better to consider environments where prediction has the eff ect of reducing uncertainty over riskier actions. A Theory of Decision- Making and Artiﬁ cial Intelligence 97

v2 R + (1 v2 )S. Note that, absent any cost, full judgment improves the decision maker’s expected payoff . Judgment does not come for free. We assume here that it takes time (although the formulation would naturally match with the notion that it takes costly eff ort). Suppose the discount factor is < 1. A decision maker can spend time in a period determining what the optimal action is for a particular state. If they choose to apply judgment with respect to state i, then there is a probability i that they will determine the optimal action in that period and can make a choice based on that judgment. Otherwise, they can choose to apply judgment to that problem in the next period. It is useful, at this point, to consider what judgment means once it has been applied. The initial assumption we make here is that the knowledge of the payoff function depreciates as soon as a decision is made. In other words, applying judgment can delay a decision (and that is costly) and it can improve that decision (which is its value) but it cannot generate experience that can be applied to other decisions (including future ones). In other words, the initial conception of judgment is the application of thought rather than the gathering of experience.8 Practically, this reduces our examination to a static model. However, in a later section, we consider the experience formulation and demonstrate that most of the insights of the static model carry over to the dynamic model. In summary, the timing of the game is as follows: 1. At the beginning of a decision stage, the decision maker chooses whether to apply judgment and to what state or whether to simply choose an action without judgment. If an action is chosen, uncertainty is resolved and payoff s are realized and we move to a new decision stage. 2. If judgment is chosen, with probability, 1 – i, they do not ﬁ nd out the payoff s for the risky action in that state, a period of time elapses and the game moves back to 1. With probability i, the decision maker gains this knowledge. The decision maker can then take an action, uncertainty is resolved and payoff s are realized, and we move to a new decision stage (back to 1). If no action is taken, a period of time elapses and the current decision stage continues. 3. The decision maker chooses whether to apply judgment to the other state. If an action is chosen, uncertainty is resolved and payoff s are realized and we move to a new decision stage (back to 1). 4. If judgment is chosen, with probability, 1 – –i , they do not ﬁ nd out the payoff s for the risky action in that state, a period of time elapses and the game moves back to 1. With probability –i , the decision maker gains this knowledge. The decision maker then chooses an action, uncertainty

8. The experience frame is considered in Agrawal, Gans, and Goldfarb (2018a). 98 Ajay Agrawal, Joshua Gans, and Avi Goldfarb

Table 3.1 Model parameters

Parameter Description

S Known payoff from the safe action R Potential payoff from the risky action in a given state r Potential payoff from the risky action in a given state θ ∈ i Label of state i {1,2} Probability of state 1 v Prior probability that the payoff in a given state is R λ θ i Probablilty that decision maker learns the payoff to the risky action i if judgment is applied for one period δ Discount factor

is resolved and payoff s are realized, and we move to a new decision stage (back to 1). When prediction is available, it will become available prior to the beginning of a decision stage. The various parameters are listed in table 3.1. Suppose that the decision maker focuses on judging the optimal action (i.e., assessing the payoff ) for i. Then the expected present discount payoff from applying judgment is ()+ + ( + ) + t t ()+ i vR (1 v)S (1 i ) i vR (1 v)S (1 i ) i vR (1 v)S t=2 = i ()vR + (1 v)S . 1 (1 i ) The decision maker eventually can learn what to do and will earn a higher payoff than without judgment, but will trade this off against a delay in the payoff . This calculation presumes that the decision maker knows the state—that i is true—prior to engaging in judgment. If this is not the case, then the expected present discounted payoff to judgment on, say, 1 alone is

1 ()max{v()μR + (1 μ)()vR+ (1 v)r + (1 v)()μr + (1 μ)()vR+ (1 v)r ,S} 1 (1 1) = 1 ()max{}v()μR + (1 μ)(vR + (1 v)r) ,S + (1 v)S , 1 (1 1) where the last step follows from equation (A1). To make exposition simpler, ˆ = ( ) ˆ we suppose that 1 = 2 = . In addition, let /1–(1– ) ; can be given a similar interpretation to , the quality of judgment. If the strategy were to apply judgment on one state only and then make a decision, this would be the relevant payoff to consider. However, because judgment is possible in both states, there are several cases to consider. First, the decision maker might apply judgment to both states in sequence. In this case, the expected present discounted payoff is A Theory of Decision- Making and Artiﬁ cial Intelligence 99

ˆ 2 (v2R + v(1 v)max{}μR + (1 μ)r,S + v(1 v)max{}μr + (1 μ)R,S + (1 v)2S ) = ˆ 2 (v2 R + (1 v2 )S), where the last step follows from equation (A1). Second, the decision maker might apply judgment to, say, 1 ﬁ rst and then, contingent on the outcome there, apply judgment to 2. If the decision maker chooses to pursue judgment on 2 if the outcome for 1 is that the risky action is optimal, the payoff becomes ˆ(v ˆ ()vR + (1 v)max{}μR + (1 μ)r,S + (1 v)max{}μr + (1 μ)()vR + (1 v)r ,S ) = ˆ (v ˆ ()vR + (1 v)S + (1 v)S). If the decision maker chooses to pursue judgment on 2 after determining that the outcome for 1 is that the safe action is optimal, the payoff becomes ˆ(v max {}μR + (1 μ)()vR + (1 v)r ,S + (1 v) ˆ ()vmax {}μr + (1 μ)R,S + (1 v)S ) = ˆ ()v max{}μR + (1 μ)()vR + (1 v)r ,S + (1 v) ˆ S . Note that this is option is dominated by not applying further judgment at all if the outcome for 1 is that the safe action is optimal. Given this we can prove the following: Proposition 1: Under (A1) and (A2), and in the absence of any signal about the state, (a) judging both states and (b) continuing after the discovery that the safe action is preferred in a state are never optimal. Proof: Note that judging two states is optimal if S ˆ > v max{}μr + (1 μ)R,S + (1 v)S

μR + (1 μ)()vR + (1 v)r ˆ > . vR + (1 v)max{}μR + (1 μ)r,S

As (A2) implies that r + (1 – )R ≤ S, the ﬁ rst condition reduces to ˆ > 1. Thus, (a) judging two states is dominated by judging one state and continuing to explore only if the risk is found to be optimal in that state. Turning to the strategy of continuing to apply judgment only if the safe action is found to be preferred in a state, we can compare this to the payoff from applying judgment to one state and then acting immediately. Note that 100 Ajay Agrawal, Joshua Gans, and Avi Goldfarb

ˆ ()v max {μR + (1 μ)(vR + (1 v)r),S} + (1 v) ˆ S

> ˆ ()v max {}μR + (1 μ)()vR + (1 v)r ,S + (1 v)S . This can never hold, proving that (b) is dominated. The intuition is similar to Propositions 1 and 2 in Bolton and Faure- Grimaud (2009). In particular, applying judgment is only useful if it is going to lead to the decision maker switching to the risky action. Thus, it is never worthwhile to unconditionally explore a second state as it may not change the action taken. Similarly, if judging one state leads to knowledge the safe action continues to be optimal in that state, in the presence of uncertainty about the state, even if knowledge is gained of the payoff to the risky action in the second state, that action will never be chosen. Hence, further judgment is not worthwhile. Hence, it is better to choose immediately at that point rather than delay the inevitable. Given this proposition, there are only two strategies that are potentially optimal (in the absence of prediction). One strategy (we will term here J1) is where judgment is applied to one state and if the risky action is optimal, then that action is taken immediately; otherwise, the safe default is taken immediately. The state where judgment is applied ﬁ rst is the state most likely to arise. This will be state 1 if > 1/2. This strategy might be chosen if

ˆ (v max {}μR + (1 μ)()vR + (1 v)r ,S + (1 v)S) > S S ˆ > ˆ , J1 v max {}μR + (1 μ)()vR + (1 v)r ,S + (1 v)S which clearly requires that R + (1 – )(vR + (1 – v)r) > S. The other strategy (we will term here J2) is where judgment is applied to one state and if the risky action is optimal, then judgment is applied to the next state; otherwise, the safe default is taken immediately. Note that J2 is preferred to J1 if

ˆ ()v ˆ ()vR + (1 v)S + (1 v)S > ˆ ()v max {}μR + (1 μ)()vR + (1 v)r ,S + (1 v)S

ˆ vvR()+ (1 v)S > v max {}μR + (1 μ)(vR + (1 v)r),S

max {}μR + (1 μ)()vR + (1 v)r ,S ˆ > . vR + (1 v)S This is intuitive. Basically, it is only when the effi ciency of judgment is suf- ﬁ ciently high that more judgment is applied. However, for this inequality to be relevant, J2 must also be preferred to the status quo yielding a payoff of S. Thus, J2 is not dominated if A Theory of Decision- Making and Artiﬁ cial Intelligence 101

2 2 max {}μR + (1 μ)()vR+(1 v)r ,S S()4v R+S(1+2v 3v ) (1 v)S ˆ > ˆ max , , J2 vR + (1 v)S 2vvR()+ (1 v)S where the ﬁ rst term is the range where J2 dominates J1, while the second term is where J2 dominates S alone; so for J2 to be optimal, it must exceed both. Note also that as →(S – r)/( R – r) (its highest possible level consistent ˆ with [A1] and [A2]), then J 2 → 1. If R + (1 – )(vR + (1 – v)r) > S, note that μR+(1 μ)()vR+(1 v)r S ˆ > ˆ > J2 J1 vR+(1 v)S v(μR+(1 μ)(vR+(1 v)r))+(1 v)S

2 (1 v)S (μR+(1 μ)()vR+(1 v)r S)>vRS( ()μR+(1 μ)()vR+(1 v)r ), which may not hold for v suffi ciently high. However, it can be shown that ˆ ˆ ˆ when J2 + J1, then the two terms of J2 are equal and the second term ˆ ˆ ˆ ˆ exceeds the ﬁ rst when J2 J1. This implies that in the range where J2 < J1, J2 dominates J1. This analysis implies there are two types of regimes with judgment only. ˆ ˆ ˆ If J2 > J1, then easier decisions (with high ) involve using J2, the next tranche of decisions use J1 (with intermediate ˆ) while the remainder ˆ ˆ involves no exercise of judgment at all. On the other hand, if J2 < J1, then the easier decisions involve using J2 while the remainder do not involve judgment at all. 3.4.2 Prediction in the Absence of Judgment Next, we consider the model with prediction but no judgment. Suppose that there exists an AI that can, if deployed, identify the state prior to a decision being made. In other words, prediction, if it occurs, is perfect; an assumption we will relax in a later section. Initially, suppose there is no judgment mechanism to determine what the optimal action is in each state. Recall that, in the absence of prediction or judgment, (A1) ensures that the safe action will be chosen. If the decision maker knows the state, then the risky action in a given state is chosen if vR + (1 – v)r > S. This contradicts (A1). Thus, the expected payoff is

VP = S, which is the same outcome if there is no judgment or prediction. 3.4.3 Prediction and Judgment Together Both prediction and judgment can be valuable on their own. The question we next wish to consider is whether they are complements or substitutes. While perfect prediction allows you to choose an action based on the 102 Ajay Agrawal, Joshua Gans, and Avi Goldfarb actual rather than expected state, it also aff ords the same opportunity with respect to judgment. As judgment is costly, it is useful not to waste con- sidering what action might be taken in a state that does not arise. This was not possible when there was no prediction. But if you receive a prediction regarding the state, you can then apply judgment exclusively to actions in relation to that state. To be sure, that judgment still involves a cost, but at the same time does not lead to any wasted cognitive resources. Given this, if the decision maker were the apply judgment after the state is predicted, their expected discounted payoff would be = {}ˆ ()+ VPJ max vR (1 v)S ,S . This represents the highest expected payoff possible (net of the costs of judgment). A necessary condition for both prediction and judgment to be ˆ ˆ ≡ ˆ ˆ ˆ optimal is that: ≥ PJ s/[ vR + (1 – v)S]. Note that PJ ≤ J1, J2. 3.4.4 Complements or Substitutes? To evaluate whether prediction and judgment are complements or substitutes, we adopt the following parameterization for the eff ectiveness of prediction: we assume that with probability e an AI yields a prediction, while otherwise, the decision must be made in its absence (with judgment only). With this parameterization, we can prove the following: ˆ ˆ Proposition 2: In the range of where < J2, e and are complements, otherwise they are substitutes. ˆ Proof: Step 1. Is J2 > R/ [2(vR + (1 – v)S)]? First, note that max{}μR + (1 μ)()vR + (1 v)r ,S > R vR + (1 v)S 2(vR + (1 v)S) 1 max{}μR + (1 μ)()vR + (1 v)r ,S > R. 2 Note that by (A2) and since > (1/ 2), S > R + (1 – )r > (1/ 2)R so this inequality always holds. Second, note that

2 2 S ()4v R + S(1 + 2v 3v ) (1 v)S R > 2vvR()+ (1 v)S 2()vR + (1 v)S 2 S ()4v2 R + S(1 + 2v 3v2 ) > ()vR + (1 v)S S(S 2R) > v(R2 6RS + S 2 ), which holds as the left-hand side is always positive while the right-hand side is always negative. Step 2: Suppose that R + (1 – )(vR + (1 – v)r) ≤ S; then J1 is never optimal. In this case, the expected payoff is A Theory of Decision- Making and Artiﬁ cial Intelligence 103

+ = ˆ ()+ + ˆ ()ˆ ()+ + eVPJ (1 e)VJ 2 e vR (1 v)S (1 e) v vR (1 v)S (1 v)S . This mixed partial derivative with respect to (e, ˆ) is v(R – 2ˆ(vR + (1 – v)S)). This is positive if R/ [2(vR + (1 – v)S)] ≥ ˆ. By Step 1, this implies ˆ ˆ that for < J2, prediction and judgment are complements; otherwise, they are substitutes. Step 3: Suppose that that R + (1 – )(vR + (1 – v)r) > S. Note that for ˆ ˆ ˆ J1 < J2, J1 is preferred to J2. In this case, the expected payoff to prediction and judgment is

eˆ ()vR + (1 v)S +(1 e) ˆ ()v max{}μR+(1 μ)()vR + (1 v)r ,S + (1 v)S . This mixed partial derivative with respect to (e,ˆ) is v(R – max{R + ˆ ˆ (1 – )(vR + (1 – v)r), S}) > 0. By Step 1, this implies that for < J2, prediction and judgment are complements; otherwise, they are substitutes. ˆ ˆ The intuition is as follows. When < J2, then, in the absence of prediction either no judgment is applied or, alternatively, strategy J1 (with one round of judgment) is optimal; e parameterizes the degree of diff erence between the expected value with both prediction and judgment and the expected value without prediction with an increase in , increasing both. However, with one round of judgment, the increase when judgment is used alone is ˆ ˆ less than that when both are used together. Thus, when < J2, prediction and judgment are complements. ˆ ˆ By contrast, when > J2, then strategy J2 (with two rounds of judgment) is used in the absence of prediction. In this case, increasing increases the expected payoff from judgment alone disproportionately more because judgment is applied on both states, whereas under prediction and judgment it is only applied on one. Thus, improving the quality of judgment reduces ˆ ˆ the returns to prediction. And so, when > J2, prediction and judgment are substitutes.

3.5 Complexity Thus far, the model illustrates the interplay between knowing the reward function (judgment) and prediction. While those results show that prediction and judgment can be substitutes, there is a sense in which they are more naturally complements. The reason is this: what prediction enables is a form of state-contingent decision-making. Without a prediction, a decision maker is forced to make the same choice regardless of the state that might arise. In the spirit of Herbert Simon, one might call this a heuristic. And in the absence of prediction, the role of judgment is to make that choice. More- over, that choice is easier—that is, more likely to be optimal—when there exists dominant (or “near dominant”) choices. Thus, when either the state space or the action space expand (as it may in more complex situations), it is 104 Ajay Agrawal, Joshua Gans, and Avi Goldfarb less likely that there will exist a dominant choice. In that regard, faced with complexity, in the absence of prediction, the value of judgment diminishes and we are more likely to see decision makers choose default actions that, on average, are likely to be better than others. Suppose now we add a prediction machine to the mix. While in our model such a machine, when it renders a prediction, can perfectly signal the state that will arise, let us consider a more convenient alternative that may arise in complex situations: the prediction machine can perfectly signal some states (should they arise), but for other states no precise prediction is possible except for the fact that one of those states is the correct one. In other words, the prediction machine can sometimes render a fi ne prediction and otherwise a coarse one. Here, an improvement in the prediction machine means an increase in the number of states in which the machine can render a fi ne prediction. Thus, consider an N- state model where the probability of state i is i. Suppose that states {1, . . ., m} can be fi nely predicted by an AI, while the remainder cannot be distinguished. Suppose that in the states that cannot be distinguished applying judgment is not worthwhile so that the optimal choice is the safe action. Also, assume that when a prediction is available, judgment is worthwhile; that is, ˆ ≥ s/ [vR + (1 – v)S]. In this situation, the expected present discounted value when both prediction and judgment are available is m N = ˆ μ ( + ) + μ VPJ i vR (1 v)S iS. i=1 i=m+1

Similarly, it is easy to see that VP = VJ = S = V0 as vR + (1 – v)r ≤ S. Note that as m increases (perhaps because the prediction machine learns to predict more states), then the marginal value of better judgment increases. That is, ˆ μ ( + ) μ ˆ m vR (1 v)S mS is increasing in . What happens as the situation becomes more complex (that is, N increases)? An increase in N will weakly lead to a reduction in i for any given i. Holding m fi xed (and so the quality of the prediction machine does not improve with the complexity of the world), this will reduce the value of prediction and judgment as greater weight is placed on states where prediction is unavailable; that is, it is assumed that the increase in complexity does not, ceteris paribus, create a state where prediction is available. Thus, complexity appears to be associated with lower returns to both prediction and judgment. Put diff erently, an improvement in prediction machines would mean m increases with N fi xed. In this case, the returns to judgment rise as greater weight is put on states where prediction is available. This insight is useful because there are several places in the economics literature where complexity has interacted with other economic decisions. These include automation, contracting, and fi rm boundaries. We discuss each of these in turn, highlighting potential implications. A Theory of Decision- Making and Artifi cial Intelligence 105

3.5.1 Automation The literature on automation is sometimes synonymous with AI. This arises because AI may power new robots that are able to operate in open environments thanks to machine learning. For instance, while automated trains have been possible for some time since they run on tracks, automated cars are new because they need to operate in far more complex environments. It is prediction in those open environments that has allowed the emergence of environmentally ﬂ exible capital equipment. Note that leads to the implication that as AI improves, tasks in more complex environments can be handled by machines (Acemoglu and Restrepo 2017). However, this story masks the message that emerges from our analysis that recent AI developments are all about prediction. Why prediction enables automated vehicles is because it is relatively straightforward to describe (and hence, program) what those vehicles should do in diff erent situations. In other words, if prediction enables “state contingent decisions,” then automated vehicles arise because someone knows what decision is optimal in each state. In other words, automation means that judgment can be encoded in machine behavior. Prediction added to that means that automated capital can be moved into more complex environments. In that respect, it is perhaps natural to suggest that improvements in AI will lead to a substitution of humans for machines as more tasks in more complex environments become capable of being programmed in a state-contingent manner. That said, there is another dimension of substitution that arises in complex environments. As noted above, when states cannot be predicted (something that for a given technology is more likely to be the case in more complex environments), then the actions chosen are more likely to be defaults or the results of heuristics that perform, on average, well. Many, including Acemo- glu and Restrepo (2017), argue that it is for more complex tasks that humans have a comparative advantage relative to machines. However, this is not at all obvious. If it is known that a particular default or heuristic should be used, then a machine can be programmed to undertake this. In this regard, the most complex tasks—precisely because little is known regarding how to take better actions given that the prediction of the state is coarse—may be more, not less, amenable to automation. If we had to speculate, imagine that states were ordered in terms of diminished likelihood (i.e., i ≥ j for all i < j). The lowest index states might be ones that, because they arrive frequently, there is knowledge of what the optimal action is in each and so they can be programmed to be handled by a machine. The highest index states similarly, because the optimal action that cannot be determined can also be programmed. It is the intermediate states that arise less frequently but not infrequently where, if a reliable prediction existed, could be handled by humans applying judgment when those states arose. Thus, the payoff could be written 106 Ajay Agrawal, Joshua Gans, and Avi Goldfarb

k m N = μ ( + ) + ˆ μ ()+ + μ VPJ i vR (1 v)S i vR (1 v)S iS, i=1 i=k+1 i=m+1 where tasks 1 through k are automated using prediction because there is knowledge of the optimal action. If this was the matching of tasks to machines and humans, then it is not at all clear whether an increase in complexity would be associated with more or less human employment. That said, the issue for the automation literature is not subtleties over the term “complex tasks,” but as AI becomes more prevalent, where might the substitution of machines for humans arise. As noted above, an increase in AI increases m. At this margin, humans are able to come into the marginal tasks and, because a prediction machine is available, use judgment to conduct state- contingent decisions in those situations. Absent other eff ects, therefore, an increase in AI is associated with more human labor on any given task. However, as the weight on those marginal tasks is falling in the level of complexity, it may not be the more complex tasks that humans are performing more of. On the other hand, one can imagine that in a model with a full labor market equilibrium that an increase in AI that enables more human judgment at the margin may also create opportunities to study that judgment to see if it can be programmed into lower index states and be handled by machines. So, while the AI does not necessarily cause more routine tasks to be handled by machines, it might create the economic con- ditions that lead to just that. 3.5.2 Contracting Contracting shares much with programming. Here is Jean Tirole (2009, 265) on the subject: Its general thrust goes as follows. The parties to a contract (buyer, seller) initially avail themselves of an available design, perhaps an industry standard. This design or contract is the best contract under existing knowledge. The parties are unaware, however, of the contract’s implications, but they realize that something may go wrong with this contract; indeed, they may exert cognitive eff ort in order to fi nd out about what may go wrong and how to draft the contract accordingly: put diff erently, a contingency is foreseeable (perhaps at a prohibitively high cost), but not necessarily foreseen. To take a trivial example, the possibility that the price of oil increases, implying that the contract should be indexed on it, is perfectly foreseeable, but this does not imply that parties will think about this possibility and index the contract price accordingly. Tirole argues that contingencies can be planned for in contracts using cognitive eff ort (akin to what we have termed here as judgment), while others may be optimally left out because the eff ort is too costly relative to the return given, say, the low likelihood that contingency arises. This logic can assist us in understanding what prediction machines might A Theory of Decision- Making and Artifi cial Intelligence 107 do to contracts. If an AI becomes available then, in writing contracts, it is possible, because fi ne state predictions are possible, to incur cognitive costs to determine what the contingencies should be if those states should arise. For other states, the contract will be left incomplete—perhaps for a default action or alternatively some renegotiation process. A direct implication of this is that contracts may well become less incomplete. Of course, when it comes to employment contracts, the eff ects may be diff erent. As Herbert Simon (1951) noted, employment contracts diff er from other contracts precisely because it is often not possible to specify what actions should be performed in what circumstance. Hence, what those contracts often allocate are diff erent decision rights. What is of interest here is the notion that contacts can be specifi ed clearly—that is, programmed—but also that prediction can activate the use of human judgment. That latter notion means that actions cannot be easily contracted—by defi nition, contractibility is programming and need- ing judgment implies that programming was not possible. Thus, as prediction machines improve and more human judgment is optimal, then that judgment will be applied outside of objective contract measures—including objective performance measures. If we had to speculate, this would favor more subjective performance processes, including relational contracts (Baker, Gibbons, and Murphy 1999).9 3.5.3 Firm Boundaries We now turn to consider what impact AI may have on fi rm boundaries (that is, the make or buy decision). Suppose that it is a buyer (B) who receives the value from a decision taken—that is, the payoff from the risky or safe action as the case may be. To make things simple, let’s assume that i = for all i, so that V = kvR()+ (1 v)S + ˆ(m k)(vR + (1 v)S) + (N m)S. We suppose that the tasks are undertaken by a seller (S). The tasks {1, . . . , k} and {m + 1, . . . , N) can be contracted upon, while the intermediate tasks require the seller to exercise judgment. We suppose that the cost of providing judgment is a function c(ˆ), which is nondecreasing and convex. (We write this function in terms of ˆ just to keep the notation simple.) The costs can be anticipated by the buyer. So if one of the intermediate states arises, the buyer can choose to give the seller a fi xed price contract (and bear none of the costs) or a cost-plus contract (and bear all of them). Following Tadelis (2002), we assume that the seller market is competitive and so all surplus accrues to the buyer. In this case, the buyer return is

9. A recent paper by Dogan and Yildirim (2017) actually considers how automation might impact on worker contracts. However, they do not examine AI per se, and focus on how it might change objective performance measures in teams moving from joint performance evaluation to more relative performance evaluation. 108 Ajay Agrawal, Joshua Gans, and Avi Goldfarb

kvR( + (1 v)S)+ max {}ˆ(m k)(vR + (1 v)S),S + (N m)S p zc()ˆ , while the seller return is: p – (1 – z)c(ˆ). Here p + zc(ˆ) is the contract price and z is 0 for a fi xed price contract and 1 for a cost- plus contract. Note that only with a cost- plus contract does the seller exercise any judgment. Thus, the buyer chooses a cost-plus over a fi xed price contract if kvR()+ (1 v)S + max {}ˆ(m k)(vR + (1 v)S),S + (N m)S c()ˆ > kvR()+ (1 v)S + (N k)S. It is easy to see that as m rises (i.e., prediction becomes cheaper), a cost- plus contract is more likely to be chosen. That is, incentives fall as prediction becomes more abundant. Now we can consider the impact of integration. We assume that the buyer can choose to make the decisions themselves, but at a higher cost. That is, c(ˆ,I) > c(ˆ) where I denotes integration. We also assume that ∂c(ˆ,I)/ ∂ˆ > ()c( ˆ )/ ˆ . Under integration, the buyer’s value is kvR( + (1 v)S) + ˆ *(m k)()vR + (1 v)S + (N m)S c( ˆ *, I) where ˆ * maximizes the buyer payoff in this case. Given this, it can easily be seen that as m increases, the returns to integration rise. By contrast, notice that as k increases, the incentives for a cost-plus contract are diminished and the returns to integration fall. Thus, the more prediction machines allow for the placement of contingencies in a contract (the larger m- k), the higher powered will seller incentives be and the more likely there is to be integration. Forbes and Lederman (2009) showed that airlines are more likely to vertically integrate with regional partners when scheduling is more complex: specifi cally, where bad weather is more likely to lead to delays. The impact of prediction machines will depend on whether they lead to an increase in the number of states where the action can be automated in a state- contingent manner (k) relative to the increase in the number of states where the state becomes known but the action cannot be automated (m). If the former, then we will see more vertical integration with the rise of prediction machines. If the latter, we will see less. The diff erence is driven by the need for more costly judgment in the vertically integrated case as m- k rises.

3.6 Conclusions In this chapter, we explore the consequences of recent improvements in machine- learning technology that have advanced the broader fi eld of artifi - cial intelligence. In particular, we argue that these advances in the ability of machines to conduct mental tasks are driven by improvements in machine prediction. In order to understand how improvements in machine prediction will impact decision- making, it is important to analyze how the payoff s of the model arise. We label the process of learning payoff s “judgment.” A Theory of Decision- Making and Artifi cial Intelligence 109

By modeling judgment explicitly, we derive a number of useful insights into the value of prediction. We show that prediction and judgment are generally complements, as long as judgment is not too diffi cult. We also show that improvements in judgment change the type of prediction quality that is most useful: better judgment means that more accurate predictions are valuable relative to more frequent predictions. Finally, we explore the role of complexity, demonstrating that, in the presence of complexity, the impact of improved prediction on the value of judgment depends on whether improved prediction leads to automated decision-making. Complexity is a key aspect of economic research in automation, contracting, and the boundaries of the ﬁ rm. As prediction machines improve, our model suggests that the consequences in complex environments are particularly fruitful to study. There are numerous directions research in this area could proceed. First, the chapter does not explicitly model the form of the prediction—including what measures might be the basis for decision-making. In reality, this is an important design variable and impacts on the accuracy of predictions and decision-making. In computer science, this is referred to as the choice of surrogates, and this appears to be a topic amenable for economic theoretical investigation. Second, the chapter treats judgment as largely a human-directed activity. However, we have noted that it can else be encoded, but have not been explicit about the process by which this occurs. Endogenis- ing this—perhaps relating it to the accumulation of experience—would be an avenue for further investigation. Finally, this is a single-agent model. It would be interesting to explore how judgment and prediction mix when each is impacted upon by the actions and decisions of other agents in a game theoretic setting.

References

Acemoglu, Daron. 2003. “Labor- and Capital- Augmenting Technical Change.” Journal of the European Economic Association 1 (1): 1–37. Acemoglu, Daron, and Pascual Restrepo. 2017. “The Race between Machine and Man: Implications of Technology for Growth, Factor Shares, and Employment.” Working paper, Massachusetts Institute of Technology. Agrawal, Ajay, Joshua S. Gans, and Avi Goldfarb. 2018a. “Human Judgment and AI Pricing.” American Economic Association: Papers & Proceedings, 108:58–63. ———. 2018b. Prediction Machines: The Simple Economics of Artiﬁ cial Intelligence. Boston, MA: Harvard Business Review Press. Alpaydin, Ethem. 2010. Introduction to Machine Learning, 2nd ed. Cambridge, MA: MIT Press. Autor, David. 2015. “Why Are There Still So Many Jobs? The History and Future of Workplace Automation.” Journal of Economic Perspectives 29 (3): 3–30. Baker, George, Robert Gibbons, and Kevin Murphy. 1999. “Informal Authority in Organizations.” Journal of Law, Economics, and Organization 15:56–73. Belloni, Alexandre, Victor Chernozhukov, and Christian Hansen. 2014. “High- 110 Andrea Prat

Dimensional Methods and Inference on Structural and Treatment Eff ects.” Jour- nal of Economic Perspectives 28 (2): 29–50. Benzell, Seth G., Laurence J. Kotlikoff , Guillermo LaGarda, and Jeff rey D. Sachs. 2015. “Robots Are Us: Some Economics of Human Replacement.” NBER Work- ing Paper no. 20941, Cambridge, MA. Bolton, P., and A. Faure-Grimaud. 2009. “Thinking Ahead: The Decision Problem.” Review of Economic Studies 76:1205–38. Brynjolfsson, Erik, and Andrew McAfee. 2014. The Second Machine Age. New York: W. W. Norton. Dogan, M., and P. Yildirim. 2017. “Man vs. Machine: When Is Automation Inferior to Human Labor?” Unpublished manuscript, The Wharton School of the Uni- versity of Pennsylvania. Domingos, Pedro. 2015. The Master Algorithm. New York: Basic Books. Forbes, Silke, and Mara Lederman. 2009. “Adaptation and Vertical Integration in the Airline Industry.” American Economic Review 99 (5): 1831–49. Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. New York: Springer. Hawkins, Jeff . 2004. On Intelligence. New York: Times Books. Jha, S., and E. J. Topol. 2016. “Adapting to Artifi cial Intelligence: Radiologists and Pathologists as Information Specialists.” Journal of the American Medical Associa- tion 316 (22): 2353–54. Lusted, L. B. 1960. “Logical Analysis in Roentgen Diagnosis.” Radiology 74:178– 93. Markov, John. 2015. Machines of Loving Grace. New York: HarperCollins Pub- lishers. Ng, Andrew. 2016. “What Artifi cial Intelligence Can and Can’t Do Right Now.” Harvard Business Review Online. Accessed Dec. 8, 2016. https://hbr .org/ 2016/ 11 / what- artifi cial- intelligence- can- and- cant- do- right- now. Simon, H. A. 1951. “A Formal Theory of the Employment Relationship.” Econo- metrica 19 (3): 293–305. Tadelis, S. 2002. “Complexity, Flexibility and the Make- or- Buy Decision.” American Economic Review 92 (2): 433–37. Tirole, J. 2009. “Cognition and Incomplete Contracts.” American Economic Review 99 (1): 265–94. Varian, Hal R. 2014. “Big Data: New Tricks for Econometrics.” Journal of Economic Perspectives 28 (2): 3–28.

Comment Andrea Prat

One of the key activities of organizations is to collect, process, combine, and utilize information (Arrow 1974). A modern corporation exploits the vast amounts of data that it accumulates from marketing, operations, human resources, ﬁ nance, and other functions to grow faster and be more

Andrea Prat is the Richard Paul Richman Professor of Business at Columbia Business School and professor of economics at Columbia University. For acknowledgments, sources of research support, and disclosure of the author’s material fi nancial relationships, if any, please see http://www .nber .org/ chapters/ c14022.ack. Comment 111 productive. This exploitation process depends on the kind of information technology (IT) that is available to the fi rm. If IT undergoes a revolution, we should expect deep structural changes in the way fi rms are organized (Milgrom and Roberts 1990). Agrawal, Gans, and Goldfarb explore the eff ects that an IT revolution centered on artifi cial intelligence could have on organizations. Their analysis highlights an insightful distinction between prediction, the process of forecasting a state of the world given observable information, and judgment, the assessment of the eff ects of the state of the world and the possible action x the organization can take in response to it, namely, the value of the payoff function u(,x). This is an important point of departure from existing work. Almost all economists—as well as computer scientists and decision scientists—assume that the payoff function u(,x) is known: the decision maker is presumed to have a good sense of how actions and states combine to create outcomes. This assumption, however, is highly unrealistic. The credit card fraud example supplied by the authors is convincing. What is the long-term cost to a bank of approving a fraudulent transaction or labeling a legitimate transaction a suspected fraud? Organizations can spend resources to improve both their prediction preci- sion and their judgment quality. Agrawal, Gans, and Goldfarb characterize the solution to this optimization problem. Their main result is that, under reasonable assumption, investment in prediction and investment in judgment are complementary (Proposition 2). Investing in prediction makes investment in judgment more benefi cial in expected value. This complementarity suggests that moving from a situation where prediction is prohibitively expensive to one where it is economical should increase the returns to judgment. In this perspective, the AI revolution will lead to an increase in the demand for judgment. However, judgment is an intrinsically diff erent problem—one that cannot be solved through the analysis of big data. Let me suggest an example. Admissions offi ces of many universities are turning to AI to choose which applicants to make off ers to. Algorithms can be trained on past admissions data. We observe the characteristics of applicants and the grades of past and present students. Leaving aside the censored observations problem arising from the fact that we only see the grades of successful applicants who decide to enroll, we can hope that AI can provide a fairly accurate prediction of an applicant’s future grades given his or her observable characteristics. The obvious problem is that we do not know how admitting someone who is likely to get high grades is going to aff ect the long- term payoff of our university. The latter is a highly complex object that depends on whether our alums become the kind of inspiring, successful, and ethical people that will add to the academic reputation and fi nancial sustainability of our university. There is likely to be a connection 112 Andrea Prat between grades and this long- term goal, but we are not sure what it is. In this setting, Agrawal, Gans, and Goldfarb teach us an important lesson. Progress in AI should induce our university leaders to ask deeper questions about the relationship between student quality and the long- term goals of our higher-learning institutions. These questions cannot be answered within AI, but rather with more theory- driven retrospective approaches or perhaps more qualitative methodologies. As an organizational economist, I am particularly interested in the implications of Agrawal, Gans, and Goldfarb’s model for the study of organizations. First, this chapter highlights the importance of the dynamics of decision-making—a seriously underresearched topic. In a complex world, organizations are not going to immediately collect all the information they could possibly need about all possible contingencies they may face. Bolton and Faure- Grimaud (2009), a source of inspiration for Agrawal, Gans, and Goldfarb, model a decision maker who can “think ahead” about future states of the world in yet unrealized states of nature. They show that the typical decision maker does not want to think through a complete action plan, but rather focus on key short- and medium- term decisions. Agrawal, Gans, and Goldfarb show that Bolton and Faure-Grimaud’s ideas are highly relevant for understanding how organizations are likely to respond to changes in information technology. Second, Agrawal, Gans, and Goldfarb also speak to the organizational economics literature on mission. Dewatripont, Jewitt, and Tirole (1999) develop a model where organizational leaders are agents whose type is unknown, as in Holmstrom’s (1999) career concerns paradigm. Each agent is assigned a mission, a set of measured variables that are used to evaluate and reward the agent. Dewatripont, Jewitt, and Tirole identify a tension between selecting a simple one-dimensional mission that will provide the agent with a strong incentive to perform well or a “fuzzy” multidimensional mission that will dampen the agent’s incentive to work hard but will more closely mirror the true objective of the organization. This tension is also present in Agrawal, Gans, and Goldfarb’s world. Should we give the organization a mission that is close to a pure prediction problem, like admitting students who will get high grades? The pro is that it will be relatively easy to assess the leader’s performance. The con is that the outcome may be weakly related to the organization’s ultimate objective. Or should we give the organization a mission that also comprises the judgment problem, like furthering the long-term academic reputation of our university? This mission would be more representative of the organization’s ultimate objective, but may make it hard to assess our leaders and give them a weak incentive to adopt new prediction technologies. One possible lesson from Agrawal, Gans, and Goldfarb is that, as the cost of adopting AI goes down, the moral hazard problem connected with judgment becomes rela- Comment 113 tively more important, thus militating in favor of incentive schemes that reward judgment rather than prediction. Third, Agrawal, Gans, and Goldfarb’s section on reliability touches on an important topic. Is it better to have a technology that returns accurate predictions with a low probability or less accurate predictions with a higher probability? The answer to this question depends on the available judgment technology. Better judgment technology increases the marginal benefi t of prediction accuracy rather than prediction frequency. More broadly, this type of analysis can guide the design of AI algorithms. Given the mapping between states, actions, and outcomes, and given the cost of various prediction technologies, what prediction technology should the organization select? A general analysis of this question may require using information theoretical concepts, introduced to economics by Sims (2003). Fourth, Agrawal, Gans, and Goldfarb show that economic theory can make important contributions to the debate over how AI will aff ect optimal organization. There is a related area where the interaction between economists and computer scientists can be benefi cial. Artifi cial intelligence typi- cally assumes a stable fl ow of instances. When a bank develops an AI-based system to detect fraud, it assumes that the available data, which is used to build and test the detection algorithm, comes from the same data-generating process as future data on which the algorithm will be applied. However, the underlying data-generating process is not an exogenously given natural phenomenon: it is the output of a set of human beings who are pursuing their own goals, like maximizing the chance of getting their nonfraudulent application accepted or maximizing their chance of defrauding the bank. These sentient creatures will in the long term respond to the fraud-detection algorithm by modifying their application strategy, for instance, by providing diff erent information or by exerting eff ort to modify the reported variables. This means that the data-generating process will be subject to a structural change and that this change will be endogenous to the fraud-detection algorithm chosen by the bank. A similar phenomenon occurs in the university admission example discussed above: a whole consulting industry is devoted to understanding admissions criteria and advising applicants on how to maximize their success chances. A change in admissions practices is likely to be refl ected in the choices that high school students make. If the data- generating process is endogenous and depends on the prediction technology adopted by the organization, the judgment problem identifi ed by Agrawal, Gans, and Goldfarb becomes even more complex. The organization must evaluate how other agents will respond to changes in the prediction technology. As, by defi nition, no data is available about not yet realized data-generating processes, the only way to approach this problem is by estimating a structural model that allows other agents to respond to changes in our prediction technology. 114 Andrea Prat

In conclusion, Agrawal, Gans, and Goldfarb make a convincing case that the AI revolution should increase the beneﬁ t of improving our judgment ability. They also provide us with a tractable yet powerful framework to understand the interaction between prediction and judgment. Future research should focus on further understanding the implications of improvements in prediction technology on the optimal structure of organizations.

References Arrow, Kenneth. J. 1974. The Limits of Organization. New York: W. W. Norton. Bolton, P., and A. Faure-Grimaud. 2009. “Thinking Ahead: The Decision Problem.” Review of Economic Studies 76: 1205–38. Dewatripont, Mathias, Ian Jewitt, and Jean Tirole. 1999. “The Economics of Career Concerns, Part II: Application to Missions and Accountability of Government Agencies.” Review of Economic Studies 66 (1): 199–21. Holmstrom, Bengt. 1999. “Managerial Incentive Problems: A Dynamic Perspective.” Review of Economic Studies 66 (1): 169–82. Milgrom, Paul, and John Roberts. 1990. “The Economics of Modern Manufac- turing: Technology, Strategy, and Organization.” American Economic Review June: 511–28. Sims, Christopher. 2003. “Implications of Rational Inattention.” Journal of Mone- tary Economics 50 (3): 665–90. Contributors

Daron Acemoglu Erik Brynjolfsson Department of Economics MIT Sloan School of Management Massachusetts Institute of 100 Main Street, E62-414 Technology Cambridge, MA 02142 50 Memorial Drive Cambridge, MA 02142-1347 Colin F. Camerer Department of Economics Philippe Aghion California Institute of Technology Collège de France 1200 East California Boulevard 3 Rue D’Ulm Pasadena, CA 91125 75005 Paris, France Judith Chevalier Ajay Agrawal Yale School of Management Rotman School of Management 135 Prospect Street University of Toronto New Haven, CT 06520 105 St. George Street Toronto, ON M5S 3E6, Canada Iain M. Cockburn School of Management Susan Athey Boston University Graduate School of Business 595 Commonwealth Avenue Stanford University Boston, MA 02215 655 Knight Way Stanford, CA 94305 Tyler Cowen Department of Economics James Bessen George Mason University Technology & Policy Research 4400 University Drive Initiative Fairfax, VA 22030 Boston University School of Law 765 Commonwealth Avenue Patrick Francois Boston, MA 02215 Vancouver School of Economics University of British Columbia IONA Building, 6000 Iona Drive Vancouver, BC V6T 2E8, Canada

611 612 Contributors

Jason Furman Daniel Kahneman Harvard Kennedy School Woodrow Wilson School 79 John F. Kennedy Street Princeton University Cambridge, MA 02138 Princeton, NJ 08544-1013

Alberto Galasso Anton Korinek Rotman School of Management University of Virginia University of Toronto Monroe Hall 246 105 St. George Street 248 McCormick Road Toronto, ON M5S 3E6, Canada Charlottesville, VA 22904

Joshua Gans Mara Lederman Rotman School of Management Rotman School of Management University of Toronto University of Toronto 105 St. George Street 105 St. George Street Toronto, ON M5S 3E6, Canada Toronto, Ontario M5S 3E6, Canada

Avi Goldfarb Hong Luo Rotman School of Management Harvard Business School University of Toronto Morgan Hall 241 105 St. George Street Soldiers Field Road Toronto, ON M5S 3E6, Canada Boston, MA 02163

Austan Goolsbee John McHale University of Chicago Booth School 108 Cairnes Building of Business School of Business and Economics 5807 S. Woodlawn Avenue National University of Ireland Chicago, IL 60637 Galway H91 TK33, Ireland

Rebecca Henderson Paul R. Milgrom Harvard Business School Department of Economics Morgan Hall 445 Stanford University Soldiers Field Road 579 Serra Mall Boston, MA 02163 Stanford, CA 94305-6072

Ginger Zhe Jin Matthew Mitchell Department of Economics Rotman School of Management University of Maryland University of Toronto 3115F Tydings Hall 105 St. George Street College Park, MD 20742-7211 Toronto, ON M5S 3E6, Canada

Benjamin F. Jones Alexander Oettl Department of Management and Scheller College of Business Strategy Georgia Institute of Technology Kellogg School of Management 800 West Peachtree Street NW Northwestern University Atlanta, GA 30308 2211 Campus Drive Evanston, IL 60208 Andrea Prat Columbia Business School Charles I. Jones Uris Hall 624 Graduate School of Business 3022 Broadway Stanford University New York, NY 10027-6902 655 Knight Way Stanford, CA 94305-4800 Contributors 613

Manav Raj Chad Syverson Stern School of Business University of Chicago Booth School of New York University Business 44 West Fourth Street 5807 S. Woodlawn Avenue New York, NY 10012 Chicago, IL 60637

Pascual Restrepo Matt Taddy Department of Economics University of Chicago Booth School of Boston University Business 270 Bay State Road 5807 S. Woodlawn Avenue Boston, MA 02215 Chicago, IL 60637

Daniel Rock Steven Tadelis MIT Sloan School of Management Haas School of Business 100 Main Street, E62-365 University of California, Berkeley Cambridge, MA 02142 545 Student Services Building Berkeley, CA 94720 Jeff rey D. Sachs Center for Sustainable Development, Manuel Trajtenberg Earth Institute Eitan Berglas School of Economics Columbia University Tel Aviv University 535 West 116th Street, MC 4327 Tel Aviv 69978, Israel New York, NY 10027 Daniel Treﬂ er Robert Seamans Rotman School of Management Stern School of Business University of Toronto New York University 105 St. George Street 44 West 4th Street, KMC 7-58 Toronto, ON M5S 3E6, Canada New York, NY 10012 Catherine Tucker Scott Stern MIT Sloan School of Management MIT Sloan School of Management 100 Main Street, E62-536 100 Main Street, E62-476 Cambridge, MA 02142 Cambridge, MA 02142 Hal Varian Betsey Stevenson School of Information Gerald R. Ford School of Public Policy University of California, Berkeley University of Michigan 102 South Hall 5224 Weill Hall Berkeley, CA 94720-4600 735 South State Street Ann Arbor, MI 48109-3091

Joseph E. Stiglitz Columbia University Uris Hall 212 3022 Broadway New York, NY 10027

Author Index

Abadie, A., 531 Angermueller, C., 168 Ablon, L., 448 Aral, S., 43 Abrahamson, Z., 420 Arntz, M., 321 Abramovitz, M., 32 Aronoff , M., 497 Acemoglu, D., 23n1, 43, 89, 90, 105, 127, Arrow, K., 9, 43, 110, 118, 148, 364n11, 366, 141, 152, 197, 198, 201, 202, 203, 412 203n4, 204, 204n5, 205, 206, 208, 210, Arthur, B. W., 150 211, 212n7, 219, 220, 220n11, 223, 224, Asher, S., 525 225n14, 238, 240, 243, 271, 283, 293n3, Athey, S., 68, 425, 448, 449, 510, 514, 515, 376n27, 376n28, 554 516, 517, 519, 523, 524, 525, 527, 528, Acquisti, A., 410, 416, 424, 440n2, 444, 448, 529, 530, 531, 532, 533, 534, 536, 538 451, 457, 457n42, 459, 483n19 Atkeson, A., 42n19 Adee, S., 426 Autor, D. H., 7, 23n1, 30, 89, 198, 202n3, Agarwal, A., 82n12, 83 203, 208, 220n11, 238, 239n3, 240, 271, Aghion, P., 122, 172, 262, 262n19, 263, 265, 309, 322, 475, 555 267, 268, 373n23, 465, 477, 479, 495 Axelrod, R., 413 Agrawal, A., 5, 39, 90, 97n8, 150, 161, 167, Ayres, R., 201 241, 425, 464, 501 Azoulay, P., 475, 477 Aguiar, M., 386n36 Airoldi, E., 80 Babcock, L., 592n5 Akerlof, G., 378 Bai, J., 532 Akerman, A., 7, 303 Baker, D., 366n14 Alexopoulos, M., 555 Baker, G., 107 Allen, R. C., 204, 209 Banbura, M., 537 Alloway, T., 29 Barkai, S., 271 Alon, T., 43 Barrat, J., 350 Alpaydin, E., 92 Baslandze, S., 263, 264 Altman, S., 325 Bastani, H., 529 Alvarez-Cuadrado, F., 241n4 Baumol, W., 38n14, 238 Anderson, R., 406, 416 Bayati, M., 529, 531, 532, 536 Andrade, E. B., 592n3 Belloni, A., 93, 522 Andrews, D., 30 Bengio, S., 401

615 616 Author Index

Bengio, Y., 71, 74, 75, 75n6, 149, 168, 168n6, Cavusoglu, H., 450 172 Cette, G., 27 Benzell, S. G., 90, 336 Chandler, A. D., 206 Berg, A., 373n24 Chapelle, O., 529 Beron, K. J., 483 Chapman, J. P., 599 Bertocchi, G., 430 Chapman, L. J., 599 Bessen, J. E., 23n1, 203, 300, 413, 555 Chernozkhukov, V., 93, 522, 523, 527 Bhatt, M. A., 592n4 Chetty, R., 270 Bickel, P. J., 522 Chevalier, J., 577n5 Binmore, K., 589, 589n1, 593 Chiou, L., 428, 448 Bjorkegren, D., 516, 519 Chong, J.-K., 593, 595 Blake, T., 582 Christie, W. G., 414 Blattberg, R. C., 598 Chui, M., 331 Blei, D. M., 507, 510, 515, 532, 533, 534 Clark, C., 295 Bloom, N., 27, 150, 223, 259n15, 268, 560 Cockburn, I., 150 Bolton, P., 92, 95, 95n5, 96n7, 100, 112 Coey, D., 514 Boppart, T., 241n4, 295 Cohen, J., 555 Borenstein, S., 413 Cohen, L., 501 Bornmann, L., 153 Comin, D., 241n4, 295 Bostrom, N., 3, 286, 381, 382, 382n31 Corrigan, B., 597 Bottou, L., 79 Costa-Gomes, M. A., 593, 593n9, 594n9 Bousquet, O., 79 Courville, A., 71, 75n6 Bowen, W., 38n14 Cowen, T., 27, 150 Brandeis, L. D., 431, 459 Cowgill, B., 562 Brander, J. A., 473 Cranor, L. F., 424, 458 Brandimarte, L., 448 Crawford, V. P., 593, 593n9, 594n9 Brandt, L., 471 Criscuolo, C., 30 Bresnahan, T. F., 4, 39, 42, 116, 119, 120, 169, 176n2, 310 Dana, J., 599 Bridgman, B., 38 Danaylov, N., 253 Brooks, R., 124 Dasgupta, P., 366n15 Broseta, B., 593n9, 594n9 Datta, A., 433 Brown, J., 314 Dauth, W., 556 Brunskill, E., 528 David, P. A., 4, 41, 42n19, 119 Brynjolfsson, E., 23, 23n1, 28n7, 30, 39, 40, Davies, R. B., 483 42, 43, 47, 50, 89, 119, 120, 150, 197, Dávila, E., 354 201, 204, 309, 555, 557, 560, 563 Dawes, R. M., 597, 598, 599, 599n15 Brzeski, C., 556 Dawsey, K., 29 Buera, F. J., 293, 293n2 Deaton, A., 26, 85 Buffi e, E. F., 373n24 Della Vigna, S., 411 Bughin, J., 408 Delli Gatti, D., 361, 362n9, 380 Burk, I., 556 De Loecker, J., 30 Busch, M. L., 474, 475n13 Dennis, B. N., 293n2 Byers, J. W., 577n5 Dewatripoint, M., 112 Byrne, D. M., 29, 319 Diamond, A., 531 Dietvorst, B. J., 426 Camerer, C. F., 589, 589n1, 590, 593, Dimakopoulou, M., 529 594n10, 595, 598 Dimico, A., 430 Campbell, K., 450 Dobson, W., 485n23 Cardarelli, R., 29 Dogan, M., 107n9 Carley, M., 130, 132 Doi, E., 599 Case, A., 26 Domingos, P., 93 Catalini, C., 150, 425, 448, 449, 454 Doms, M., 562 Author Index 617

Dorantes, C., 450 Fortunato, M., 25 Dorn, D., 208 Fredriksson, P. G., 483 Dosi, G., 366n14 Frey, C. B., 291, 321, 331, 350n1, 553, 555, Doudchenko, N., 531 556 Dover, Y., 577n5 Friedman, J., 93 Drandakis, E., 376n27 Frosst, N., 76n7 Dubé, J.-P., 410 Fudenberg, D., 413, 594 Dudik, M., 528 Fujii, H., 465 Duﬂ o, E., 539 Fung, A., 459 Dunne, T., 562 Furman, J. L., 30, 122, 140, 150, 553 Dupuit, J., 296, 298 Duranton, G., 480, 567 Gaarder, I., 7, 303 Dwork, C., 453 Gabaix, X., 604 Dzamba, M., 116 Gaggl, P., 303 Gal, M. S., 583 Eckles, D., 538 Gal, P., 30 Edwards, D. D., 598 Galasso, A., 495, 496, 497, 498, 501 Edwards, J. S., 598 Gans, J., 5, 39, 90, 97n8, 171, 425, 454, 464, Edwards, L., 294 550 Edwards, W., 595 Garicano, L., 267 Eeckhout, J., 30 Geanakoplos, J., 354, 368n19 Egami, N., 510 Geank, 369 Einhorn, H. J., 598, 599 Gehring, J., 25 Elkin-Koren, N., 583 Gentzskow, M., 411 Elsby, M. W. L., 270, 329 Gibbons, R., 107 Engel, E., 295 Glaeser, E. L., 516, 537, 548 Engerman, S. L., 430 Goel, S., 516 Engstrom, R., 537 Goeree, J., 593 Erlingsson, U., 453 Goh, G., 151n1, 168 Ethier, W. J., 476 Goldberg, L. R., 597, 601n19 Etzioni, O., 410 Golden, J. M., 577n5 Evenson, R. E., 140 Goldenshluger, A., 529 Ezrachi, A., 414 Goldfarb, A., 5, 39, 90, 97n8, 148, 150, 161, 167, 425, 427, 448, 464, 482, 483, Fajgelbaum, P., 480 483n19, 550, 593n8 Farrell, J., 424 Goldin, C., 209, 322 Faure-Grimaud, A., 92, 95, 95n5, 96n7, Goldman, M., 536 100, 112 Gomez-Uribe, C., 603 Faust, D., 597 Good, I. J., 238, 253 Fehr, E., 358n8 Goodfellow, I., 66, 71, 75n6, 401 Feldman, M. P., 560 Goolsbee, A. D., 310, 312, 314 Feldstein, M., 29 Gopalan, P., 510 Feraud, R., 529 Gordon, R. D., 305, 310, 349 Fernald, J. G., 27, 29, 32, 319 Gordon, R. J., 27, 150, 175, 210, 223, Feurer, M., 70 259n15, 264 Filippas, A., 577n5 Graepel, T., 429 Finklestein, A., 503 Graetz, G., 201, 274, 319, 554 Fleming, L., 161, 479 Graff Zivin, J. S., 475, 477 Florêncio, D., 452 Graham, M., 459 Foellmi, R., 295 Green, J., 502 Forbes, S., 108 Greenstein, S., 119 Ford, M., 201, 204, 301 Greenwald, B., 354, 364n11, 366n15, Forsythe, R., 590n2 368n19, 369, 370, 412 618 Author Index

Gregory, T., 321 Hobijn, B., 270, 329 Griliches, Z., 116, 120, 170 Hoch, S. J., 598 Grissen, D., 516, 519 Hochreiter, S., 73, 76 Groover, M., 201, 205 Hodas, N., 151n1, 168 Groshen, E. L., 350 Hoff man, D., 444, 457 Gross, R., 451 Hoff man, M., 447n17 Grossman, G. M., 464, 476, 477, 478, 479, Hofman, J. M., 510 480, 482 Hogarth, R. M., 598 Grove, W. M., 597 Holmes, T. J., 43 Guerrieri, V., 293n3 Holmstrom, B., 112, 269 Gurun, U., 501 Holt, C., 593 Gutiérrez, G., 30 Hoos, H., 575 Guvenen, F., 29 Hornik, K., 74, 525 Hortaçsu, A., 43, 593n8 Ha, Y., 599 Horton, J. J., 577n5 Hahn, J., 522 Hotborn, T., 525 Haile, P. A., 515 Howitt, P., 122, 262n19, 263, 477 Hainmueller, J., 531 Hubbard, F., 494n2, 500 Hall, B. H., 132 Huber, P., 494 Hall, R. E., 47 Hunt, N., 603 Hansen, C., 93, 522 Hutter, F., 575 Hanson, G. H., 208 Hutter, M., 237n1 Hanson, R., 386n36 Harari, Y., 383 Imbens, G. W., 68, 517, 522, 523, 524, 525, Hardt, M., 80 529, 530, 536, 538 Harrell, E., 443, 444 Iriberri, N., 593, 594n10 Hart, O., 269 Irwin, D. A., 477 Hartford, J., 68, 75, 79n9, 526, 594 Isçan, T. B., 293n2 Haskel, J., 18 Hastie, T., 93, 526 Jaff e, A. B., 132, 150 Hatzius, J., 29 Jaravel, X., 312 Haugeland, J., 69 Jarrell, G., 502 Hawkins, J., 93 Jayadev, A., 366n14 Hay, B., 494n2, 499 Jean, N., 537 Hazan, E., 408 Jewitt, I., 112 He, K., 75 Jha, S., 94, 95 Heckman, J. J., 85, 182 Jiang, N., 528 Heifels, A., 116 Jin, G. Z., 442n4 Helpman, E., 169, 464, 476, 477, 478, 479, Joachims, T., 528 480 Johnson, E. J., 589, 593, 603 Hemous, D., 241 Jones, B. F., 127, 150, 155, 161, 259, 259n15, Hendel, I., 412 373n23 Henderson, R., 42, 43, 150 Jones, C. I., 46, 150, 152, 159, 160, 170n9, Herley, C., 452 171n9, 240, 252, 259n15, 373n23 Herrendorf, B., 204, 241n4 Jones, R. W., 472 Hersh, J., 537 Jordan, M. I., 509, 532 Hicks, J., 350, 368 Jovanovic, B., 41 Hillis, A., 516 Hinton, G. E., 68, 74, 76n7, 124, 125, 149, Kaboski, J. P., 293, 293n2 168, 168n6, 172 Kahn, L. B., 447n17 Hirschman, A., 178 Kahneman, D., 596, 600, 604 Hitt, L., 42, 43, 120 Kaldor, N., 240 Ho, T.-H., 592n3, 593, 595 Kallus, N., 528 Author Index 619

Kaplan, J., 3 Laibson, D., 606 Kaplow, L., 500 Lambrecht, A., 425, 426 Kapur, D., 150 Lancaster, K., 362 Karabarbounis, L., 270, 329 Landes, D., 206 Karagözog˘lu, E., 589 Lane, J., 554, 562 Katz, L. F., 7, 209, 322 Langford, J., 528 Kehoe, P. J., 42 Lanier, J., 84 Kehrig, M., 270 Lashkari, D., 241n4, 295 Kendrick, J. W., 43 Lawrence, R. Z., 294 Kennan, J., 590n2 LeCun, Y., 75, 149, 168, 168n6, 172 Kennedy, C., 376 Lederman, M., 108, 550 Keynes, J. M., 379 Legg, S., 237n1 Kislev, Y., 140 Leung, M. K. K., 121 Kitagawa, T., 527 Levin, J., 514, 515 Klayman, J., 599 Levine, D. K., 43 Kleinberg, J., 15, 507, 516, 518, 548, 588, Levine, S., 38 597 Levinthal, D., 161 Klenow, P. J., 310, 312, 477 Levitt, S. D., 43 Klette, T. J., 479 Levy, F., 37, 202n3, 238, 239n3 Kleven, H. J., 270 Lewis, E., 562 Knetsch, L., 604 Lewis, G., 526, 527 Ko, M., 450 Lewis, M., 596 Kogler, D. F., 560 Lewis, R. A., 541n2 Koh, D., 329 Lewis, W. A., 294 Kohno, T., 446 Leyton-Brown, K., 569, 575, 594 Kolb, D. A., 207 Li, D., 447n17, 529 Kollmeyer, C., 293n2 Li, L., 528, 529, 580, 581 Komarova, T., 538 Liang, A., 588, 594 Kominers, D., 501, 516 Lim, K., 465, 479 Kongsamut, P., 241n4, 295 Lindor, R., 494 Korinek, A., 354, 365, 366n14, 371, 373, Lipsey, R., 362 383, 384 List, J. A., 43 Korolova, A., 453 Litan, R. E., 441n2, 494 Kortum, S. S., 259, 259n15, 479 Liu, Y., 38 Kosinski, M., 429 Long, N., 241n4 Kotlikoff , L. J., 336, 344 Lovallo, D., 600 Krajbich, I., 592n6 Lowenstein, G., 448, 592n5 Kravitz, L., 325 Lu, Y., 529 Krisiloff , M., 325 Luo, H., 495, 496, 497, 498, 501 Krizhevsky, A., 68, 74, 125 Lusinyan, L., 29 Krueger, A. B., 7, 482 Lusted, L. B., 94 Krugman, P. R., 464, 472, 479, 480 Krussell, P., 265 Malthus, T. R., 384 Künzel, S., 527 Mamoshina, P., 168n7 Kurakin, A., 401 Managi, S., 465 Kurakin, P. R., 464 Mandel, M., 555 Kurzweil, R., 238, 253, 253n12, 350, 373n23, Manning, A., 379 381, 382 Mantoux, P., 200, 203 Kuznets, S., 205, 206 Manuelli, R. E., 205, 243n6 Manyika, J., 331 Lada, A., 540 Marchant, G., 494n2 Laff ont, J.-J., 514 Marco, A., 130, 132 LaGarda, G., 336 Markov, J., 93 620 Author Index

Markusen, J. R., 476 Murray, C., 325 Marthews, A., 425 Murray, F., 141 Marx, M., 479 Mussa, M. L., 471 Massey, C., 426 Mutz, R., 153 Masterov, D. V., 578, 579 Myers, A., 130 Matsuyama, K., 294, 295 Myerson, R. B., 568 Mayer, U. F., 578, 579 Mayer, W., 471 Naecker, J., 592 Mayzlin, D., 577n5 Naik, N., 516, 537 McAfee, A., 23, 23n1, 30, 40, 50, 89, 150, Nakamura, L., 29 201, 204, 309, 555 Nave, G., 590 McCall, J. J., 582 Neal, R. M., 74 McClelland, J. L., 602 Neelin, J., 589n1 McDonald, A. M., 424, 458 Neiman, B., 270, 329 McElheran, K., 560 Nekipelov, D., 538 McFadden, D., 85, 514 Nelson, P., 581 Mcguckin, R. H., 562 Nelson, R., 118, 161 McHale, J., 150, 241 Newbery, D., 362 McLaren, J., 469 Newell, A., 123 McSweeny, T., 583 Newhouse, D., 537 Meade, J. E., 362 Ng, A., 93, 509, 532 Meadows, M., 420 Ng, S., 532 Meehl, P. E., 596, 597 Ngai, L. R., 241n4, 294 Meltz, M. J., 488 Nickell, S., 293n2 Mestieri, M., 241n4, 295 Nielsen, M., 152, 161 Mian, A., 208 Nilsson, N., 122, 207, 208 Michaels, G., 201, 274, 319, 554 Nordhaus, W. D., 28, 152, 172, 238 Mikolov, T., 76 North, D. C., 577 Milgrom, P. R., 43, 111, 567, 569, 574, 577 Nosko, C., 577n5, 582 Miller, A., 428, 429, 448, 483 Miller, S. M., 201 O’Dea, B., 583 Milliment, D. L., 483 Odlyzko, A., 441n2 Minsky, M., 28, 208 Oettle, A., 241 Miremadi, M., 331 Olmstead, A. L., 204, 205, 206 Mishel, L., 322 Olsen, M., 241 Misra, S., 410 O’Mahony, S., 141 Mitchell, T., 39, 557, 563 O’Neil, C., 433 Mnih, V., 84 O’Reilly, B., 499 Mobius, M. M., 510 Orlikowski, W. J., 43 Mogstad, M., 7, 303 Orszag, P., 30 Mojon, B., 27 Osborne, M. A., 291, 321, 331, 350n1, 553, Mokyr, J., 29, 121, 150, 175, 201, 206, 209 555, 556 Monro, S., 78 Osindero, S., 74 Moore, G. E., 384n35, 498 Oskamp, S., 601 Moore, M., 496 Ossard, H., 514 Morris, D. Z., 37 Östling, R., 593n8 Mortensen, D. T., 379, 582 Ostrovsky, M., 568 Muellbauer, J., 85 Mullainathan, S., 169n8, 511, 512, 518, 588 Pajarinen, M., 556 Murdoch, J. C., 483 Pál, J., 510 Murnane, R. J., 202n3, 238, 239n3 Palfrey, T., 593 Murphy, K., 107 Parchomovsky, G., 494 Author Index 621

Pate, R. H., 420 Rosenblatt, F., 73, 124 Pelzman, S., 502 Rosenfeld, J., 322 Peretto, P. F., 241 Rossi-Hansberg, E., 476 Peterson, N., 599 Roth, A. E., 539, 567, 584, 589 Peysakhovich, A., 540, 592 Rousseau, P. L., 41 Phelps, E., 376n27 Rouvinen, P., 556 Philippon, T., 30 Rowthorn, R., 291, 293n2 Pierce, D. G., 413 Rubin, D. B., 517, 522, 527, 529 Pihur, V., 453 Rubinstein, A., 427 Piketty, T., 360 Ruffi n, R. J., 472 Pissarides, C. A., 241n4, 294, 379 Ruiz, F. J., 515, 533, 534 Polemarchakis, H., 354, 368n19, 369 Rumelhart, D. E., 73, 77, 124, 602 Polinsky, M., 494n2 Porter, M. E., 481, 494 Sabour, S., 76n7 Poschke, M., 241n4 Sachs, J. D., 336, 344 Posner, R. A., 439 Saez, E., 270, 360 Prantl, S., 263 S¸ahin, E., 270, 329 Pratt, G. A., 40 Salakhutdinov, R. R., 125 Proserpio, D., 577n5 Salomons, A., 23n1, 309, 555 Puga, D., 480 Samuelson, P., 376 Santaeulalia-Llopis, R., 329 Raghaven, M., 518 Saon, G., 25 Ramaswamy, R., 291, 293n2 Sawyer, J., 597 Ramlogan, R., 480 Saxenian, A.L., 4, 478 Rao, J. M., 516, 536, 541n2 Schierholz, H., 322 Rasmussen, W. D., 200, 203, 205n6, 206 Schmidhuber, J., 73, 76 Rauch, J. E., 469 Schmidt, K. M., 358n8 Rebelo, S., 241n4, 295 Schmitt, J., 322, 379 Recht, B., 80 Schmitz, J. A., 43 Redding, S., 293n2 Schmucki, R., 497 Reinsdorf, M. B., 29 Schultz, P. H., 414 Rennie, J., 80 Schumpeter, J., 148 Restrepo, P., 23n1, 43, 90, 105, 127, 152, 197, Schwartz, M., 568 201, 202, 203, 203n4, 204, 204n5, 206, Scotchmer, S., 121, 496, 502 208, 210, 211, 212n7, 219, 220, 220n11, Scott, S. L., 528 223, 224, 225n14, 238, 240, 241, 243, Seater, J. J., 241 271, 283, 376n27, 379, 554 Segal, I., 414, 569, 574 Rhode, P. W., 204, 205, 206 Seira, E., 514 Rhodes, E., 325 Seshadri, A., 205, 243n6 Rivera-Batiz, L. A., 477, 478 Shah, A. K., 517 Rivkin, J., 161 Shaked, A., 589n1 Robbins, H., 78 Shavell, S., 494n2 Roberts, J., 43, 111 Shaw, J. C., 123 Robinson, P. M., 524, 527 Shiller, B. R., 410 Rock, D., 557 Shroff , R., 516 Rodrik, D., 293 Silver, D., 63, 66, 453 Roesner, F., 446 Simcoe, T., 487 Rogerson, R., 204, 241n4 Simmons, J. P., 426 Romanosky, S., 444, 450, 457, 457n42 Simon, H. A., 107, 123, 207–8 Romer, P. M., 46, 122, 149, 153, 155n2, 159, Sims, C., 113 169, 171, 171n9, 172, 255, 477, 478 Simsek, A., 362n10 Rosenberg, N., 150, 169 Singer, Y., 80 622 Author Index

Slovic, P., 596 Taddy, M., 83, 526, 527 Smith, A., 590 Tadelis, S., 107, 577n5, 578, 579, 580, 581, Smith, M. D., 43 582 Smith, N., 29 Tang, J., 453 Sokoloff , K. L., 430 Taylor, C. R., 416, 424, 441n2, 448, 459, Solomonoff , R. J., 253n12 483n19 Solove, D., 447, 455 Tegmark, M., 382, 384n34 Soloveichik, R., 29 Teh, Y.-W., 74 Solow, R. M., 24, 46, 350 Telang, R., 444, 450, 457, 457n42 Somanchi, S., 444 Teodoridis, F., 150, 161 Song, J., 30 Tetenov, A., 527 Sonnenschein, H., 589n1 Thaler, R., 604 Sopher, B., 590n2 Thomas, K., 443 Sorensen, O., 161 Thomas, P., 528 Spencer, B. J., 473 Thompson, W. R., 82 Spezio, M., 594n10 Thrnton, B., 598 Spiegel, M., 589n1 Tibshirani, R., 93, 525, 527 Spiegel, Y., 412 Tirole, J., 106, 112, 264, 268, 269 Spier, K., 494, 499, 502 Topol, E. J., 94, 95 Spiess, J., 169n8, 511, 512 Tory, J., 485n23 Spindler, M., 522 Toulis, P., 80 Srivastava, N., 80 Trajtenberg, M., 4, 39, 116, 119, 132, 150, Stahl, D. O., 593 169, 176n2 Stantcheva, S., 360 Treﬂ er, D., 465, 467, 479, 485n23 Stein, A., 494 Troske, K. R., 562 Stern, A., 325 Tschantz, M. C., 433 Stern, S., 122, 141 Tucker, C. E., 148, 425, 426, 427, 428, 429, Stevenson, B., 194n5 448, 449, 482, 483, 483n19 Stigler, G. J., 439, 582 Turing, A. M., 123, 385 Stiglitz, J. E., 30n9, 354, 360, 362, 364, Tuzel, S., 210 364n11, 366n14, 366n15, 368n19, 369, Tversky, A., 596 370, 370n21, 371, 373, 376n27, 376n28, 378, 412 Ugander, J., 538 Stillwell, D., 429 Uyarra, E., 480 Stinchcombe, M., 74 Stivers, A., 442n4 Vadlamannati, K. C., 483 Stole, L., 268 Valentinyi, Á., 204, 241n4 Strehl, A., 527 Van der Laan, M. J., 522, 527 Streitwieser, M. L., 562 Van Seijen, H., 62, 84 Strotz, R. H., 427 Varian, H. R., 93, 310, 410, 413, 424, 425, Stucke, M. E., 414 440, 511 Stutzman, F., 451 Venables, A. J., 480 Suﬁ , A., 208 Vesteger, M., 420 Summers, L. H., 175 Vickrey, W., 567 Sutskever, I., 68, 74, 125 Vijverberg, W. P. M., 483 Sutton, J., 467, 589n1 Vincent, N., 270 Swaffi eld, J., 293n2 Vines, P., 446 Swaminathan, A., 528 Vinge, V., 238, 253, 253n12, 373n23 Sweeney, L., 433 Viscusi, K., 496, 498 Swire, P. P., 441n2 Vishnu, A., 151n1, 168 Syverson, C., 23, 29, 43, 210, 319, 320 Von Hippel, E., 499 Author Index 623

Von Weizacker, C. C., 376 Wright, J. R., 594 Von Winterfeldt, D., 595 Wu, L., 43 Vuong, Q., 514 Xiao, M., 593n8 Wager, S., 523, 525, 527, 528 Xie, D., 241n4, 295 Wagman, L., 416, 424, 441n2, 448, 459, Xu, H., 577 483n19 Wallach, I., 116 Yakovlev, E., 538 Wan, M., 534 Yang, S., 42, 47 Wang, J., 475, 477, 594n10 Yellen, J., 378 Warren, S. D., 431, 459 Yeomans, M., 517 Waseem, M., 270 Yildirim, P., 107n9 Wattal, S., 450 Yu, M., 465, 479 Weil, D., 459 Yudkowsky, E., 253n12 Weingast, B. R., 577 Weiss, A., 354 Zanna, L.-F., 373n24 Weitzman, M., 149, 151, 157, 171, 171n9, Zeevi, A., 529 172, 241 Zeileis, A., 525 Western, B., 322 Zeira, J., 152, 198, 212n7, 238, 239, 241 Westlake, S., 18 Zervas, G., 577n5 Whinston, M. D., 148 Zhang, M. B., 210 White, H., 74, 511 Zheng, Y., 329 Williams, H., 122, 364n11 Zhou, D., 529, 581 Williams, R., 124 Zhou, X., 580 Wilson, P. W., 593 Zhu, X., 471 Winter, S., 161 Zierahn, U., 321 Wolfers, J., 195n5 Zubizarreta, J. R., 523 Wooldridge, J. M., 522 Zweimüller, J., 295 Wright, G., 42n19 Zwiebel, J., 268 Wright, G. C., 303

Subject Index

Note: Page numbers followed by “f” or “t” refer to fi gures or tables, respectively. adoption, technological: implications of 351; inequality and, 320–23; internal speed of, for job market and inequality, agreements and, 463; international 310– 12 macroeconomics and, 488; knowledge adversarial artifi cial intelligence, 401 externalities and, 470–71; likely produc- aggregate productivity statistics, technolo- tivity eff ects of, and acceleration of, 45– gies and, 26–28 46; longer-term prospects of, 381–86; AI. See artifi cial intelligence (AI) market structure and, 262–63; as “next AlphaGo (Google), 63 big thing,” 175; political economy of, Amazon Go concept, 67 11, 394–95; prediction costs and, 92–93; applied artifi cial intelligence, 208 privacy and, 425–26; privacy concerns artifi cial intelligence (AI), 1; and automa- and, 423–24; for promoting trust in tion of production, 239–50; as basis online marketplaces, 576–81; recent for learning, 120–21; benefi t of more, approach to, 93; for reducing search 318–20; bibliometric data on evolution frictions, 581–83; return of Malthus of, 128–32; capital shares and, 270–74, and, 381–86; revolution, international 272–73f; in context, 84–85; defi ned, eff ects of, 393–94; in Schumpeterian 3–4, 62–67, 93, 122, 237, 468; econo- model with creative destruction, 276– mies of scale from data and, 468–69; 79; sectoral reallocation and, 263–64; economies of scale from overhead statistics on, 465–66, 466t; studies on of developing AI capabilities, 469– economic eff ect of, 556– 58; theory of 70; evolution of, 122–25; expected privacy in economics and, 424–26; as productivity eff ects of, 45– 46; firm tool, 16–17; world’s largest companies organization and, 264–70; future of and exposure to, 465–67, 467t. See also research on economics of, 17; as general machine learning (ML) purpose technology, 4–7, 39–41; in idea artifi cial intelligence capital, measuring, production function, 250–52; impact 46– 50 of, on innovation, 125–28; impact of artifi cial intelligence–general purpose tech- long-term decline in labor force partici- nology (GPT) era: education strategies pation rate and, 323–25; implications for, 179–82; human-enhancing innova- of, 349–53; income distribution and, tions vs. human-replacing innovations

625 626 Subject Index artifi cial intelligence–general purpose tech- consumer attitude, 448–49 nology (GPT) era (continued) consumer privacy: challenging issues in, for, 184–85; professionalization of 457–59; consumer attitude and, 448–49; personal services strategies for, 182–84; consumer risk and, 443–48; data risk top skills required for employment in, and, 442–43; nature of problem of, 442; 180– 81, 181t policy landscape in United States, 454– artifi cial intelligence revolution, inter- 57; supply side actions and, 450–54. national eff ects of, 393– 94 See also privacy Atomwise, case of, 115–16, 120, 154 consumer surplus, 11; distribution of, automatic teller machines (ATMs), security 391– 93 policy and, 416 contracting, 106–7 automation, 3–4, 105–6; basic model, 336– convolutional neural networks (CNNs), 41; Baumol’s cost disease and, 241–50; 75– 76, 75n6 to date, and capital shares, 270–74; cooperation, evolution of, 414 decline in labor share and, 329–31; cost disease, Baumol’s, 8–9, 238–39; auto- deepening of, 198, 204–5, 216–17; eco- mation and, 241–50 nomic adjustment and, 208–9; employ- creative destruction, 260–61 ment and, 190–91; excessive, 224–26; model of, 211–14; of production, and data, 61; acquisition methods, 403–4; de- artifi cial intelligence, 239–50; produc- creasing marginal returns of, 406, 407f; tivity and, 210–11; sector of economy economics of, 14; importance of, 13– aff ected by, 330–33; studies on employ- 14; important characteristics of, 404–6; ment on, 555–58; wages and, 200–211; localization rules, trade policies and, winners, 190; work and, 200–211; Zeira 485–86; persistence of predictive power model of growth and, 239–41 of, 427–28; privileged access to govern- average treatment eff ects, 522– 24 ment, trade policies and, 486–87; types of, and generation of spillovers, bandits (algorithms), problem of, 528–29 431– 34 Baumol’s cost disease. See cost disease, data access, 405–6 Baumol’s data generation, as pillar of artifi cial intel- BenchSci search technology, 153 ligence, 62f, 65–66 buy/make decisions (fi rm boundaries), data ownership, 405–6 107– 8 data persistence, 426–27; predictive power and, 427–28 capital accumulation, 198, 204–5, 216 data pipeline, 402 capital shares, and automation to date, data pyramid, 404, 405f 270– 74 data repurposing, 428–31 causal inference, new literature on, 519–34 data security: challenging issues in, 457– Children’s Online Privacy Protection Act of 59; policy landscape in United States, 1998 (COPPA), 454 454– 57 cloud- computing facilities, 402– 3 data spillovers, 431–34 cluster policies, 480– 81 data warehouses, 402–3 clusters, regional, theory of, 479 decision- making, baseline model for, 95– CNNs (convolutional neural networks), 75– 103; complexity and, 103– 8 76, 75n6 deepening of automation, 198, 204– 5, collusion, strategies for facilitating, 413– 14 216– 17 combinatorial- based knowledge production Deep Genomics, 154 function, 154–61; potential uses of new, deep learning, 3, 71–77, 400; as general pur- 170–71; with team production, 161–67 pose invention in method of invention, Communications Act (1986), 456 139–43; as general purpose technology, competition policy, innovations and, 141–43 133–39; as new discovery tool, 167–69; complexity, 103–4 patent systems and, 142 Subject Index 627 deep learning networks, 94 GPT. See general purpose technology (GPT) deep learning techniques, 25 Gramm-Leach- Bliley Act (GLBA), 454 deep neural networks (DNNs), 25–26, 61, growth: impact of artifi cial intelligence on, 63; structure in, 76–77 7– 9. See also economic growth demand, importance of, 301–2 destruction, creative, 260–61 Health Insurance Portability and Account- diff erence- in-diff erence models, 530–31 ability Act of 1996 (HIPAA), 454 digital information, 334–35 HEI (human-enhancing innovations), direct network eff ects, 412 184– 85 displacement eff ect, 8, 198, 208, 214 heterogeneous treatment eff ects, 524– 28 DNNs. See deep neural networks (DNNs) hierarchical Poisson factorization, 510 domain structure, as pillar of artifi cial intel- HIPAA. See Health Insurance Portabil- ligence, 62f, 63–65 ity and Accountability Act of 1996 double machine learning, 523–24 (HIPAA) HRI (human-replacing innovations), 184–85 economic growth: artifi cial intelligence and, human-enhancing innovations (HEI), 184–85 262–70; prospects for technology- human-replacing innovations (HRI), 184–85 driven, 149–53; Zeira model of automation and, 239–41. See also growth idea production function, artifi cial intelli- economics, impact of machine learning on gence in, 250–52 practice of, 15–16 implementation/ restructuring lags, as expla- education, factory model of, 180 nation for Solow paradox, 31–36 Electronic Communications Privacy Act of incentive auctions, machine learning and, 1986: (ECPA), 456 569– 76 employment: automation and, 190–91; income, artifi cial intelligence and, 189–90 levels of, and new technologies, 220–21; income distribution: artifi cial intelligence long-run vs. short run, 192–94; studies and, 351; impact of AI on, 11 on automation and, 555–57; work out- income inequality: artifi cial intelligence and, side of, 194–95 320–23; impact of AI on, 7–8, 11–12; evolution of cooperation, 414 speed of technological adoption and, 310– 12 factory model of education, 180 income redistribution, political economy of, Federal Trade Commission (FTC), 454–55 394– 95 fi rm boundaries (make/buy decisions), indirect network eff ects, 412 107– 8 industrial regulation, trade policies and, 487 fi rm-level data: need for, 558–59; strategies inequality. See income inequality for collecting, 561–62 information technology (IT), 24 fi rm-level research questions, 560–61 innovation, 115–18; competition policy and, fi rms: artifi cial intelligence and, 262–70; 141–43; early stage, 121–22; impact impact of machine learning on, 12 of artifi cial intelligence on, 125–28; institutions and, 141–43; management general purpose machine learning (GPML), and organization of, 140–41; product 67–71 liability and, 494 general purpose technology (GPT), 2, 65, institutions, innovations and, 141–43 119– 20, 169– 70; artifi cial intelligence intelligence- assisting innovation (IA), as, 4–7, 39–41; deep learning as, 133– 350– 51 39; viewing today’s Solow paradox International Federation of Robotics through previous history of, 44–45 (IFR), 16 generative adversarial networks (GANs), international macroeconomics, artifi cial 66– 67 intelligence and, 488 GPML (general purpose machine learning), international trade, economics of data 67– 71 and, 14 628 Subject Index invention of a method of inventing (IMI), machine learning– provision industries, 120– 21, 124 414– 15 inverted-U pattern, 293; simple model of, machine learning– using industries, 408f; 297– 301 boundaries and, 409– 10; fi rm size and, 409– 10; price diff erentiation and, 410– JDM (judgment and decision-making) re- 11; pricing and, 410; returns to scale search, 596–98 and, 411– 14; structure of, 406– 8 job displacement, 310–12 macroeconomics, international, artifi cial job losses, 291 intelligence and, 488 job markets, speed of technological adop- make/buy decisions (fi rm boundaries), tion and, 310–12 107– 8 jobs, impact of artifi cial intelligence on, 7–8, Maluuba, 62– 63 9– 11 market design, introduction to, 567– 69 judgment: in absence of prediction, 96–101; massive open online courses (MOOCS), as complements/substitutes to predic- 181 tion, 102–3; creating role for, 91; predic- matrix completion problem, 531–32 tion and, 91–92 matrix factorization, 51, 511 judgment and decision-making (JDM) meta technologies, 153 research, 596–98 ML. See machine learning (ML) model averaging, 511 knowledge creation, 477–79 MOOCS (massive open online courses), knowledge externalities, and artifi cial intel- 181 ligence, 470–71 knowledge spillovers, 479 neural networks, 72–74, 123, 124–25, 510, 511 labor: comparative advantage of, and new new economic geography (NEG), 479–80 tasks, 217–18; model of demand for, 211–14 online marketplaces, using artifi cial intelli- labor demand, technology and, 214–21 gence to promote trust in, 576–81 labor productivity growth, technologies and, optical lenses, invention of, 121 26– 28 optimism, sources of technological, 24–26 learning by doing, 412– 13 liability, innovation and, 494; empirical evi- Pareto improvement, 363 dence on, 496– 98; theoretical model of, patent systems, deep learning and, 142 494– 96 Pen Register Act (1986), 456 liability, tort, development of artifi cial intel- policy analysis, using methods of prediction ligence technologies and, 498–502 for, 516–19 political economy: of artifi cial intelligence, machine learning (ML), 3, 24–25; applica- 11; of income redistribution, 394–95; tions of, 401–2; defi ned, 509– 10; of technological disruptions, 176–79 double, 523–24; early use cases of, 510– prediction: in absence of judgment, 101–2; 15; for fi nding behavioral variables, artifi cial intelligence as tool for, 16–17; 587–96; general purpose, 67–71; human as complements/substitutes to judg- prediction as imperfect, 496–603; im- ment, 102–3; costs of, and AI, 92–93; pact of, 12–13, 15–17, 507–9; incentive judgment and, 91–92; using methods auctions and, 569–76; new literature of, in policy analysis, 516–19 on, 519–34; overview, 399–406; predic- premature deindustrialization, 393–94 tions about impact of on economics, price discrimination, artifi cial intelligence 534–42; regulation and, 12–15; super- and, 604 vised, 511–12; unsupervised, 510–11; principal components analysis, 74, 92, vertical integration and, 408–9. See also 510 artifi cial intelligence (AI) privacy, 13–14; artifi cial intelligence and, Subject Index 629

425–26; current models of economics search frictions, artifi cial intelligence for and, 424–25; data spillovers and, 431. reducing, 581–83 See also consumer privacy security policy, 416 privacy policy, 416–17 singularities, 253–61; examples of techno- privacy regulation, trade policies and, logical, 254–58; objections to, 258–61 482– 85 skills: mismatch of technologies and, 221– privileged access to government data, trade 23; technologies and, 209 policies and, 486–87 Solow paradox, 24; potential explanations production, automation of, and artifi cial for, 28–31 intelligence, 239–50 source code, trade policies and, 487–88 productivity: automation and, 210–11; miss- spectrum reallocation, 569–76, 572f ing growth of, and new technologies, spillovers, data, 431 223– 26 spreadsheet software, invention and impact productivity eff ects, 198, 203– 4, 214– 16 of, 90 productivity growth: low current, reasons stochastic gradient descent optimization, why it is consistent with future techno- 77– 81 logical growth, 41–44; rates of, tech- strategic trade policy, 473–74 nologies and, 26–28; slow, and future structural change, 293–96 productivity growth, 31–36 structural models, 532–34 productivity optimism, technology-driven superstar scientists, role of, 474–76 case for, 36–39 supervised machine learning, 511; methods for, 511–12 radiology, case of, 94–95 supplementary analysis, 530 random forest, 511 support vector machines, 511 R&D. See research and development (R&D) symbolic processing hypothesis, 123 recommender systems, 603 symbolic systems, 123 regional clusters, theory of, 479 regression trees, 511 tasks, new, 205–7; comparative advantage regularization on norm of matrix, 510 of labor and, 217–18; creation of, 198; regularized regression, 511 model of, 211–14 regulation, 12–15; machine learning and, technological changes: factor-biased, 376– 12– 15 77; and levels of employment, 220–21; reinforcement learning, 25, 66, 81–84, 400– types of, 212–13 401 technological disruptions, political economy reinstatement eff ect, 8, 198, 206 of, 176–79 research and development (R&D), 336; pro- technological growth, reasons why it is con- ductivity, eff ects of rise in, 341– 43 sistent with low current productivity research tools, economics of new, 118–22 growth, 41–44 robotics, 123–24; tort law and, 493–94. technological optimism, sources of, 24–26 See also robots technological progress: channels of inequal- robots, studies on, 554–55. See also robotics ity and, 365–70; determining scenarios Romer/Jones knowledge production func- that best describe economy, 363–64; tion, 151–52 endogenous, 364–65; fi rst- best scenario, 353–56; imperfect markets scenario, scale, economies of, and artifi cial intelli- 361–62; perfect markets but costly gence, 470 redistribution scenario, 358–61; perfect Schumpeterian model with artifi cial intel- markets ex post and no costs of redis- ligence, 276–79 tribution scenario, 356–58; welfare and, scientifi c discovery, rate of, 6 353– 65; worker- replacing, redistribu- scientists, role of, 472–73; superstar, 474–76 tion and, 370–77 scope, economies of, and artifi cial intel- technological singularities, examples of, ligence, 470 254– 58 630 Subject Index technological unemployment, 377–81 universal basic income (UBI), 312–14; cost technologies: future progress of, and low of replacing current safety net with, current productivity growth, 41–44; 325– 26 labor demand and, 214–21; mismatch unsupervised machine learning, 510–11 of skills and, 221–23; skills and, 209 USA Patriot Act (2001), 456 technology-driven economic growth, prospects for, 149–53 vertical integration, machine learning and, tort law, robotics and, 493 408–9 tort liability, 14 vertical research spillovers, 122 total factor productivity growth, 32 trade models, basic, 476–77 wages, automation and, 200–211 trade policies: data localization rules and, welfare, technological progress and, 353–65 485–86; industrial and strategic, case work, automation and, 200–211 for, 471–81; industrial regulation and, worker-replacing technological progress: 487; privacy regulation and, 482–85; dynamic implications of, 373–74; redis- privileged access to government data tributing innovators’ surplus and, 374– and, 486–87; role of university-related 76; redistribution and, 370–77; static talent, 472–76; source code and, 487– pecuniary externalities of, 371–72 88; strategic, 473–74 Zeira model of automation and growth, UBI. See universal basic income (UBI) 239–41 unemployment, technological, 377–81 “zero-shot” learning systems, 66