Evaluating the Atheoretical Turn in Economics 1

Form and Function: Evaluating the Atheoretical Turn in Economics

Meredith M. Paker Nuﬃeld College University of Oxford meredith.paker@nuﬃeld.ox.ac.uk

Prepared for Philosophy and Methodology of Economics Economic and Social History June 2017

1 Introduction

Since the late 1990s, research in economics has increasingly employed “atheoretical” experimental and quasi-experimental methods.1 As laboratory and field experiments became more accepted in the discipline, paralleling the rise of behavioral economics, statistical quasi-experimental methods were developed and standardized. These tools, including difference-in-difference, regression discontinuity, and instrumental variable approaches, have made quasi-experimental analyses of natural or institutional variation more accessible, popular, and persuasive. Experimentally-based approaches are now common throughout the discipline and are especially prominent in empirical microeconomic research, where they have displaced structural and theory-driven work in fields such as labor, education, and public economics.2 Two main reasons are often cited for the growth of work in experimental economics. First, experimentally-based methods are seen as “atheoretic” when compared with tradi-

1. “Economists are prone to fads, and the latest is machine learning,” The Economist, November 2016, http://www.economist.com/news/finance-and-economics/21710800-big-data-have-led- latest-craze-economic-research-economists-are-prone, In addition to an increase in machine learning methods, the figure shows a remarkable upward trend in laboratory, randomized control trial, difference-in-difference, and regression discontinuity approaches. 2. John Rust, “Comments on ‘Structural vs. Atheoretic Approaches to Econometrics’ by Michael Keane,” Journal of Econometrics 156, no. 1 (2010): p. 21.

1 tional structural methods that make a priori parametric assumptions.3 Second, structural modeling is difficult, time consuming, and less likely to be appreciated by academic jour- nals looking for intuitive and easily-explained work. This skews professional incentives toward the more accessible and time-efficient experimental methods.4 Regardless of the motive, the move from broadly structural to broadly experimental research requires some methodological consideration. Experimental work abandons the traditional deductive model of reasoning in favor of simpler “atheoretic” methods, osten- sibly avoiding the sticky issues of theory validation and realism of assumptions. However, critics have argued that these experimental methods may still involve strong assumptions5 and may rely on unsound inductive reasoning for external validation.6 Thus the shift in economic methods from structural to experimental reflects a deeper shift in the “form” of economic reasoning. This might also imply a shift in the “function” of economics as a discipline. Though there are still disagreements about the exact goals of the field, both realists and instrumentalists would agree that some level of policy- relevance is desirable.7 A shift toward experimental methods, however, could threaten the ability of economics to be policy-relevant. In this essay, I will begin by considering the methodological developments of the twentieth century, where issues of deduction and empirics became central debates. With this essential grounding, I will then compare and contrast the experimental and structural approaches to economics on a general level. I will argue that experimentally-based methods require unsound inductive reasoning to achieve external validity. Then, I will question whether this makes experimental economics a weaker tool for policy than analyses rely- ing on deductive methods, which are developed to be more universal and which explicitly state their premises. Finally, to conclude, I will briefly consider whether the experimental turn in economics also reflects a shift in the intended purpose of economics. Is policy relevance becoming less important to the discipline?

3. Michael P. Keane, “Structural vs. Atheoretic Approaches to Econometrics,” Journal of Econometrics 156, no. 1 (2010): pp. 3-4. 4. Rust, “Comments on ‘Structural vs. Atheoretic Approaches to Econometrics’ by Michael Keane,” p. 21. 5. James Heckman, “Instrumental Variables: A Study of Implicit Behavioral Assumptions Used in Making Program Evaluations,” The Journal of Human Resources 32, no. 3 (1997): 441–462; Mark R. Rosenzweig and Kenneth I. Wolpin, “Natural ‘Natural Experiments’ in Economics,” Journal of Economic Literature 38, no. 4 (2000): 827–874. 6. Volker Gadenne, “External validity and the new inductivism in experimental economics,” Rational- ity, markets, and morals 4, no. 1957 (2013): 1–19. 7. Daniel M. Hausman, The Inexact and Separate Science of Economics (Cambridge: Cambridge University Press, 1992), pp. 285-288.

2 2 The tradition of deduction in economics

Economics has been portrayed and conducted as a deductive science since the classical economists. Though Adam Smith is a notable exception, Ricardo, Malthus, and J.S. Mill all relied on deductive methods in their analyses. The neoclassical tradition and its further developments have continued in this vein, and the deductive foundations of economics have been the subject of much consideration and criticism in philosophy of economics. In this section, I will present Mill’s 1836 deductive methodology for political economy in detail and then will consider modern variants of this deductive method. I will also present the problem of induction as an insoluable challenge to purely empirical work. Counter-methodologies have also been influential, though not dominant, since the late nineteenth century. The German Historical School, early Institutional economics, and the Wisconsin School all emphasized the historical dependency of economic theory, which is an important threat to the validity of global deductions. While space does not permit a detailed discussion of these schools of thought, I will present the problem of induction as an insoluble challenge to purely empirical work in this section and will briefly consider the major empiricist and instrumentalist alternatives to deduction in the next section. Ultimately, this conversation will serve as important backdrop to our consideration of the methodologies of empirical microeconomics dominating the field today.

J.S. Mill’s a priori method for political economy

John Stuart Mill famously makes the case for a priori deductive reasoning in his 1836 essay “On the Definition of Political Economy and the Method of Investigation Proper to It.” Though Mill was an inductivist and empiricist, supporting observational and experimental foundations for knowledge in the natural sciences, he considered political economy a separate field posing special challenges of two types. First, Mill’s political economy offers only a limited ability to conduct experiments due to its consideration of entire systems of governments and broad policies, which are hard to vary for experiment. Second, in political economy and in other “moral sciences” there are many factors that can contribute to an effect, making it difficult to control for all influential conditions. These two characteristics prevent political economy from obtaining Bacon’s experimen- tum crucis, which proves a causal effect by negating all other possible explanations for some occurrence. Thus political economy needs to rely on a priori reasoning, deducing predictions from an assumed hypothesis. While Mill’s a priori method of political economy is deductive, it relies on an es-

3 tablished and unquestionable hypothesis as a beginning premise. To Mill, this assumed hypothesis should be a universal principle that is achieved inductively from observation about human nature. He believes it is possible to induce principles of human nature from observation, writing, “Although sufficiently ample grounds are not afforded in the field of politics, for a satisfactory induction by a comparison of the effects, the causes may, in all cases, be made the subject of specific experiment...The desires of man, and the nature of the conduct to which they prompt him, are within reach of our observation.”8 Thus, to Mill, we can use the laws of nature and our observations about the actions of man as universal principles that provide unshakable premises for valid deduction. These laws can sometimes be psychological in nature, determined introspectively, or can relate to the natural sciences, determined through experiment. In both cases, the laws should be proven empirically, through observation or experiment, prior to the deduction. From these universal laws, the political economist can deduce specific laws about the operation of the economy. Further predictions can then be deduced from these more specific laws. Mill offers a famous description of a theorist deriving laws inductively and then making deductions: “...those who are called theorists aim at embracing a wider field of experience, and, having argued upwards from particular facts to a general principle including a much wider range than that of the question under discussion, then argue downwards from that general principle to a variety of specific conclusions.”9 The final conclusions are always justified in this deductive-nomological model if the premises hold. To Mill, this condition is always met because at the top of this chain of logic is an infallible universal law induced from observation. Thus regardless of how many levels of deduction there are, the final predictions must be true because the premises hold. However, Mill understood that the deduced principles of political economy do not always reflect the observed world. He explains this discordance using the concept of disturbing causes. Because the moral sciences consider effects that are determined by a concurrence of causes, it is always possible that not all relevant circumstances have been taken into account in an analysis. If predictions do not match reality, then, a political economist just needs to take more of these disturbing causes into account. This leaves political economy only a science of tendencies, where any causal relationship it uncovers is just “a power acting with certain intensity in that direction.”10 A posteriori reasoning can indicate when more disturbing causes need to be considered. Therefore, discrepancies between empirical evidence and the deductions from theory

8. J.S. Mill, “On the Deﬁnition and Method of Political Economy,” in The Philosophy of Economics: An Anthology, 3rd, ed. Daniel M. Hausman (New York: Cambridge University Press, 2008), p. 48-49. 9. Ibid., p. 44. 10. Ibid., p. 56.

4 can only be due to disturbing causes that have not been taken into account, and these only threaten the accuracy of prediction in a real-world setting, not the truth of the deduction. Empirical evidence cannot falsify or refute the original premises—the universal principles about human nature and about the political economy—because these are inductively-determined laws. Mill’s a priori method was inﬂuential among the early methodologists, including J.N. Keynes and Lionel Robbins. Robbins’s philosophy of science is extremely similar to Mill’s, where the laws of economics are deduced from facts about human behavior that are obvious from experience.11

The hypothetico-deductive method and deduction since Mill

The hypothetico-deductive (HD) method is the standard view of theory appraisal, attributed to Hempel and Popper but with some nineteenth and twentieth century an- tecedents. It is similar to Mill’s a priori method discussed above but permits the evaluation of the assumed theory based on the accuracy of its economic predictions. The HD method again begins with an assumed hypothesis from which one deduces predictions. However, these predictions are then tested against empirical evidence. If the predictions are correct, confidence in the initial hypothesis increases, while if the predictions fail, confidence can decrease. While it is possible that a failure of evidence to match the predictions of the model is caused by disturbing causes or invalid deductions, the HD method is primarily intended to permit the confirmation or refutation of the assumed theoretical hypothesis. Popper makes only a small adjustment to the HD model, arguing that in the final step, falsification of hypotheses is more valid than confirmation of hypotheses. This is due to the asymmetry between verifiability and falsifiability. A universal statement can be falsified by a contradictory basic statement, but a set of basic statements cannot imply a universal statement. Popper’s version of the HD model thus remains primarily deductive. Hausman offers an additional deductive method for economics, which he calls “the inexact a priori method.” His argument centers around the idea that laws in economics are inexact—they can be only approximately true, they can reflect statistical regularities or probabilities instead of certainties, they can describe causal relationships in ideal conditions without interferences instead of in realistic conditions, and they can be qualified by ceteris paribus clauses that are difficult to break down.12 Hausman argues that so long as an “inexact” law is lawlike, reliable, refinable, and

11. Lionel Robbins, “The Nature and Signiﬁcance of Economic Science,” in The Philosophy of Eco- nomics: An Anthology, 3rd, ed. Daniel M. Hausman (New York: Cambridge University Press, 2008), p. 78-79, originally published in 1935. 12. Hausman, The Inexact and Separate Science of Economics, p. 128.

5 excusable, it can be used as an assumed hypothesis in Mill’s a priori method. Like Mill, Hausman argues that the assumed hypothesis should not be questioned if an empirical test fails. Instead, one should consider the deduction itself, whether other factors interfered, and whether the assumed laws have enough power and relevance.13

Inductive reasoning?

Induction, which features prominently in the Mill model of a priori reasoning, is not a sound logical system. The “problem of induction” has been written about extensively in the philosophy literature, and most philosophers of science believe the problem is insoluable. It is often attributed to Hume, who argues in An Enquiry Concerning Human Understanding 14 that inductive-type reasoning is driven by subjective sentiments of a repeated connection between two events, not reflecting true objective or causal relationships. Briefly, the essence of the problem is whether induction can lead to new knowledge in a philosophically sound way. Is there justification to generalize universal properties of something based on particular instances of it? Can we assume that the future will follow the same patterns of the past? It is difficult to respond to these statements and “prove” induction without making assumptions about the uniformity of nature and without using circular arguments that rely themselves on induction. Deductive models do not pose these same challenges. Deductive logic is complete, so the assumptions that are premises of an argument yield the predictions of the argument if those assumptions are true. Because there is no sound system of logic like this for inductive reasoning, empiricist and instrumentalist alternatives to the deductive models described above try to circumvent the problem of induction. Few philosophers of science would make the case for purely inductive methods of reasoning.

3 Empiricist critiques of the HD model

In the Mill-Robbins-Hausman type deductive model, empirics play only a secondary role because they cannot refute the assumed hypotheses. Instead, they can only indicate when some disturbing cause or error of deduction has taken place. In contrast, in the HD model, especially the Popperian variant, empirics can refute a theory in addition to indicating a disturbing cause. In this section, I will consider models that give empirics an

13. Hausman, The Inexact and Separate Science of Economics, p. 147-148. 14. David Hume, “An Enquiry Concerning Human Understanding,” in Enquiries Concerning Human Understanding and Concerning the Principles of Morals, 3rd (Oxford: Oxford University Press, 1989), pp. 73–79, originally published in 1748.

6 equal or larger role, placing empirical tests of predictive power at the center of economic activity. Though there are many methodologies in this vein, I consider Hutchinson’s ultra-empiricism, Samuelson’s operationalism, Machlup’s more modest empiricism, and Friedman’s instrumentalism. An early empiricist critique comes from T.W. Hutchinson,15 who argues that all economic statements are either empirical and falsifiable or analytical and tautological. To Hutchinson, the propositions of pure theory are analytical-tautological and thus cannot contain any empirical content or be used for useful deduction. Instead, economic work should concern itself with induction from testable empirical statements and should avoid qualifying these with ceteris paribus clauses. Empirical content should not be “read into the propositions of pure theory just because of their necessity and inevitability.”16 The obvious critique of this ultra-empiricism is that generalizations cannot be made without induction. Hutchinson’s sharp criticism thus does not provide a working model for an empiricist approach to economics. Attempting to reconcile economic theory with Hutchinson’s empiricism, Paul Samuelson develops “operationalism,” which he uses to argue for revealed preference theory. In Samuelson’s model, one should test all of the possible predictions of a theory to see which hold empirically. Then, one should replace the original theory with a revised theory that only implies these true consequences. While this method escapes the problem of induction, it has been criticized by commentators such as Hausman for its incoherence.17 Specifically, it is difficult or perhaps even impossible to define a full set of empirical consequences when initial conditions need to be taken into account, and it is infeasible to create a “pared down” version of a theory omitting only certain elements. Fritz Machlup offers a more modest form of empiricism, arguing that the basic theoretical postulates of economics cannot and should not be tested empirically, but that their consequences must be tested. He distinguishes fundamental or heuristic hypotheses from specific or factual hypotheses. The former, which include the fundamental assumptions of economics, are unverifiable and only need to meet a weak criterion of “understandabil- ity,” not realism. Instead of testing these fundamental hypotheses, we should test their consequences by considering the validity of the deduced specific hypotheses.18 In this instrumentalist argument, the accuracy of predictions then is the only relevant criterion for evaluating the fundamental hypothesis. The main problem with this argument is that

15. T.W. Hutchinson, The Signiﬁcance and Basic Postulates of Economic Theory (London: MacMillan / Co, 1938). 16. Ibid., p. 65. 17. Hausman, The Inexact and Separate Science of Economics, p. 156-158. 18. Fritz Machlup, “The Problem of Veriﬁcation in Economics,” Southern Economic Journal 22, no. 1 (1955): 1–21.

7 many of the postulates of economic theory are actually verifiable and can be observed as true or false in certain situations. It is unclear in Machlup’s model what one should do when observation contradicts the fundamental assumptions of economics that are not supposed to be questioned. Milton Friedman makes a similar instrumentalist argument in “The Methodology of Positive Economics.”19 He famously argues that because the primary goal of positive economics is to make accurate predictions, the realism of assumptions in theoretical work is irrelevant. Instead, hypotheses can only be tested empirically and falsified or corroborated. He restricts his evaluation of theories to a narrow set of phenomena, writing that a “theory is to be judged by its predictive power for the class of phenomena which it is intended to ‘explain.”’20 This makes it easier to preserve existing economic theory in spite of empirical evidence that some basic tenets do not always hold. Friedman’s perspective has been criticized from a number of angles. One of the most salient criticisms, again raised by Hausman, is that the realism of assumptions does matter when revising a falsified theory and when considering how a model fits a new situation, two essential functions in the scientific process.21

4 Application to current empirical microeconomic research

The above discussion, though necessarily brief, has set the stage for a more detailed look at two broad classifications of current empirical economic methods—“structural” and “experimental.” I will consider these classes of methods separately, attempting to define their underlying methodologies. At the end of the section, I will draw conclusions about what the differences in methodologies imply for the policy-relevance of economics. It is important to note that there is some overlap between these two broad groups of models. For example, structural models might use parameters estimated experimentally, while instrumental variable approaches actually derive from early structural work.22 Ad- ditionally, laboratory experiments in economics are conducted for a variety of purposes, including to test theory.23 Because this paper questions the ability of different methodologies in economics to contribute to policy conversation, I am particularly concerned

19. Milton Friedman, “The Methodology of Positive Economics,” in The Philosophy of Economics: An Anthology, 3rd, ed. Daniel M. Hausman (New York: Cambridge University Press, 2008), 145–178, originally published in 1953. 20. Ibid., p. 149. 21. Hausman, The Inexact and Separate Science of Economics, p. 167. 22. Rust, “Comments on ‘Structural vs. Atheoretic Approaches to Econometrics’ by Michael Keane,” p. 21. 23. Rachel Croson, “Why and How to Experiment: Methodologies from Experimental Economics,” 1 (2014).

8 with experiments that aim to contribute policy-relevant results.

Experimental economics and the “credibility revolution”

Experimental work in economics generally refers to laboratory experiments, where sub- jects are brought into a controlled space and asked to play games or make decisions. In this paper, I use the term more generally to also include field experiments, where the effect of a policy or intervention is analyzed in a real-world setting, and “quasi-experiments,” which rely on natural or institutional variation to replicate experimental settings statis- tically. In all three cases, an experiment is constructed that compares a group treated with some intervention with a control group. If there is reason to believe that these two groups are similar, like if they were assigned randomly as in a randomized control trial (RCT) or if they were developing similarly before the intervention, then the differences in their out- comes indicates the effect of the intervention. While a laboratory experiment might just difference means between the two populations, field experiments and quasi-experiments typically try to control for underlying differences between the groups and for other inter- fering trends. The basic statistical models used in these cases are difference-in-difference (DID), regression discontinuity (RD), and instrumental variable (IV) specifications. The major popularizers of experimentally-based economics are Angrist and Pischke, who have written an extremely influential companion econometrics text Mostly Harmless Econometrics24 that almost entirely avoids questions of model-building and forecasting.25 Instead, they reference a “credibility revolution” driven by a focus on research design.26 They write, “[A]pplied economists are now less likely to pin a causal interpretation of the results on econometric methodology alone. Design-based studies are distinguished by their prima facie credibility and by the attention investigators devote to both an institutional and a data-driven case for causality.”27 Instead of focusing on building models that perform well in predictive exercises, experimental work emphasizes plausible research design and the reduction of specific threats to validity like treatment assignment (RCT), exclusion restrictions (IV), group-specific trends (DID), and bunching (RD). The objec-

24. Joshua D. Angrist and Jörn-Steffen Pischke, Mostly Harmless Econometrics (Princeton: Princeton University Press, 2009). 25. Andrew Gelman, “A Statistician’s Perspective on ‘Mostly Harmless Econometrics: An Empiricist’s Companion,’ by Joshua D. Angrist and Jörn-Steffen Pischke,” Stata Journal 9, no. 2 (2009): pp. 316-318. 26. James H. Stock, “The Other Transformation in Econometric Practice: Robust Tools for Inference,” Journal of Economic Perspectives 24, no. 2 (2010): 83–94, offers an interesting alternative explanation for a credibility revolution driven by the development of robust tools for statistical inference. 27. Joshua D. Angrist and Jörn-Steffen Pischke, “The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics,” Journal of Economic Perspectives 24, no. 2 (2010): p. 5.

9 tive is not to establish an identification strategy, since this is implied by the research design, but instead to convincingly address any possible biases of the resulting estimates. These straightforward experimental methods are highly internally valid when exe- cuted properly. However, in abandoning deductive models based on universal theory, experimental work faces significant challenges to external validity. The results of experiments and quasi-experiments need to be generalized to a higher level of universality to be applicable in policy settings different than the one directly analyzed. This means that the particular experiment used needs to be perfectly representative of the population being generalized about.28 While random sampling can permit this representativeness in a finite population, economics generally aims to develop laws and tendencies that hold for open classes of individuals.29 Thus to achieve this generalizability, experimental work needs to solve the problem of induction. Section 3 of this paper reviewed the argument that inductive reasoning is unsound and offered some alternatives to both the deductive models and to standard induction. Of these models, the experimental approach is probably most similar to Hutchinson’s ultra-empiricism, where theory is disregarded in favor of testable empirical statements. This model does not circumvent the problem of induction, and we see experimentalists similarly unconcerned about induction.30 Why is this the case? To start, it is likely that many economists doing experimentally- based work already believe their work is policy-relevant. Experimental and quasi-experimental methods are often used to analyze the effect of some specific policy intervention, making the work policy-relevant in a trivial sense. If the desire to conduct policy-relevant research is merely formal, to achieve research grants, or expedient, to write many quick papers in a publish or perish professional climate, then this limited scope of relevance might be enough. Additionally, were we to conclude that no general theories of economics can hold, this limited scope would be pragmatic and realistic.31 Angrist and Pischke make a similar argument, stating that “Empirical evidence on any given causal effect is always local, derived from a particular time, place, and research design. Invocation of a superficially

28. Gadenne, “External validity and the new inductivism in experimental economics,” p. 4-5, though the whole essay is of interest. 29. Edward E Leamer, “Tantalus on the Road to Asymptopia,” Journal of Economic Perspectives 24, no. 2 (2010): 31–46, offers a comedic diatribe on the issue of finite samples. “Best to remember that no matter how far we travel, we always remain in the Land of Finite Sample, infinitely far away from Asymptopia” p. 43. 30. Francesco Guala, The Methodology of Experimental Economics (Cambridge: Cambridge University Press, 2005), pp. 201-202, presents induction as valid and desirable in the progress of scientific knowledge. 31. Francesco Guala and Luigi Mittone, “Experiments in economics: External validity and the robustness of phenomena,” Journal of Economic Methodology 12, no. 4 (2005): p. 496-497.

10 general structural framework does not make the underlying variation or setting more representative. Economic theory often suggests general principles, but extrapolation of causal effects to new settings is always speculative.”32 However, this argument does not hold when a notion of degrees of certainty is introduced. Both methods can involve some speculation, but that does not make structural approaches as speculative as experimental approaches. Finally, experimentalists might view their work as part of a larger program achieving general results by a process of accumulation and approaching larger questions by com- bining work on smaller questions that are more easily analyzed.33 However, this is not feasible without some sort of aggregation rule or some model indicating how to reconcile differences in experimental conditions between results. Therefore, the problem of generalizability is serious in experimental economics, perhaps limiting the ability of these methods to offer broadly policy-relevant results.

Structural economics and the role of assumptions

Structural economics research is more traditional and has been used in various forms since the mid-twentieth century. Throughout this discussion, I have in mind the most recent forms of structural economics work, which use large datasets, rich models, and simulation-type computer programming. Structural work uses economic theory and other assumptions to develop a model, and then tests this model on real-world data. In many cases, one dataset, or one subset of a dataset, is used to estimate the parameters of a structural model. Then the model is tested on existing data to check its fit. If there is sufficient reason to believe the model is good fit for the process being analyzed, the model can be used to make predictions on out-of-sample data. Otherwise, the model is revised. In addition to making predictions, the form of the model itself can offer insights into the mechanism by which a certain outcome might occur. The major criticism of structural work is that it involves “too many assumptions.” Structural economists often make a priori parametric assumptions about functional forms (of, say, a utility function) or about the distributions of random variables (say, that wages are log-normally distributed). By making these assumptions, they impose the structure that enables identification of unknown parameters and unobserved state variables. Struc- turalists respond to the critique of too many assumptions by arguing that these a priori

32. Angrist and Pischke, “The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics,” p. 23. 33. Keane, “Structural vs. Atheoretic Approaches to Econometrics,” pp. 23-35.

11 identifying assumptions are essential in answering interesting and complex questions. The realism of assumptions of the model is secondary to the performance of the model in validation exercises that test the model’s fit to in-sample data and its predictions for out-of-sample data.34 On a surface level, this view clearly evokes Friedman’s instrumentalist argument—the accuracy of the assumptions that are made in a structural model is irrelevant; only the ability of the model to make predictions matters. To Friedman, this ability to make predictions then falsifies or corroborates the model as a useful predictive tool. But, if economics aims to be policy-relevant, the realism of assumptions needs to be interrogated. It is important to know if a new situation, like a new policy application of a structural model, will be sufficiently similar for the predictive power of the model to hold. A Friedmanian methodology would then limit the ability of structural models to inform policy debate. However, while structural economists state they are not attempting to uncover the “true data-generating process” of some system, they do still hope that their models make sense of and give organization to the systems they represent.35 Even if imperfectly, structural models aim to understand the mechanisms behind some predicted effect and thus require at least some approximation to reality in the assumptions used to construct the model. Methodologically, then, they should also be associated with the basic HD method, especially in its Popperian form. The emphasis of this method on strong empirical testing to revise the assumed model closely aligns with the actual practice of structural economists. The inexactness of the models being tested also harks back to Hausman’s revision of Mill’s a priorism. In any form, this reliance on deductive reasoning gives structural economic work the universality required for policy-relevance.36 Thus to structural economists, the multitude of assumptions in their work is a small price to pay for generalizable results addressing complicated questions. John Rust sum- marizes this view in his commentary on structural economics, writing, “[I]t is not legiti- mate (or intelligent) to pretend that we don’t need to impose very strong assumptions to obtain interesting conclusions about causal effects and to obtain a deeper understanding of economic phenomena...structural econometricians are generally more ambitious in the types of questions and issues that they try to address, and they are willing to impose more assumptions and to formulate economic models to try to get answers.”37 By rely- ing on difficult deductive methods, structurally-based research introduces the generality

34. Keane, “Structural vs. Atheoretic Approaches to Econometrics,” pp. 15-17. 35. Ibid., p. 16. 36. Gadenne, “External validity and the new inductivism in experimental economics,” pp. 7-11. 37. Rust, “Comments on ‘Structural vs. Atheoretic Approaches to Econometrics’ by Michael Keane,” p. 24.

12 necessary for the broad external validity and policy applications desired in economics.

5 Implications for the purpose of economic sciences

These arguments imply that experimental economics, which has many benefits, may be less equipped to address broad policy questions due to the challenges of generalizability. In contrast, structural methods begin with a higher level of universality and thus avoid any problems of induction. The experimental turn in economics, which represents a shift in methods from structural to experimental, thus has an impact on the ability of the discipline to contribute research that is relevant in broad policy settings. This raises an important question worth further consideration: is this change acceptable? Economics has traditionally been a discipline with high aspirations to understand human nature and to predict and shape human decision-making. Is it acceptable for the discipline to increasingly study narrow questions that are easily approached experimentally? Are our questions informing our research, or does the availability of natural quasi-experiments direct our questions?38 Without policy-relevance, is economics in dan- ger of becoming an “elite infotainment industry”?39 My inclination is that the value of economics as a tool for understanding human nature and for contributing to substan- tive policy analysis should not be overlooked. Instead of narrowing focus, a broader and more interdisciplinary focus that achieves these goals more accurately with the help of psychology, sociology, and political science could be a step in the right direction. In any case, I think it is clear that economics should use and value all of the tools at its disposal. A strong preference for experimental over structural work disadvantages methods that can be applied in a variety of policy situations. There is no reason that the field has to swing so strongly in the experimentalist direction that the offerings of structural work fall by the wayside. Finally, the different questions experimental and structural research can ask and answer further strengthen the case for a plurality of methods.40 However, to achieve this plurality, more emphasis might need to be given to structural methods in teaching and in publications to bring them back onto equal footing with experimental methods.

38. Gelman, “A Statistician’s Perspective on ‘Mostly Harmless Econometrics: An Empiricist’s Com- panion,’ by Joshua D. Angrist and Jörn-Steﬀen Pischke,” p. 315. 39. Rust, “Comments on ‘Structural vs. Atheoretic Approaches to Econometrics’ by Michael Keane,” p. 22. 40. Aviv Nevo and Michael D. Whinston, “Taking the Dogma out of Econometrics: Structural Modeling and Credible Inference,” Journal of Economic Perspectives 24, no. 2 (2010): 69–82; Richard Blundell, “Comments on: Michael P. Keane ‘Structural vs. atheoretic approaches to econometrics’,” Journal of Econometrics 156, no. 1 (2010): 25–26, both make this argument.

13 6 Conclusion

In this paper, I have connected the methodological developments of the twentieth century to the debate between experimental and structural approaches to economics. I have argued that experimentally-based methods require unsound inductive reasoning to achieve external validity. Experimental economics is thus a weaker tool for policy analysis than structural economics, which relies on deductive methods. Deductive methods are developed to be more universal and pose fewer challenges of generalization, oﬀering research in economics the external validity it requires to be broadly policy-relevant. This raises questions about the purpose and goals of economics—is policy-relevance what economists are truly after? I conclude by suggesting that the discipline should value the very diﬀer- ent contributions of both structural and experimental economics, which would require a renewed emphasis on structural methods in teaching and in publication.

References

Angrist, Joshua D., and Jörn-Steﬀen Pischke. Mostly Harmless Econometrics. Princeton: Princeton University Press, 2009.

. “The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics.” Journal of Economic Perspectives 24, no. 2 (2010): 3–30.

Blundell, Richard. “Comments on: Michael P. Keane ‘Structural vs. atheoretic approaches to econometrics’.” Journal of Econometrics 156, no. 1 (2010): 25–26.

Croson, Rachel. “Why and How to Experiment: Methodologies from Experimental Eco- nomics.” 1 (2014).

Friedman, Milton. “The Methodology of Positive Economics.” In The Philosophy of Eco- nomics: An Anthology, 3rd, edited by Daniel M. Hausman, 145–178. New York: Cambridge University Press, 2008.

Gadenne, Volker. “External validity and the new inductivism in experimental economics.” Rationality, markets, and morals 4, no. 1957 (2013): 1–19.

Gelman, Andrew. “A Statistician’s Perspective on ‘Mostly Harmless Econometrics: An Empiricist’s Companion,’ by Joshua D. Angrist and Jörn-Steﬀen Pischke.” Stata Journal 9, no. 2 (2009): 315–320.

14 Guala, Francesco. The Methodology of Experimental Economics. Cambridge: Cambridge University Press, 2005.

Guala, Francesco, and Luigi Mittone. “Experiments in economics: External validity and the robustness of phenomena.” Journal of Economic Methodology 12, no. 4 (2005): 495–515.

Hausman, Daniel M. The Inexact and Separate Science of Economics. Cambridge: Cam- bridge University Press, 1992.

Heckman, James. “Instrumental Variables: A Study of Implicit Behavioral Assumptions Used in Making Program Evaluations.” The Journal of Human Resources 32, no. 3 (1997): 441–462.

Heckman, James, and Edward Vytlacil. “Structural Equations, Treatment Eﬀects, and Econometric Policy Evaluation.” Econometrica 73, no. 3 (2005): 669–738.

Hume, David. “An Enquiry Concerning Human Understanding.” In Enquiries Concerning Human Understanding and Concerning the Principles of Morals, 3rd, pp. 73–79. Oxford: Oxford University Press, 1989.

Hutchinson, T.W. The Signiﬁcance and Basic Postulates of Economic Theory. London: MacMillan / Co, 1938.

Keane, Michael P. “A Structural Perspective on the Experimentalist School.” Journal of Economic Perspectives 24, no. 2 (2010): 47–58.

. “Structural vs. Atheoretic Approaches to Econometrics.” Journal of Econometrics 156, no. 1 (2010): 3–20.

Leamer, Edward E. “Tantalus on the Road to Asymptopia.” Journal of Economic Per- spectives 24, no. 2 (2010): 31–46.

Machlup, Fritz. “The Problem of Veriﬁcation in Economics.” Southern Economic Journal 22, no. 1 (1955): 1–21.

Marcellesi, Alexandre. “External validity: Is there still a problem?” Philosophy of Science 82, no. 5 (2015): 1308–1317.

Mill, J.S. “On the Deﬁnition and Method of Political Economy.” In The Philosophy of Economics: An Anthology, 3rd, edited by Daniel M. Hausman, 41–58. New York: Cambridge University Press, 2008.

15 Nevo, Aviv, and Michael D. Whinston. “Taking the Dogma out of Econometrics: Struc- tural Modeling and Credible Inference.” Journal of Economic Perspectives 24, no. 2 (2010): 69–82.

Oswald, Donald J. “J.S. Mill’s ‘a priori’ Deductive Methodology: A Case Study In Post- Modern Philosophy Of Science.” Review of Social Economy 48, no. 2 (1990): 172– 197.

Robbins, Lionel. “The Nature and Signiﬁcance of Economic Science.” In The Philosophy of Economics: An Anthology, 3rd, edited by Daniel M. Hausman, 73–99. New York: Cambridge University Press, 2008.

Rosenzweig, Mark R., and Kenneth I. Wolpin. “Natural ‘Natural Experiments’ in Eco- nomics.” Journal of Economic Literature 38, no. 4 (2000): 827–874.

Rust, John. “Comments on ‘Structural vs. Atheoretic Approaches to Econometrics’ by Michael Keane.” Journal of Econometrics 156, no. 1 (2010): 21–24.

Sims, Christopher A. “But Economics Is Not an Experiemental Science.” Journal of Eco- nomic Perspectives 24, no. 2 (2010): 59–68.

Spiegler, Ran. “Behavioral Economics and the Atheoretical Style (working paper).” 2017.

Stock, James H. “The Other Transformation in Econometric Practice: Robust Tools for Inference.” Journal of Economic Perspectives 24, no. 2 (2010): 83–94.

“Economists are prone to fads, and the latest is machine learning.” The Economist, November 2016. http://www.economist.com/news/finance- and- economics/ 21710800-big-data-have-led-latest-craze-economic-research-economists -are-prone.