Economic Research Evolves: Fields and Styles

The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters.

Citation Angrist, Joshua et al. “Economic Research Evolves: Fields and Styles.” American Economic Review 107, 5 (May 2017): 293–297

As Published http://dx.doi.org/10.1257/AER.P20171117

Publisher American Economic Association

Version Final published version

Citable link http://hdl.handle.net/1721.1/113680

Terms of Use Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. American Economic Review: Papers & Proceedings 2017, 107(5): 293–297 https://doi.org/10.1257/aer.p20171117

Economic Research Evolves: Fields and Styles†

By Joshua Angrist, Pierre Azoulay, Glenn Ellison, Ryan Hill, and Susan Feng Lu*

Economic research has become more empiri- I. Classification cal, a shift documented by Hamermesh 2013 , among others. We examine this shift in( detail) Our analysis of publication and citation rates here, showing that it consists of changing research uses data from the Web of Science and Econlit styles mostly within, rather than across, fields of databases pertaining to the articles published in economics. We also gauge the extent to which the our economics journal list. shift in publication style is paralleled by a change in impact. Our analysis exploits a machine-learn- A. Fields ing-based classification of economics journal content into fields and styles, developed as part of Our field classification scheme exploits four a project analyzing citations to economics from sources of information: JEL codes; titles and other disciplines Angrist et al. 2017 . keywords; the publishing journal; and the fields The ability to( classify papers automatically) of the papers that a paper cites. We begin by lets us take a broad look at economic research: compiling a training dataset that includes a set our dataset includes 134,892 papers published of papers for which Ellison’s 2002 JEL-to- in 80 journals between 1980 and 2015.1 We 17-field mapping seems likely( to be) reliable. present unweighted analyses of the full journal The training data also include papers in a few sample and weighted analyses that emphasize “field journals” e.g., the Journal of Labor highly-cited journals. Economics assigned( to the field journal’s field. A machine-learning) algorithm trained on this dataset is used to generate an initial field classification for each paper. A clustering algorithm then uses data on each paper’s initial classifica- * Angrist: Department of Economics, MIT, 50 Memorial tion and the initial classifications of the papers it Drive, Cambridge, MA 02142, and NBER e-mail: angrist@ cites to assign a final field the online Appendix mit.edu ; Azoulay: Sloan School of Management,( 100 ( ) details all procedures used in this study . Main Street, Cambridge, MA 02142, and NBER e-mail: The clustering algorithm is instructed) to pro- [email protected] ; Ellison: Department of Economics,( MIT, 50 Memorial) Drive, Cambridge, MA 02142, and duce ten fields. Inspecting their contents, we label NBER e-mail: [email protected] ; Hill: Department of these fields microeconomics, macroeconomics, Economics,( MIT, 50 Memorial Drive,) Cambridge, MA econometrics, public finance, labor, international, 02142 e-mail: [email protected] ; Lu: Purdue University, ( ) finance, industrial organization IO , develop- 403 W. State Street, West Lafayette, IN 47907, and ment, and miscellaneous. The miscellaneous( ) field Northwestern University e-mail: [email protected] . We thank our extraordinarily efficient( and skilled research) assis- is an amalgam of several smaller fields including tants, Gina Li and Suhas Vijaykumar. Thanks also go to the economic history, environmental economics, AEA’s Liz Braunstein for help with Econlit, to Ben Jones, experimental economics, law and economics, Satyam Mukherjee, Brian Uzzi, and Heidi Williams, for help political economy, and urban economics. with the Web of Science data, and to our discussant, John Rust, for comments. † Go to https://doi.org/10.1257/aer.p20171117 to visit the B. Styles article page for additional materials and author disclosure statement s . ( ) We classify papers as belonging to one of 1 Journals in this set were among the 33 most cited by three research styles: theoretical, empirical, or the American Economic Review in any of 1968, 1978, 1988, 1998, or 2008, plus other economics journals that were com- econometrics. Papers in fields other than econo- parably well cited by the flagship journal of another disci- metrics are classified as theoretical or empirical. pline such as the American Political Science Review . We aim to label papers as “empirical” if they use ( ) 293 294 AEA PAPERS AND PROCEEDINGS MAY 2017 data to estimate economically meaningful param- eters. Papers that cover methodological issues 0.3 while also producing substantively meaningful eights 0.25 estimates were also classified as empirical. To 0.2 distinguish economic theory from econometric 0.15 theory, papers classified in the econometrics field 0.1 using the process described above were classified( as falling into a distinct econometrics) style. 0.05 Papers are also classified into styles by a Composite journal w machine-learning algorithm. Our training data- 1980 1985 1990 1995 2000 2005 2010 2015 set for this purpose contains 5,850 papers: 1,507 hand-classified for use in Ellison 2002 ; American Economic Review Econometrica ( ) Journal of Political Economy Quarterly Journal of Economics and 3,343 additional randomly selected papers Review of Economic Studies Review of Economics and Statistics hand-classified mostly by our research assis- tants who were also trained . We used this data- Figure 1. Page Rank Weighting Scheme set to( train a random forest )algorithm that takes as input article titles, journal identifiers, the Note: The figure plots five-year moving averages of compos- t ite weights for each of the top six economics journals. assigned field, JEL codes, keywords, the publi- (μ k) cation decade, and abstracts where available .2 ( ) t Our style classifications are less accurate in The importance weights, w j , reflect the extent the 1970s, so we focus on post-1980 papers to which journal j on our journal list is cited by when reporting the distribution of publica- a weighted composite( of six top journals.) This tions by style and on post-1990 citing articles imaginary “composite top journal” allows for when reporting citations to styles. Citations are changing importance within the top-six group backward-looking. ( the top six includes the usual top five plus the ) Review( of Economics and Statistics, which once C. Journal Weights rivaled the top five in importance . The first step in the construction of importance weights) detailed Different journals publish and cite different in the online Appendix applies Google’s( Page types of papers. Scholars are especially inter- Rank algorithm to the matrix) giving the fraction ested in the content of prestigious, highly-cited of each of the top six journal’s citations to the oth- t general interest journals, paying less attention ers. This produces a value, μ k , for each of the six. to other journals. We use a weighting scheme We then compute a weighted average of citation t t t t to capture this hierarchy of journal importance; fractions, w j k μ k c kj . The final w j are the t ̃ ≡ ∑ these “importance weights” are denoted w j . For five-year moving averages of this series. example, our importance-weighted measure of Figure 1 plots the smoothed time series t t t t t citations to style r is cr j w j c jr , where c jr is of composite weights μ k for each of the top the fraction of journal j≡’s year∑ t citations which six. Rising from rough( parity) with the AER, are to papers of style r. Econometrica leads during the late 1980s, The distribution of publications across fields peaking as the most important journal in the and styles is reported using both weighted and early 1990s, with declining weight through unweighted counts. Like our weighted citation 2008. The AER and the Quarterly Journal of measure, the weighting scheme for publications Economics grow in importance from about 1990 produces a measure of output that emphasizes on. These trends—for example, the rise and fall papers that appear in top journals. For example, of Econometrica— should be kept in mind when the weighted measure of publications in fieldf is interpreting weighted results. s t w t s t , where s t is the fraction of journal f ≡ ∑j j jf jf II. Economic Research Evolves j’s publications which are in fieldf . Descriptive statistics in the online Appendix show that fields interact with styles. Papers in the microeconomics field are mostly though not ( 2 Abstracts are unavailable prior to 1986. entirely classified as theoretical, while papers in ) VOL. 107 NO. 5 Economic Research Evolves 295

0.25 0.18 0.16 0.2 0.14 0.15 0.12 articles articles 0.1 0.1 0.08 0.06

Share of 0.05

Share of 0.04 0.02 0 0 1980 1985 1990 1995 2000 2005 2010 2015 1980 1985 1990 1995 2000 2005 2010 2015 Publication year Publication year

Micro Metrics Macro PF Micro Metrics Macro PF International Labor Finance IO International Labor Finance IO Development Misc Development Misc

Figure 2. Publication Shares by Field Figure 3. Weighted Publication Shares by Field

Note: The figure shows five-year moving averages of Note: The figure shows five-year moving averages of unweighted field shares. weighted publication shares. the applied micro fields of labor, development, nomics, and political economy, fields that were and public finance are mostly empirical. The col- once perhaps on the sidelines. lection of smaller fields grouped under the miscel- The empirical shift in economic scholarship is laneous heading are nearly two-thirds empirical. a within-field phenomenon, a pattern documented Figure 2 traces the evolution of economics in Figure 4, which plots the weighted propor- journal output by field, showing the unweighted tion of publications in each field classified as fraction of papers in each field among those empirical. In the early 1980s, development and published in the journals on our list between labor were the only fields in which the majority 1980–2015. Perhaps surprisingly, this figure of weighted publications were empirical. The shows the microeconomics field growing strongly weighted empirical share has since grown in all since the mid 1980s, to the point where micro is fields, now exceeding 90 percent in labor and now the largest field, bypassing macroeconomics development. International and public finance are in the mid-2000s. Microeconomics’ increasing also now majority empirical. Even macroeconom- publication share reflects both a proliferation of ics, criticized in the wake of the Great Recession theory journals and their increasing length.3 In for an excess of ivory-tower theorizing, has seen contrast, the publication shares for labor and IO its empirical share grow by over 50 percent. In have both declined since the mid-late 1980s. most fields, these trends reflect both increasing Which fields have the more influential jour- numbers of empirical papers and the improved nals been publishing? Figure 3 plots weighted journal placement of empirical work excepting field shares. Microeconomics has the largest IO, whose unweighted empirical share (is flat . share throughout, while macro also maintains a Changes in the overall empirical share )also high share. The largest “applied micro” fields, reflect within-field more than cross-field trends. labor, IO, and public finance, havedeclining Figure 5 plots unweighted publication style shares. weighted shares in the early years and no recent Within-field shifts are muted somewhat by strong growth. In contrast with Figure 2, however, the growth in the mostly theoretical microeconomics importance-weighted statistics show substan- field. Even so, the share of economics publications tial growth in our “miscellaneous” field. This devoted to empirical work, which held steady at amalgamated category includes environmental about 50 percent from 1980 to 1995, has since economics, experimental economics, urban eco- increased to a little over 60 percent. With the share devoted to econometrics essentially unchanged, this increase came out of the theoretical share. 3 For example, Games and Economic Behavior started in 1989 and Economic Theory in 1991. These two published The weighted distribution of publications by 145 and 73 papers in 2014. Journal of Economic Theory has style in Figure 6 shows a more dramatic rise grown from 64 papers in 1980 to 130 in 2014. in empirical work. In the early-mid 1980s, the 296 AEA PAPERS AND PROCEEDINGS MAY 2017

1 1 0.9 0.9 0.8 0.8 0.7 0.7 0.6 0.5 0.6 0.4 0.5 0.3 0.4

Fraction empirical 0.2 0.3

0.1 Share of articles 0.2 0 0.1 1980 1985 1990 1995 2000 2005 2010 2015 0 Publication year 1980 1985 1990 1995 2000 2005 2010 2015 Publication year Micro Macro PF International Labor Finance IO Development Empirical Theoretical Econometrics Misc

Figure 4. Weighted Fraction Empirical by Field Figure 5. Publications by Style

Note: Five-year moving averages of the weighted fraction of Note: Five-year moving averages of unweighted publication publications in each field that are empirical. shares in each style.

1 1 0.9 0.9 0.8 0.8 0.7 0.7 0.6 0.6 articles 0.5 0.5 0.4 0.4 0.3 0.3 Citation share Share of 0.2 0.2 0.1 0.1 0 0 1980 1985 1990 1995 2000 2005 2010 2015 1990 1995 2000 2005 2010 2015 Publication year Citing year

Empirical Theoretical Econometrics Empirical Theoretical Econometrics

Figure 6. Weighted Publications by Style Figure 7. Weighted Citations by Style

Note: Five-year moving averages of weighted publication Note: Five-year moving averages of weighted citation shares shares in each style. in each style. weighted empirical share was only around one- citations going to econometrics, empirical work third. This lower starting point reflects the fact is now cited more often than theoretical work. that empirical papers were once disproportion- Increased citations to empirical work naturally ately found in less-cited journals. The weighted reflect the fact that more cited papers are empir- empirical share has increased steadily since ical. But this change also reflects movement of around 1985, and now exceeds 55 percent. empirical work into better, more-cited journals. The shift in citation shares toward empirical work is even stronger than the publication shift. III. Citations Per Paper Figure 7, which reports the weighted distribution of styles of cited papers for citations made in An analysis of citations per paper highlights 1990–2015, shows empirical work garnering less the different dimensions of increasing empirical than 30 percent of weighted citations in 1990. At impact. For each paper i published in year t i , the time, about half of recent publications were( we model the conditional mean of weighted cita( )- empirical. The empirical citation share is now tions to this paper ci as an exponential function almost 50) percent. With roughly 10 percent of of style dummies (Em) p and Met , a vector X of ( i i) i VOL. 107 NO. 5 Economic Research Evolves 297 article-level covariates, and a battery of year-spe- Panel A. Controls for length and number of authors cific field and journal indicators, indexed byf i and j i . Baseline controls include a cubic poly( )- 0.5 nomial( ) for article page length and dummies for the number of authors. The model is 0.0

0.5 E c X , Emp , Met, f i , j i , t i − [ i | i i i ( ) ( ) ( )] 1.0 t t t t t − exp 1 Empi 2 Meti 3 Xi j i f i . = [β + β + β + δ ( ) + γ ( )] Empirical style effect 1.5 − Because many papers are never cited and the 1980 1985 1990 1995 2000 2005 2010 2015 citation distribution is highly skewed, an expo- Publication year nential model fits the conditional mean function Panel B. Including eld and journal controls of interest better than a linear model 37 percent of the papers are never cited by other( papers 0.5 in the sample . The coefficient t captures a 1 0.0 time-varying )covariate-adjusted βlog ratio of empirical to theoretical citations per paper. 0.5 Theoretical work published in the 1980s and −

1990s was cited far more often than empirical 1.0 work of the same period. This can be seen in − Empirical style effect panel A of Figure 8, which plots the time series 1.5 t − of estimates of 1 from a model omitting field 1980 1985 1990 1995 2000 2005 2010 2015 and journal effects.β Relative citation rates to Publication year empirical work grew steadily starting in the late Figure 8. Empirical Theory Citations per Publication 1980s, but only around the year 2000 did cita- / tion rates for empirical papers reach parity with Notes: Poisson regression estimates of the empirical effect citation rates for theoretical work. on weighted citations per paper. Panel A estimates are from The estimates of a model with field and jour- models estimated separately by year, with flexible controls nal controls reported in panel B of Figure 8 show for paper length and number of authors. Estimates in panel B that much of the theoretical citation advantage add field and journal controls. Confidence bands use robust standard errors. can be attributed to differences in the distribution of papers across fields and journals. Controlling largest field, while some applied micro fields for field and journal dummies—that is, looking have shrunk. But more empirical papers are being within fields and journals—the empirical cita- published and they are appearing in more influ- tion deficit shrinks to less than 50 percent in the ential journals. Citations to empirical work have early 1980s and disappears in the late 1980s. grown even more than empirical output, although After 2000, empirical papers are cited more than the empirical share of citations is just now reach- theoretical work in the same field that was pub- ing 50 percent. lished in the same journal and year. The increasing attention to empirical work therefore reflects References factors beyond improved journal placement or persistent field-specific citation norms. Angrist, Joshua, Pierre Azoulay, Glenn Ellison, and Susan Feng Lu. 2017. “Inside Job or Deep IV. Summary Impact? Assessing the Influence of Economics Scholarship.” Unpublished. Using machine-learning methods to classify Ellison, Glenn. 2002. “The Slowdown of the Eco- economics papers into fields and styles, we doc- nomics Publishing Process.” Journal of Politi- ument major shifts in research output and the cal Economy 110 5 : 947–93. types of papers referenced. The growth in empir- Hamermesh, Daniel S.( 2013.) “Six Decades of Top ical work reflects a substantial shift within rather Economics Publishing: Who and How?” Jour- than across fields. Microeconomics remains the nal of Economic Literature 51 1 : 162–72. ( )