A Surveybased Journal Ranking by Agricultural Economists
Total Page:16
File Type:pdf, Size:1020Kb
Journal of Agricultural Economics, Vol. 62, No. 3, 2011, 710–732 doi: 10.1111/j.1477-9552.2011.00308.x Going Beyond Impact Factors: A Survey-based Journal Ranking by Agricultural Economists Roland Herrmann, Ernst Berg, Stephan Dabbert, Siegfried Po¨chtrager and Klaus Salhofer1 (Original submitted March 2010, revision received August 2010, accepted April 2011.) Abstract A consistent and comprehensive ranking of journals relevant for agricultural economists cannot rely on impact factors for at least two major reasons: (i) the scientific database by Thomson Reuters, on which the standard impact factor is based, includes only a very limited number of relevant journals; (ii) the standard impact factor cannot be compared across research fields of different sizes. Survey-based journal rankings may overcome these problems. We report on such a survey-based ranking initiated by the Agricultural Economics Associations of Germany and Austria. Results of the ranking and a classification of journals, i.e. a rating, are provided for 160 selected journals. Scientific quality is assessed by an index based on the researchers’ perception of the quality standards of each journal and of the quality of its published articles. The survey-based ranking allows a much more comprehensive and consistent ranking than the impact factor, as specific agricultural economics journals can be directly compared with neighbouring economic and interdisciplinary journals to which agricultural econo- mists submit their work. The low impact factors of core agricultural economics journals are put into perspective. The scientific quality of the top agricultural economics journals is assessed as being rather high and above most of the 1 Roland Herrmann is with the Institute of Agricultural Policy and Market Research, Univer- sity of Giessen, Germany. E-mail: [email protected] for correspon- dence. Ernst Berg is with the Institute for Food and Resource Economics, University of Bonn, Germany. Stephan Dabbert is with the Department of Farm Management, University of Hohenheim, Germany. Siegfried Po¨chtrager is with the Institute for Marketing and Inno- vation, BOKU, Vienna, Austria. Klaus Salhofer is with the Enviromental and Agricultural Policy Group, Technische Universitaet, Muenchen, Germany. We thank David Harvey and three anonymous referees for their very helpful and detailed comments and suggestions. Thanks are due to the Gesellschaft fu¨r Wirtschafts- und Sozialwissenschaften des Landbaues e. V. (GEWISOLA) and O¨sterreichische Gesellschaft fu¨r Agraro¨konomie (O¨GA) for financial support and members of both associations for their participation in the survey. Very helpful research assistance by Marco Huigen, who edited the data in a data bank, and by Matthias Staudigel and Sascha Weber in the statistical analysis is greatly appreciated. Ó 2011 The Agricultural Economics Society. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK, and 350 Main Street, Malden, MA 02148, USA. Ranking of Journals Relevant for Agricultural Economists 711 relevant interdisciplinary journals from agricultural and food sciences that are typically characterised by higher impact factors. Agricultural economists’ percep- tions on the scientific quality of the journals vary more across journals than perceptions of their relevance. Keywords: Agricultural economics; Germany and Austria; impact factor; journal ranking; survey-based evaluation. JEL classifications: A11, A12, D19. 1. Introduction Evaluations of scientific performance have become widespread and more frequent, and the results of such evaluations are often crucial for the allocation of human and other resources within the research system. In this context, publications (journal articles, monographs, book chapters, etc.) are certainly the single most important output, particularly in the social sciences. A critical appraisal of this type of output requires the measurement of both quantity and quality, where the latter refers to the intellectual influence of the publications. Ideally, a careful and comprehensive quality assessment should be based on the contents and methodology of the work an individual or an institution has published. However, this procedure is time- consuming and complex and is not practical in all cases. Given that journal review systems aim to assure a certain quality standard, journal quality could serve as a proxy judgement of the quality of articles published in it. This leads to journal rankings that have gained notable interest. In the scientific community and within universities and research institutions, publications in high-quality peer-reviewed journals are seen today as the most important dimension of the overall output of researchers, institutes or faculties. Assessing the scientific quality of journals can be based on expert judgements obtained from primary surveys, on bibliometric analyses of secondary databases or on a combination of both (Jokic and Ball, 2006). In the majority of cases, the bibliometric approach is used, based typically on citation analysis (Moed, 2005) and assuming that the number of citations is positively correlated with the quality of the cited articles. The standard impact factor (Garfield, 1972, 1994) is the most widespread among the indices of citation analysis. Its calculation relies on citation databases established and maintained originally by the International Scientific Institute (ISI) and now by Thomson Reuters. Impact factors are computed regularly from the Thomson Scientific database and reported annually in the company’s Journal Citation Report (JCR). The pros and cons of the impact factor have been widely debated. In the disci- plines of biology, medical sciences, chemistry and physics, which all enjoy excellent coverage of journals in the ISI database (Moed, 2005, p. 138), the impact factor has been the most important indicator for many years, and is often viewed as ‘a reason- able indicator of quality’ if used for a comparison of scientific journals within one of these disciplines (Saha et al., 2003). Some publications are much more critical, however, and stress conceptual weaknesses (Jemec, 2001), and some even recom- mend that the impact factor of journals should not be used for evaluating research (Seglen, 1997). In particular, journal impact factors depend on a number of deter- minants that are unrelated to journal quality. The following drawbacks of the Ó 2011 The Agricultural Economics Society. 712 Roland Herrmann et al. impact factor are the most pronounced: (i) impact factors differ widely according to the size of a research field (Althouse et al., 2009); the larger the number of journals included in the JCR database, the higher the impact factors of top journals in that field; (ii) impact factors tend to be higher in rapidly growing fields, where the turnover time between journal submission and publication is short, and references are only short-lived. Both arguments imply that impact factors should not be com- pared across disciplines in fields of study, because such comparisons would yield wide differences across those fields (Althouse et al., 2009) with, for example, lower impact factors in agricultural economics than in biotechnology or microbiology. One option to circumvent the disadvantages of the standard impact factor in assessing journal quality is to improve citation analysis and to derive superior indices. Examples are weighted impact factors as suggested by some authors (Jemec, 2001; Habibzadeh and Yadollahie, 2008), the h-index and its variants (http://sci2s. ugr.es/hindex/), the eigenfactor (http://www.eigenfactor.org/) and advanced statisti- cal indicators developed in the general economics literature on journal quality (e.g. Ritzberger, 2008). With regard to journals that are relevant for agricultural economists, however, it is more difficult to apply sophisticated citation analyses using these improved methods. Although general economics has a ‘good’ coverage in the JCR database (Moed, 2005, p. 138), agricultural economics has not. An impact factor was provided by Thomson Reuters for only nine journals in the cate- gory ‘Agricultural Economics and Policy’ up to 2008 and 11 as from 2009.2 Other well-established or new agricultural economics journals are not covered, such as the Journal of Agricultural and Food Industrial Organization, the Journal of Wine Economics and a number of European agricultural economics journals including those published in German or French. This very limited coverage strongly reduces the value of the impact factor for assessing agricultural economics journals, espe- cially for non-English speakers. The few specific citation analyses for agricultural economics also concentrate on only 12 (Barrett et al., 2000) or seven (Burton and Phimister, 1996) agricultural economics journals. Interdisciplinary ties are also much stronger for agricultural economics than for general economics (Zapata, 2009). Agricultural economists may publish in their own, in economics or in interdisciplin- ary journals of agricultural, food and environmental sciences. Though quite a number of these interdisciplinary journals have impact factors, these may not be comparable with those of agricultural economics journals, as argued above. What we are lacking is a comprehensive and consistent journal ranking that covers all areas where agricultural economists publish their research. For this reason, the Agricultural Economics Associations of Germany and Austria3 attempted to establish a journal ranking based on expert assessment