A Surveybased Journal Ranking by Agricultural Economists

Journal of Agricultural Economics, Vol. 62, No. 3, 2011, 710–732 doi: 10.1111/j.1477-9552.2011.00308.x

Going Beyond Impact Factors: A Survey-based Journal Ranking by Agricultural Economists

Roland Herrmann, Ernst Berg, Stephan Dabbert, Siegfried Po¨chtrager and Klaus Salhofer1

(Original submitted March 2010, revision received August 2010, accepted April 2011.)

Abstract A consistent and comprehensive ranking of journals relevant for agricultural economists cannot rely on impact factors for at least two major reasons: (i) the scientific database by Thomson Reuters, on which the standard impact factor is based, includes only a very limited number of relevant journals; (ii) the standard impact factor cannot be compared across research fields of different sizes. Survey-based journal rankings may overcome these problems. We report on such a survey-based ranking initiated by the Agricultural Economics Associations of Germany and Austria. Results of the ranking and a classification of journals, i.e. a rating, are provided for 160 selected journals. Scientific quality is assessed by an index based on the researchers’ perception of the quality standards of each journal and of the quality of its published articles. The survey-based ranking allows a much more comprehensive and consistent ranking than the impact factor, as specific agricultural economics journals can be directly compared with neighbouring economic and interdisciplinary journals to which agricultural economists submit their work. The low impact factors of core agricultural economics journals are put into perspective. The scientific quality of the top agricultural economics journals is assessed as being rather high and above most of the

1 Roland Herrmann is with the Institute of Agricultural Policy and Market Research, Univer- sity of Giessen, Germany. E-mail: [email protected] for correspon- dence. Ernst Berg is with the Institute for Food and Resource Economics, University of Bonn, Germany. Stephan Dabbert is with the Department of Farm Management, University of Hohenheim, Germany. Siegfried Po¨chtrager is with the Institute for Marketing and Inno- vation, BOKU, Vienna, Austria. Klaus Salhofer is with the Enviromental and Agricultural Policy Group, Technische Universitaet, Muenchen, Germany. We thank David Harvey and three anonymous referees for their very helpful and detailed comments and suggestions. Thanks are due to the Gesellschaft fu¨r Wirtschafts- und Sozialwissenschaften des Landbaues e. V. (GEWISOLA) and O¨sterreichische Gesellschaft fu¨r Agraro¨konomie (O¨GA) for ﬁnancial support and members of both associations for their participation in the survey. Very helpful research assistance by Marco Huigen, who edited the data in a data bank, and by Matthias Staudigel and Sascha Weber in the statistical analysis is greatly appreciated.

2011 The Agricultural Economics Society. Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK, and 350 Main Street, Malden, MA 02148, USA. Ranking of Journals Relevant for Agricultural Economists 711

relevant interdisciplinary journals from agricultural and food sciences that are typically characterised by higher impact factors. Agricultural economists’ perceptions on the scientiﬁc quality of the journals vary more across journals than perceptions of their relevance.

Keywords: Agricultural economics; Germany and Austria; impact factor; journal ranking; survey-based evaluation. JEL classiﬁcations: A11, A12, D19.

1. Introduction Evaluations of scientific performance have become widespread and more frequent, and the results of such evaluations are often crucial for the allocation of human and other resources within the research system. In this context, publications (journal articles, monographs, book chapters, etc.) are certainly the single most important output, particularly in the social sciences. A critical appraisal of this type of output requires the measurement of both quantity and quality, where the latter refers to the intellectual influence of the publications. Ideally, a careful and comprehensive quality assessment should be based on the contents and methodology of the work an individual or an institution has published. However, this procedure is time- consuming and complex and is not practical in all cases. Given that journal review systems aim to assure a certain quality standard, journal quality could serve as a proxy judgement of the quality of articles published in it. This leads to journal rankings that have gained notable interest. In the scientific community and within universities and research institutions, publications in high-quality peer-reviewed journals are seen today as the most important dimension of the overall output of researchers, institutes or faculties. Assessing the scientific quality of journals can be based on expert judgements obtained from primary surveys, on bibliometric analyses of secondary databases or on a combination of both (Jokic and Ball, 2006). In the majority of cases, the bibliometric approach is used, based typically on citation analysis (Moed, 2005) and assuming that the number of citations is positively correlated with the quality of the cited articles. The standard impact factor (Garfield, 1972, 1994) is the most widespread among the indices of citation analysis. Its calculation relies on citation databases established and maintained originally by the International Scientific Institute (ISI) and now by Thomson Reuters. Impact factors are computed regularly from the Thomson Scientific database and reported annually in the company’s Journal Citation Report (JCR). The pros and cons of the impact factor have been widely debated. In the disciplines of biology, medical sciences, chemistry and physics, which all enjoy excellent coverage of journals in the ISI database (Moed, 2005, p. 138), the impact factor has been the most important indicator for many years, and is often viewed as ‘a reason- able indicator of quality’ if used for a comparison of scientific journals within one of these disciplines (Saha et al., 2003). Some publications are much more critical, however, and stress conceptual weaknesses (Jemec, 2001), and some even recom- mend that the impact factor of journals should not be used for evaluating research (Seglen, 1997). In particular, journal impact factors depend on a number of deter- minants that are unrelated to journal quality. The following drawbacks of the

2011 The Agricultural Economics Society. 712 Roland Herrmann et al. impact factor are the most pronounced: (i) impact factors differ widely according to the size of a research field (Althouse et al., 2009); the larger the number of journals included in the JCR database, the higher the impact factors of top journals in that field; (ii) impact factors tend to be higher in rapidly growing fields, where the turnover time between journal submission and publication is short, and references are only short-lived. Both arguments imply that impact factors should not be compared across disciplines in fields of study, because such comparisons would yield wide differences across those fields (Althouse et al., 2009) with, for example, lower impact factors in agricultural economics than in biotechnology or microbiology. One option to circumvent the disadvantages of the standard impact factor in assessing journal quality is to improve citation analysis and to derive superior indices. Examples are weighted impact factors as suggested by some authors (Jemec, 2001; Habibzadeh and Yadollahie, 2008), the h-index and its variants (http://sci2s. ugr.es/hindex/), the eigenfactor (http://www.eigenfactor.org/) and advanced statistical indicators developed in the general economics literature on journal quality (e.g. Ritzberger, 2008). With regard to journals that are relevant for agricultural economists, however, it is more difficult to apply sophisticated citation analyses using these improved methods. Although general economics has a ‘good’ coverage in the JCR database (Moed, 2005, p. 138), agricultural economics has not. An impact factor was provided by Thomson Reuters for only nine journals in the category ‘Agricultural Economics and Policy’ up to 2008 and 11 as from 2009.2 Other well-established or new agricultural economics journals are not covered, such as the Journal of Agricultural and Food Industrial Organization, the Journal of Wine Economics and a number of European agricultural economics journals including those published in German or French. This very limited coverage strongly reduces the value of the impact factor for assessing agricultural economics journals, especially for non-English speakers. The few specific citation analyses for agricultural economics also concentrate on only 12 (Barrett et al., 2000) or seven (Burton and Phimister, 1996) agricultural economics journals. Interdisciplinary ties are also much stronger for agricultural economics than for general economics (Zapata, 2009). Agricultural economists may publish in their own, in economics or in interdisciplinary journals of agricultural, food and environmental sciences. Though quite a number of these interdisciplinary journals have impact factors, these may not be comparable with those of agricultural economics journals, as argued above. What we are lacking is a comprehensive and consistent journal ranking that covers all areas where agricultural economists publish their research. For this reason, the Agricultural Economics Associations of Germany and Austria3 attempted to establish a journal ranking based on expert assessment of a broad

2 Up to 2008: Agricultural Economics, American Journal of Agricultural Economics, Australian Journal of Agricultural and Resource Economics, Canadian Journal of Agricultural Economics, European Review of Agricultural Economics, Food Policy, Journal of Agricultural and Resource Economics, Journal of Agricultural Economics, Review of Agricultural Economics. As from 2009, also: Agricultural Economics – Zemeˇdeˇlska Ekonomika and ITEA – Informacio´nTe´cnica Econo´mica Agraria. The journal Agribusiness has been included since 2009 in the category ‘Food Science and Technology’. 3 Gesellschaft fu¨r Wirtschafts- und Sozialwissenschaften des Landbaues e.V. (GEWISOLA) and O¨sterreichische Gesellschaft fu¨r Agraro¨konomie (O¨GA).

2011 The Agricultural Economics Society. Ranking of Journals Relevant for Agricultural Economists 713 selection of relevant journals. This article presents the methodology and major results of this appraisal. The focus was on two main questions: (i) What are agricultural economists’ perceptions of the scientiﬁc quality of their journals? (ii) What constitutes a high-quality journal for agricultural economists? A comprehensive list of journals relevant to their work is utilised. Then the perceptions of the experts, most of whom are members of the German and Austrian Associations of Agricul- tural Economists, are measured in a way that makes it possible to distinguish two dimensions of scientiﬁc quality.

2. Rankings by Citation Analysis vs. Survey-based Rankings In this section, we brieﬂy summarise the tools of bibliometric analysis in comparison with expert judgement in evaluating journal quality and comment on the strengths and weaknesses of each approach, as well as hybrid methods.

2.1. Bibliometric analyses Bibliometric analyses are largely based on citation indices. The basic version of the impact factor is deﬁned as

It ¼ðCt1 þ Ct2Þ=ðAt1 þ At2Þ; ð1Þ where It is the impact factor of a scientific journal in year t, Ct)1 (Ct)2) are the number of citations in year t referring to articles in that journal of year t ) 1 (t ) 2). At)1 (At)2) stands for the number of articles published in that particular journal in year t ) 1(t ) 2). Essentially, the impact factor measures the average number of citations for the ‘average article’ in a journal within a defined period. Journals in general economics have been ranked by means of citation analysis by Liebowitz and Palmer (1984), Laband and Piette (1994), Kalaitzidakis et al. (2003), Schlinghoff and Backes-Gellner (2002) and others. Citation-based rankings have also been developed for subdisciplines in applied economics (Barrett et al., 2000) and for individual subdisciplines such as marketing (Baumgartner and Pieters, 2003) or economic history (Di Vaio and Weisdorf, 2010). In recent years, substantial effort has been put into improvements in methodology when developing new citation-based rankings. In an influential paper, Palacios-Huerta and Volij (2004) pro- pose measuring intellectual influence using an axiomatic approach based on the properties of invariance to reference intensity, weak homogeneity and weak consistency, and invariance to splitting of journals. More recent rankings of economics journals on the basis of the invariant method include those of Kodrzycki and Yu (2006) and Ritzberger (2007, 2008).

2.2. Expert judgement The quality of journals can also be assessed by having a number of experts giving them a ranking according to a predeﬁned grading scheme and subsequently aggre- gating their rankings. Examples from the ﬁeld of business management are the ranking lists provided by the German Academic Association for Business Research (VHB) (Hennig-Thurau et al., 2004 and Schrader and Hennig-Thurau, 2009) and the University of Economics and Business Vienna (WU). There is one survey- based analysis in general economics (Axarloglou and Theoharakis, 2003), where the

2011 The Agricultural Economics Society. 714 Roland Herrmann et al. diversity of journal quality perceptions among economists was explored, and one in agricultural economics (Lusk and Hudson, 2009), where authors’ revealed preferences for agricultural economics journals were investigated. This method allows a more precise deﬁnition of what is meant by quality and is not limited by the set of journals contained in published citation databases. Finally, a survey can span journals from several disciplines, as far as they are relevant for the respondents’ research. On the other hand, the validity of the results depends on the competence of the respondents and on the sample coverage. Also, responses are susceptible to strategic behaviour. Experts may prefer those journals that belong to their narrower ﬁeld of research or journals in which they have published or those for which they have served as reviewers. Furthermore, they may not know equally well all the journals they judge.

2.3. Hybrid methods Hybrid methods combine bibliometric analyses with expert judgements, e.g. by developing an initial ranking based on citation indices and adjusting or supplement- ing this by expert judgement. In this way, one can, for instance, incorporate journals that are not originally contained in the citation database by comparative evaluation. Furthermore, it allows for the correction of possible biases in the citation index. The 2008 journal ranking by the German Economic Association (GEA) is based on a hybrid approach (Schneider and Ursprung, 2008). Hybrid methods can also be used for pooling different rankings to obtain a kind of meta-ranking (Schulze et al., 2008).

3. The Survey-based Ranking 3.1. Methodology Following Hennig-Thurau et al. (2004), we captured two dimensions of scientific quality for each journal: (i) the scientific requirements for submitted articles in the review process as defined by the editorial board and ⁄ or the reviewers, i.e. the quality standard (S), and (ii) the scientific quality (Q) achieved in the articles actually published in the journal. Respondents were asked to assess both dimensions according to their experience over the period 2000–2007. It was a criterion for a valid response on S that the respective respondent had submitted articles to or had been a reviewer for the journal in question during this period. In applying this procedure, we posited that those researchers who had submitted an article to a journal or had acted as a reviewer were able to evaluate the quality standard of that journal. In addition, we expected that those who read a journal regularly were capable of judg- ing the scientific quality of articles in a journal, as were those who acted as referees or had submitted one or more papers. We posit that S and Q are of equal importance. In the quality index, the average perceptions of S and Q are utilised, and equal weights are then assigned to both quality dimensions. The number of respondents who have assessed the standards (quality of articles) for journal z is given by nz (mz). Then, Sz and Qz represent the average assessments with regard to the two quality dimensions assigned to journal z and are computed as:

2011 The Agricultural Economics Society. Ranking of Journals Relevant for Agricultural Economists 715

1 Xnz S ¼ Sz ð2Þ z n i z i¼1 and 1 Xmz Q ¼ Qz: ð3Þ z m i z j¼1 Using the average assessments for constructing the index avoids assigning system- atically higher weights to the quality of articles, as all journals have considerably more readers than reviewers or submitters, i.e. mz always exceeds nz. Considering that the average valuation of a journal’s standards will become less reliable with smaller sample sizes, only those journals for which at least 10 assessments were available ((nz + mz) ‡ 10) were included in the ranking. A further condition was that at least one respondent had submitted an article to the journal in question.4 To account for the declining reliability of the mean assignment for S if nz becomes small, we reduced the weight of this quality dimension for sample sizes below 10. Therefore, our quality index for journal z (QIz) is calculated as the weighted average between the mean assessments of the two quality dimensions for journal z: QIz ¼ azSz þð1 azÞQz ð4Þ with 0:5 for nz>10; az ¼ ð5Þ 0:05 nz for 0 nz 10: S and Q are given values between 1 (very low) and 10 (very high), and the same applies to Sz, Qz and QIz.

3.2. Results and ranking A total of 270 journals were included in the analysis.5 The survey was carried out through an online questionnaire sent to individuals from Germany, Austria and Switzerland who have conducted research in the ﬁeld of agricultural economics. After an initial letter to 684 addresses, the mailing list was carefully revised with regard to the objectives of the study and 570 persons were then invited by e-mail to participate in the online survey. A total of 397 responses were received, of which 305 responses contained all the necessary data, i.e. a response rate of 54%, which is very high for an online survey.

4 The latter condition may be rather restrictive as some of the leading general economics journals may be read by agricultural economists, but not necessarily regarded as a market for their own research. In our survey, this led to the exclusion of Econometrica, Journal of Economic Literature and Quarterly Journal of Economics. All these journals received more than 50 assessments, would have received a mean Q value of 8.00 and more, but no submis- sions to these journals took place. 5 For the preselection of journals and the sample of agricultural economists, for the questionnaire and many details of the data collection, see the Background Report by Dabbert et al. (2009).

2011 The Agricultural Economics Society. 716 Roland Herrmann et al.

On average, the respondents read 25 journals regularly. However, there was considerable variation. A total of 224 respondents (72%) had submitted articles them- selves to one or more of the journals they evaluated, and 152 (49%) had experience as reviewers for these journals. On average, manuscripts had been submitted to six different journals, and those who had reviewed articles did so for 4.4 journals, again with considerable variation. The average number of assessments per respondent was 6.7 regarding a journal’s standards and 23.8 assessments with regard to its scientific quality. Table 1 presents the calculated quality index and the number of assessments for those 160 journals that meet the criteria outlined in section 3.1. Overall, the top ten journals indicate a strong appreciation of general economics journals by agricultural economists. Seven of the top ten are economics journals (the American Eco- nomic Review is ranked highest with a quality index of 8.95), one is a marketing journal and two are core journals of agricultural economics – the American Journal of Agricultural Economics (AJAE) and the European Review of Agricultural Econom- ics (ERAE). With index values of 8.29 and 7.79, respectively, these two are well above all other core agricultural economics journals. The top ten of agricultural economics journals are completed by the Australian Journal of Agricultural and Resource Economics (AJARE) (7.28), Agricultural Economics (7.16), Journal of Agri- cultural Economics (JAE) (7.07), Food Policy (7.01), Journal of Agricultural and Resource Economics (JARE) (6.61), Canadian Journal of Agricultural Economics (CJAE) (6.49), Agricultural Finance Review (AFR) (6.41) and Journal of Agricultural and Food Industrial Organization (JAFIO) (6.41). Other journals that were ranked relatively high focus on the economics of rural development or land use, such as the Journal of Rural Studies (with an index value of 6.98) and Land Economics (6.90) or on rural sociology, such as Sociologia Ruralis (7.64) and Rural Sociology (6.48). Some interdisciplinary journals in which agricultural economists occasionally publish were comparable as well: Agronomy Journal with a quality index of 7.29, Agriculture, Ecosystems and Environment (7.26), the Journal of Chain and Network Science (6.86), Agricultural Systems (6.80), the Jour- nal of Land Use Science (6.65) and Agricultural and Human Values (6.52). Several journals of environmental economics or environmental science also ranged between 7.4 and 6.4. The ranking in Table 1 allows a comprehensive comparison of the quality of journals relevant for agricultural economics and its neighbouring fields. Eight of the core journals in agricultural economics, which have an impact factor, are among the top ten agricultural economics journals in our ranking. However, our ranking is more comprehensive than the JCR, as we aimed to capture the whole journal market for which the respondents produce articles. The magnitude of impact factors and the order of core agricultural economics journals have been rather volatile over time. In the period 2004–2009, the impact factor of the American Journal of Agricultural Economics varied between 0.622 (2004) and 1.196 (2006) and its ranking in the subject categories between fourth (2008 and 2009) and first (2006). The impact factor of the European Review of Agri- cultural Economics ranged between 0.681 (2006) and 1.271 (2007) and the journal’s ranking moved between fifth (2009) and first (2007, 2005 and 2004). The impact factor of Food Policy went up from 0.532 (2004) to 1.606 (2008) and the journal’s ranking from fifth (2005) to first (2008 and 2009). In contrast, survey-based rankings seem to be much more stable – especially for the top journals – as researchers’ perceptions of the quality of journals do not change much from one year’s issue to

2011 The Agricultural Economics Society. Ranking of Journals Relevant for Agricultural Economists 717

Table 1 Ranking and rating of 160 journals, ordered by the quality index

Number of Quality Rank no. Name of the journal assessments index Rating

1 American Economic Review 92 8.95 A+ 2 Journal of Econometrics 31 8.48 A+ 3 The Economic Journal 41 8.36 A+ 4 American Journal of Agricultural Economics 237 8.29 A+ 5 Journal of Economic Dynamics and Control 18 8.26 A+ 6 Review of Economics and Statistics 34 8.25 A+ 7 Journal of Applied Econometrics 28 7.96 A 8 Journal of Economic Behaviour and Organisation 42 7.82 A 9 Marketing Science 15 7.81 A 10 European Review of Agricultural Economics 269 7.79 A 11 The Journal of Development Economics 40 7.73 A 12 Journal of Productivity Analysis 20 7.70 A 13 Sociologia Ruralis 61 7.64 A 14 American Journal of Sociology 20 7.60 A 15 Economic Development and Cultural Change 48 7.46 A 16 Economics Letters 38 7.44 A 17 Journal of Environmental Economics and 30 7.41 A Management 18 Ecological Economics 87 7.36 A 19 Journal of Marketing 29 7.32 A 20 Ko¨lner Zeitschrift fu¨r Soziologie und 28 7.32 A Sozialpsychologie 21 Agronomy Journal 12 7.29 A 22 Australian Journal of Agricultural and Resource 95 7.28 A Economics 23 Agriculture, Ecosystems and Environment 51 7.26 A 24 Public Choice 32 7.24 A 25 Journal of Institutional and Theoretical Economics 33 7.19 A 26 Agricultural Economics 266 7.16 A 27 Environmental and Resource Economics 55 7.12 A 28 The World Economy 30 7.12 A 29 World Development 74 7.10 A 30 Journal of Agricultural Economics 197 7.07 A 31 Ecological Modelling 34 7.03 A 32 Ecology and Society: A Journal of Integrative 17 7.01 A Science for Resilience and Sustainability 33 Food Policy 169 7.01 A 34 ZfbF: Schmalenbachs Zeitschrift fu¨r 50 7.00 A Betriebswirtschaftliche Forschung 35 Journal of Rural Studies 78 6.98 B 36 World Bank Economic Review 70 6.91 B 37 Applied Economics 73 6.90 B 38 Land Economics 52 6.90 B 39 Oxford Development Studies 23 6.89 B 40 Empirical Economics 23 6.88 B 41 Journal on Chain and Network Science 25 6.86 B

2011 The Agricultural Economics Society. 718 Roland Herrmann et al.

Table 1 (Continued)

Number of Quality Rank no. Name of the journal assessments index Rating

42 Environment and Planning C – Government 24 6.82 B and Policy 43 Agricultural Systems 75 6.80 B 44 Economic Modelling 29 6.80 B 45 Journal of Institutional Economics 20 6.80 B 46 Advances in Consumer Research 11 6.77 B 47 Climatic Change 20 6.75 B 48 Zeitschrift fu¨r Betriebswirtschaft 71 6.74 B 49 Resource and Energy Economics 15 6.73 B 50 American Journal of Economics and Sociology 12 6.70 B 51 Review of Income and Wealth 10 6.70 B 52 Journal of Land Use Science 11 6.65 B 53 Journal of Agricultural and Resource Economics 100 6.61 B 54 Journal of Development Studies 26 6.61 B 55 European Journal of Law and Economics 19 6.58 B 56 Agriculture and Human Values 25 6.52 B 57 Canadian Journal of Agricultural Economics 98 6.49 B 58 Review of World Economics 19 6.48 B 59 Rural Sociology 47 6.48 B 60 Applied Economics Quarterly 10 6.47 B 61 Computers and Electronics in Agriculture 23 6.45 B 62 Journal of Consumer Behaviour 33 6.44 B 63 Agricultural Finance Review 19 6.41 B 64 Journal of Agricultural and Food Industrial 26 6.41 B Organisation 65 Land Use Policy 52 6.38 B 66 Journal of Policy Modelling 35 6.33 B 67 Review of the Economics of the Household 10 6.31 C 68 Development and Change 43 6.31 C 69 Journal of Agricultural and Food Economics 27 6.31 C 70 Journal of Food Products Marketing 11 6.30 C 71 Review of Agricultural Economics 107 6.30 C 72 Ecological Indicators 11 6.27 C 73 Journal of Wine Economics 11 6.26 C 74 Journal of Environmental Planning and Management 26 6.25 C 75 Regional Studies 27 6.22 C 76 Schmollers Jahrbuch: Journal of Applied Social 26 6.21 C Science Studies 77 International Journal of Agricultural Resources, 27 6.20 C Governance and Ecology 78 Agribusiness: An International Journal 101 6.16 C 79 Society and Natural Resources 14 6.15 C 80 The Developing Economies 11 6.15 C 81 Jahrbu¨cher fu¨r Nationalo¨konomie und Statistik 70 6.15 C 82 German Economic Review 24 6.13 C 83 Supply Chain Management 27 6.12 C 84 International Journal of Consumer Studies 22 6.10 C

2011 The Agricultural Economics Society. Ranking of Journals Relevant for Agricultural Economists 719

Table 1 (Continued)

Number of Quality Rank no. Name of the journal assessments index Rating

85 Journal of Environmental Management 36 6.09 C 86 Renewable Agriculture and Food Systems 17 6.09 C 87 Food Quality and Preference 26 6.08 C 88 Agricultural and Resource Economics Review 48 6.07 C 89 Agrarwirtschaft 385 6.06 C 90 Applied Economics Letters 32 6.06 C 91 Journal of Comparative Economics 15 6.05 C 92 British Food Journal 33 6.04 C 93 Journal of Applied Economics 27 6.01 C 94 China Economic Review 10 5.94 C 95 Journal of Consumer Policy 14 5.92 C 96 The Journal of African Economies 14 5.87 C 97 Journal of Regional Science 17 5.87 C 98 Livestock Production Science 10 5.83 C 99 Appetite 15 5.81 C 100 Journal of Consumer Marketing 17 5.81 C 101 Journal of Environmental Policy and Planning 13 5.81 C 102 Acta Agriculturae Scandinavica – Section C Food 36 5.79 C Economics 103 Cahiers d’E´conomie et Sociologie Rurales 66 5.76 C 104 Journal of Sustainable Agriculture 32 5.75 C 105 Agricultural Economics Review 43 5.72 C 106 The International Food and Agribusiness 38 5.66 D Management Review 107 The Economics of Transition 24 5.65 D 108 Outlook on Agriculture 40 5.63 D 109 Agroforestry Systems 16 5.57 D 110 Journal of International Agricultural Trade and 31 5.57 D Development 111 Water Policy 16 5.54 D 112 Journal of Rural Development 49 5.54 D 113 Agricultural Economics and Management 17 5.53 D 114 Quarterly Journal of International Agriculture 149 5.52 D 115 Journal of International Food and Agribusiness 26 5.51 D Marketing 116 Schriften der Gesellschaft fu¨r Wirtschafts- und 264 5.49 D Sozialwissenschaften des Landbaues e.V. 117 Agrekon: Quarterly Journal on Agricultural 40 5.48 D Economics 118 Journal of Agricultural Education and Extension 10 5.47 D 119 Journal of Agribusiness 41 5.47 D 120 Forest Policy and Economics 12 5.43 D 121 Zeitschrift fu¨r Agrargeschichte und Agrarsoziologie 63 5.38 D 122 Savings and Development 12 5.37 D 123 Perspektiven der Wirtschaftspolitik 21 5.33 D 124 Precision Agriculture 17 5.33 D 125 Post-Communist Economies 25 5.31 D

2011 The Agricultural Economics Society. 720 Roland Herrmann et al.

Table 1 (Continued)

Number of Quality Rank no. Name of the journal assessments index Rating

126 AgBioForum 25 5.29 D 127 European Journal of Agricultural Education 18 5.29 D and Extension 128 Die Bodenkultur 63 5.21 D 129 African Journal of Agricultural and Resource 35 5.20 D Economics 130 Economie Rurale. Revue Francaise d’Economie 41 5.18 D et de Sociologie Rurales 131 Zeitschrift fu¨r Umweltpolitik und Umweltrecht 39 5.18 D 132 Jahrbuch fu¨r Regionalwissenschaft 29 5.17 D 133 ZAU – Zeitschrift fu¨r Angewandte Umweltforschung 31 5.15 D 134 Eastern European Economics 21 5.14 D 135 Agricultural and Food Science 10 5.12 D 136 Jahrbuch der Absatz- und Verbrauchsforschung 27 5.07 D 137 Journal of Rural Cooperation 18 5.07 D 138 International Journal of Wine Business Research 13 5.04 D 139 Journal of Agriculture and Rural 15 5.00 D Development in the Tropics and Subtropics 140 Planung und Analyse: Zeitschrift fu¨r 13 5.00 D Marktforschung und Marketing 141 Intereconomics 26 4.95 E 142 eJade: The Electronic Journal of Agricultural 22 4.94 E and Development Economics 143 Raumforschung und Raumordnung 50 4.92 E 144 Land, Agrarwirtschaft und Gesellschaft – Zeitschrift 52 4.90 E fu¨r Land und Agrarsoziologie 145 Agricultural Economics (Zemeˇdeˇlska´ ekonomika) 17 4.86 E 146 EuroChoices 134 4.83 E 147 elektronische Zeitschrift fu¨r Agrarinformatik 72 4.81 E 148 WiSt (Wirtschaftswissenschaftliches Studium) 104 4.81 E 149 Jahrbuch der O¨sterreichischen Gesellschaft fu¨r 114 4.77 E Agraro¨konomie 150 Geographische Rundschau 31 4.74 E 151 Osteuropa-Wirtschaft 25 4.69 E 152 Agrarwirtschaft und Agrarsoziologie 55 4.65 E 153 Landtechnik 30 4.54 E 154 Berichte u¨ber Landwirtschaft 308 4.53 E 155 Indian Journal of Agricultural Economics 14 4.45 E 156 WISU (Das Wirtschaftsstudium) 67 4.36 E 157 Landbauforschung Vo¨lkenrode 119 4.26 E 158 Zeitschrift fu¨r das gesamte 45 4.16 E Genossenschaftswesen 159 Forst und Holz 13 3.82 E 160 La¨ndlicher Raum: Rundbrief der Agrarsozialen 99 3.39 E Gesellschaft

Source: Own computations.

2011 The Agricultural Economics Society. Ranking of Journals Relevant for Agricultural Economists 721 the next. Our ranking confirms what many agricultural economists will have expected, and what surveys of European and German agricultural economists already showed clearly in 1984 (Herrmann et al., 1985): the AJAE and ERAE are the top journals in our profession. As the respondents were asked to apply the same criteria when they assessed the standards in the review process and the quality of journal articles, the ranking in Table 1 allows a comparison of journal quality across fields from the respondents’ points of view. Multidisciplinary journals and those from neighbouring, but larger fields, such as agronomy or environmental management, tend to have a higher impact factor than agricultural economics. Table 1 shows that journals such as Agronomy Journal (with an impact factor of 1.35 on average for 2004–2007), Agri- culture, Ecosystems and Environment (1.71), Science for Resilience and Sustainability (1.65), Food, Quality and Preference (1.46), Journal of Environmental Management (1.64) and Appetite (1.46) received a similar quality index or, in several cases, a lower index in the survey-based ranking than the best agricultural economics journals.

3.3. Rating of journals The ranking list generated by means of the above quality index delineates the relative position of each journal within the complete set of journals included in the evaluation. However, the differences in index values between consecutive entries are often small, so that the sequence of the respective journals may be random. To deal with this problem, we subdivided the values of QI into rating classes (as have almost all producers of journal rankings). By switching to an ordinal scale, the problem of distinguishing between similar assessments is reduced to the boundaries between adjacent classes, but is avoided within rating classes. As in other journal rankings, we opted for a partition into ﬁve classes from A to E (Hennig-Thurau et al., 2004), with the upper class split into A and A+. The inner rating classes B to D were equally wide. As the marks assigned to each journal are restricted to values between 1 and 10, the probability of the average grading of a journal get- ting close to the class limits is much lower for the outer classes (A and E) than for the inner ones. Wider class intervals were therefore chosen for the categories A and E than for the categories B to D. This approach gives the following rating classes: A+: ‡8 A: 7.00–7.99 B: 6.33–6.99 C: 5.67–6.32 D: 5.00–5.66 E: <5 The information provided by the survey allowed the selected journals to be ranked according to the methodology described in the previous section. Figure 1 shows the distribution of the evaluated journals across the rating classes (see also Table 1). Of the top ten core journals of agricultural economics, one, i.e. the AJAE,isin the rating class A+. Five journals, i.e. ERAE, AJARE, Agricultural Economics, JAE and Food Policy, are in the rating class A, and the remaining four, i.e. JARE, CJAE, Agricultural Finance Review and JAFIO, belong to rating class B.

2011 The Agricultural Economics Society. 722 Roland Herrmann et al.

4% 13% 17%

A+ (≥8) A (7.00–7.99) 22% B (6.33–6.99) C (5.67–6.32) D (5.00–5.66) E (<5) 20%

24%

Figure 1. Proportion of journals in the different rating classes (n = 160)

Statistical tests indicated that differences in mean quality indices between the chosen rating classes were highly signiﬁcant in all cases.6

4. Sensitivity and Validity of the Results The sensitivity of the ranking and the rating first investigated how the choice of the index and, in particular, the weighting of assessments of standards (S) and the quality of contributions (Q) affected the results. One alternative index was computed with az = 0.5 in all cases including those with nz < 10. Another alternative index was calculated, where each vote on S and Q was counted uniformly and, therefore, the ratio between Q and S in the quality index was variable across journals. The ranking proved to be remarkably stable: using the two alternative indices, a large majority of journals was grouped in the same rating class as with the main index. In addition, we tested whether the audience size, i.e. the number of assessments of journals, affected the quality index. The correlation coefficient, however, was 0.081 and statistically insignificant. After many consistency checks on the data, the validity of the survey results was tested in two ways: (i) we compared the results of the survey-based ranking with the findings of other recent journal rankings to see whether the different rankings were compatible and, if our ranking differed, whether these differences would be explained; (ii) further tests were done to detect any strategic biases in the survey results and how this affected the internal validity of the ranking.

4.1. Comparison with other journal rankings We compared our ranking of journals with rankings from a number of other recent citation-based studies (WU02, CNRS07, ABDC08, ABS08 and KEELE) and survey-based studies (VHB03 and VHB08) that have attempted to produce rankings for journals relevant to our ﬁeld of study. The results are given in Table 2. Between

6 All two-sided unpaired t-tests indicated, at the 99.9% level, that the quality indices for the superior rating classes are signiﬁcantly different than those of inferior rating classes. This result was conﬁrmed for all neighbouring rating classes as well. Detailed statistical results are available from the authors upon request.

2011 The Agricultural Economics Society. Ranking of Journals Relevant for Agricultural Economists 723

Table 2 How the rank order of journals correlates with other rankings*

Ranking

WU02 CNRS07 ABDC08 ABS08 KEELE VHB03 VHB08

Spearman’s rank 0.46 0.49 0.67 0.53 0.65 0.80 0.83 correlation with GEW ⁄ O¨GA P-value 0.003 0.000 0.000 0.000 0.000 0.000 0.000 n 39 63 75 64 58 27 32

*GEW ⁄ O¨GA = GEWISOLA ⁄ O¨GA ranking; WU02 = journal rankings of Wirtschaftsuniversita¨t Wien, 2002; CNRS07 = ‘Categorization of Journals in Economics and Management’, Centre National de la Recherche Scientiﬁque, 2007; ABDC08 = ‘Journal Rating List’, Australian Business Deans Council, 2008; ABS08 = ‘Academic Journal Quality Guide Version 2’, Association of Business Schools, 2008; KEELE = journal rating, Keele University; VHB03 (VHB08) = journal rating, Verband der Hochschullehrer der Betriebswirtschaftslehre, 2003 (2008). Source: Own calculations.

27 and 75 journals were covered in both our ranking and in these other studies. Table 2 reveals that the rank orders are positively and significantly correlated. The highest correlation is with the VHB-JOURQUAL surveys among German academ- ics in business management. Overall, the Spearman rank correlation coefficients are in a medium range, as in the study by Mingers and Harzing (2007). These authors compared 12 journal rankings from business management and computed a mean Spearman rank correlation coefficient of 0.52. In Table 3, QI is compared with the impact factor for all journals covered by our survey-based ranking and the Social Science Citation Index (SSCI) and Science Citation Index (SCI) databases. Table 3 shows a positive and significant correlation between the quality attached to a journal in our survey-based ranking and its impact factor. However, the correlation coefficient is only 0.43 for the journals

Table 3 How quality indices of journals correlate with impact factors*

Pearson correlation coefﬁcient with QI in Type of impact factor GEW-O¨GA P-value n

Impact factors SSCI 0.42 0.001 60 Impact factors SCI 0.50 0.005 31 Impact factors SSCI and SCI 0.43 0.000 79

*GEW ⁄ O¨GA, GEWISOLA-O¨GA ranking; SSCI, Social Science Citation Index; SCI, Science Citation Index. Source: Own calculations.

2011 The Agricultural Economics Society. 724 Roland Herrmann et al. under both the SSCI and SCI. The following reasoning is consistent with a positive, but moderate correlation: 1 Several major economic journals have a high impact factor and are also top journals in our ranking. The American Economic Review, the Economic Journal and the Review of Economics and Statistics are cases in point. This clearly contributes to the positive correlation between QI and the impact factor. 2 The nine agricultural economics journals with a published impact factor are ranked quite highly in the survey-based ranking. This lowers the positive correlation between QI and the impact factor, as their impact factor is relatively low, given the lower market size and citation potential of agricultural economics. 3 Conversely, journals from neighbouring ﬁelds (e.g. Agronomy Journal or Public Health Nutrition) tend to have high impact factors, but come lower in our ranking that also induces lower correlations. 4 In a number of cases, there are other reasons why the evaluation of journals by the respondents differs from those by impact factors. The dominant role of the AJAE in agricultural economics from the respondents’ points of view, for example, is not matched by that journal’s impact factor. Citations per article in the AJAE are often lower than in other core agricultural economics journals. This may be due to the lower citation rates of purely methodological papers, of which the AJAE has absolutely and relatively more.

4.2. Tests for strategic response One weakness of expert-based journal ranking is that it is open to strategic response, i.e. authors may tend to overvalue journals in which they have published to increase their own reputation or that of their (close) peers. Although we have no information about where respondents published, we do know which journal(s) they had submitted to. Using this information, we attempted to determine whether respondents evaluate those journals to which they had submitted an article more favourably. To do so, we estimated the following cluster-specific effects model (Cameron and Trivedi, 2005): z z z z z z z Qj ¼ az þ bSSubj þ beRevj þ bmMalej þ bpPhDj þ bhHabilj þ ej ; ð6Þ z where j indexes individuals and z journals. Qj is the value (between 0 and 10) which respondent j gave to journal z with regard to the quality of published papers, and Sub is a dummy with the value 1 if an expert had submitted to the evaluated journal. As additional explanatory variables, we included Rev, Male, PhD and Habil, all being dummies with the value 1 if an expert had been a reviewer for the evaluated journal, was male, held a PhD or a Habilitation.7 All bs are parameters to be estimated. The intercepts az represent unobserved journal-specific effects. If az are purely random, with a distribution that does not depend on any observable explanatory variables, equation (6) characterises a cluster-specific random-effects model. If the intercepts az are possibly correlated with the regressors, equation (6) stands for a cluster-specific fixed-effects model. In both cases, we can apply estimation techniques derived from panel data analysis (Wooldridge, 2003).

7 The Habilitation is an academic qualiﬁcation in Germany, Austria and Switzerland.

2011 The Agricultural Economics Society. Ranking of Journals Relevant for Agricultural Economists 725

Table 4 How perceptions on the scientiﬁc level of a journal are affected by personal criteria of the respondents – Fixed-effects model*

Equation (6)

Variables ⁄ statistical ﬁt Coefﬁcient P-value

C 6.379 0.000 Sub 0.319 0.000 Rev 0.061 0.418 Male )0.143 0.002 PhD )0.117 0.028 Habil )0.260 0.000 R2 0.304 – F 17.773 0.000 N 6291 – Hausman test statistic 19.268 0.002

*Variables are deﬁned in the text. The scientiﬁc level is measured by perceptions of the qual- z ity of the articles in a journal, i.e. it is identical with the quality component Qj entering the quality index in equation (4). Source: Own calculations.

To test for random vs. fixed effects, a Hausman test is applied. As we can reject the null hypothesis that az and the explanatory variables are independent of each other, at least at the 99% level, we estimated equation (6) as a fixed-effects model. Results of the fixed-effects model and the Hausman test are presented in Table 4. The constant 6.38 gives the average value across all journals for scientific quality. The coefficient for the variable Sub indicates that scientists who had submitted to a journal evaluated that journal by an average (and statistically significant) 0.32 points higher than those who had not. No significant effect exists for those serving as a reviewer. In addition, men evaluated the journals by 0.14 points lower, on average, than women, and the average evaluation of the quality of a journal’s published articles decreased with the degree a respondent holds. Most critical were researchers holding a Habilitation, followed by those with a PhD and those with a masters or diploma.8 Overall, it can be concluded that those who submitted articles to a journal tend to assess that journal more favourably. Hence, there is an upward bias for journals to which a higher ratio of evaluators was submitted. However, this effect seems moderate given that the submitters’ ratio varies between a minimum of 0.014 and maximum of 0.557 with a mean of 0.200. Comparing two journals, therefore, this bias is ceteris paribus (0.557–0. 014) · 0.319 = 0.173 in the most extreme case and 0.319 · 0.2 = 0.064 on average.

8 We also extended the model with interaction dummies (Male · Sub, PhD · Sub, Habil · Sub). However, the simpler model presented here could not be rejected based on a log- likelihood ratio test, and the coefﬁcients of the interaction dummies were not signiﬁcantly different from zero.

2011 The Agricultural Economics Society. 726 Roland Herrmann et al.

5. A Deeper Look at Quality Perceptions of Agricultural Economists: Journal Quality and Journal Relevance In section 3, agricultural economics journals were ranked with the quality index QI and journals were grouped into rating classes. Some important questions remain: (i) Are the quality assessments driven by the respondents’ perceptions of the standards set in the journals’ review processes (S) or the quality of contributions (Q)? (ii) Are the evaluations of scientiﬁc quality (QI) related to their assessments of journal relevance (R)? Based on the perceptions of agricultural economists concerning S, Q and R, it is also possible to characterise the respondents’ views of a high-quality journal.

5.1. Quality standards, quality of contributions and relevance of journals Table 5 first depicts descriptive statistics of agricultural economists’ perceptions of S, Q and QI. The standard deviation and the coefficient of variation for S are much higher than those for Q and for QI. Apparently, agricultural economists see a much higher dispersion in review process standards than in the quality of contributions that results from these. This is a strong empirical argument for the use of a quality index that includes both the S and Q aspects of journal quality. The lower half of Table 5 illustrates the correlation between the four performance criteria. In general, views on S and Q are significantly and positively correlated amongst each other, and the assessments of both quality components are significantly and positively correlated with QI. As Table 5 indicates, the relationship between Q and QI is particularly strong, with a correlation coefficient of 0.98. Another important performance criterion of scientific journals is how relevant these journals are for the work of researchers. We asked the German and Austrian agricultural economists about their views on the relevance for their work of the journals they know. This criterion was not directly included in the measurement of QI. Beyond the scientific quality of a journal, ‘relevance’ covers its other functions, such as its usefulness for teaching, policymaking or industry management.

Table 5 Agricultural economists’ perception of journal performance: Some statistical indicators*

Standards in the review Quality of Quality processes contributions Relevance index (S) (Q) (R) (QI)

Arithmetic mean (x) 6.39 6.11 5.34 6.15 Standard deviation (s) 1.60 0.92 0.70 1.02 Coefﬁcient of variation (v) (%) 25.00 15.10 13.00 16.50 Correlation coefﬁcients S 1.00 Q 0.770*** 1.00 R 0.514*** 0.568*** 1.00 QI 0.847*** 0.979*** 0.592*** 1.00

*n = 160. In all descriptive statistics for ‘standards in the review process’ n = 159, as no vote on S was available for the American Journal of Economics and Sociology. ***Statistically signiﬁcant at the 99.9% level. Source: Own calculations.

2011 The Agricultural Economics Society. Ranking of Journals Relevant for Agricultural Economists 727

It is not clear a priori how high-quality journals perform relative to lower-quality journals in terms of relevance (R). One might expect high-level journals to have a stronger impact on scientific work and therefore that a positive relationship between R and QI seems plausible. However, an opposite hypothesis is that high-quality journals stress primarily new and specific analytical techniques and are only inter- ested in applied articles even on important policy issues, if the article is combined with a new technique. If the treatment of empirical or policy issues is considered an aspect of relevance, then high-quality journals might score lower on this account. The evidence in the lower half of Table 5 shows that R is significantly and positively related to QI and to the quality components. However, with correlation coefficient values between 0.51 and 0.60, the linkages between R and S on the one side and Q on the other side are clearly weaker than between QI and its components S and Q. Table 5 reveals that the relevance of journals is assessed more critically than the quality of contributions and to an even greater degree than the standards in the review process. Moreover, the standard deviation of the views on R is much smaller than on Q and S, and is thus smaller than the standard deviation of QI. The relevance of the journals is judged as more similar than their quality. Across journals, a lower mean value is also attached to R than to QI, Q and S. Consequently, the coefficient of variation of views on R is lower – at 13.0% – than that of views on Q, S and QI. This implies that, when we move from journal- reviewing standards to the quality of contributions and from there to the relevance of journals, the views of agricultural economists become more uniform as well as more critical. Descriptive statistics are presented for the individual rating classes in Table 6. It is noticeable that not only QI but also S and Q decline from rating class A+ to E. Views on R also decline continuously. However, the mean views of R diminish only from 5.86 in rating class A+ to 4.62 in rating class E, whereas the mean quality index falls from 8.43 to 4.57. We performed statistical significance tests on these differences across rating classes.9 There are highly significant differences in agricultural economists’ perceptions of journal quality across rating classes. For instance, at the 99.9% level, the arithmetic mean of Q is always evaluated significantly better when the journal is in a higher than a lower rating class. The evidence is similar for S, but only significant at somewhat lower levels (between A+ and A, A+ and B, and A and B). The perceptions of R are very different: the unpaired t-tests indicate a higher relevance, at the 95% level or more, for the upper rating classes compared with the lower rating classes D and E. Among the higher rating classes, from C to A+, respondents only attach a higher relevance to the mean article in A compared with C journals. All other paired tests between the upper classes do not exhibit any significant differences in terms of R.

5.2. What characterises a high-quality journal? Tests for individual rating classes The perceptions of agricultural economists of what characterises a high-quality journal can be scrutinised if we look at individual rating classes with the paired t-tests in Table 7. Of particular interest are the ﬁndings for the top three rating classes A+, A and B. For all three top-rated classes, the respondents assessed the quality

9 Detailed statistical results are available from the authors upon request.

2011 The Agricultural Economics Society. 728 Roland Herrmann et al.

Table 6 Descriptive statistics for performance criteria of the journals in different rating classes*

SQRQI

Journals rated A+ (n = 6): Arithmetic mean 8.80 8.34 5.86 8.43 s 0.80 0.26 0.45 0.27 v in % 9.10 3.10 7.70 3.20 Journals rated A (n = 28): Arithmetic mean 7.99 7.15 5.83 7.37 s 0.91 0.35 0.53 0.29 v in % 11.40 4.90 9.00 3.90 Journals rated B (n = 30): Arithmetic mean 7.41 6.52 5.60 6.67 s 1.02 0.22 0.58 0.18 v in % 13.70 3.40 10.30 2.70 Journals rated C (n = 40): Arithmetic mean 6.25 6.06 5.45 6.07 s 0.77 0.23 0.63 0.18 v in % 12.30 3.70 11.50 3.00 Journals rated D (n = 35): Arithmetic mean 5.19 5.40 4.93 5.34 s 1.03 0.23 0.50 0.20 v in % 19.80 4.30 10.10 3.80 Journals rated E (n = 20): Arithmetic mean 4.29 4.71 4.62 4.57 s 0.84 0.41 0.63 0.41 v in % 19.60 8.70 13.60 9.00

*s is the standard deviation and v the coefﬁcient of variation. The relevant journals in the categories are those shown in Table 1. Source: Own calculations.

Table 7 Do standards, quality of contributions and journal relevance differ within rating classes? Paired t-tests*

Rating classes S and QSand RQand R

A+ 1.432 (0.212) 7.300 (0.001) 9.787 (0.000) A 4.562 (0.000) 11.684 (0.000) 10.092 (0.000) B 4.586 (0.000) 8.568 (0.000) 8.587 (0.000) C 1.382 (0.175) 5.408 (0.000) 6.288 (0.000) D )1.233 (0.226) 1.373 (0.179) 5.740 (0.000) E )2.036 (0.056) )1.440 (0.166) 0.680 (0.505)

*t-values are shown first, P-values in parentheses. Two-sided paired tests are applied. Source: Own calculations. components, i.e. S and Q, as significantly higher than the relevance of the articles. In all cases, the error probability is <0.1%, given a two-sided test. In classes B and C, the respondents regarded S as being very significantly higher than Q. Only for

2011 The Agricultural Economics Society. Ranking of Journals Relevant for Agricultural Economists 729 journals in the highest rating class, i.e. A+, do agricultural economists assess S and Q as being not statistically different even at the 90% level. In summary, we can state that a high-quality journal is seen by agricultural economists as one where the standards set in its review process are signiﬁcantly higher than the quality of contributions (with the exception of the A+ journals) and where both the standards and the quality of contributions are signiﬁcantly higher than the relevance of its articles.

6. Discussion and Implications The journal ranking presented in this article and the allocation of journals to rating classes are in several important aspects superior to a ranking and rating with impact factors: (i) Comprehensiveness: our ranking is much more comprehensive with respect to journal coverage than the JCR and its impact factors. Beyond English- language and particularly American journals, which dominate under the JCR, relevant German-language journals are covered, as are many more European journals. (ii) Consistent results: the journals in agricultural economics and related disciplines are assessed consistently by agricultural economists. This makes it possible to com- pare journals across this range of disciplines. (iii) Less volatility: the results put the low and, often volatile, impact factors allocated to the core journals of agricultural economics into perspective. Perceptions of agricultural economists are more stable than citations. If the scientiﬁc performance of agricultural economists is to be evaluated with the help of journal rankings, we believe that our ranking goes beyond other rankings with regard to German-speaking countries. Our ranking and rating show internal validity. Strategically biased responses, where present, seem rather weak, and disadvantages of the impact factors are avoided. Agricultural economists, if their journal publications were judged according to our ranking, would not feel compelled to publish in journals outside the ﬁeld of agricultural economics to reach higher impact factors. Of course, the external validity of the results has to be tested in more detail. Simi- lar studies for the European Union seem worthwhile to show whether the results from German and Austrian agricultural economists remain valid on a larger scale. This German–Austrian exercise might be a prototype for survey-based analyses of how other European, US or international agricultural economists evaluate their journals. Apart from the ranking and rating, the analysis provides new evidence about perceptions of journal quality among agricultural economists. Apparently, from the respondents’ points of view, high-quality journals are mainly characterised by high standards and high-quality contributions. However, the standards set in journal review processes are assessed as being higher than the quality of contributions. The relevance of the journal articles for the work of agricultural economists is viewed as more uniform across the journal rating classes than are the quality of articles and the journals’ review standards. There are patterns in agricultural economists’ perceptions of journals that have so far been neglected and which have implications for journal editors as well as for authors. Editors of high-quality journals should reconsider the design of their review processes to combine high achievements in standards, quality of contributions and relevance. For authors, it is interesting to note how their peers

2011 The Agricultural Economics Society. 730 Roland Herrmann et al. characterise high-quality journals. By publishing articles in journals in the highest rating classes, they primarily demonstrate that their work is on a par with the highest academic standards, though not necessarily more relevant or that it signifies a higher-quality contribution than when published in a lower- rated journal. As a final comment, it is necessary to stress that the analysis does not cover all aspects of scientific performance and that the results of the ranking and rating should be applied with care. First, the list of journals ranked and rated is not complete and not representative for all purposes. Some important journals had to be excluded as respondents did not submit articles to, for example, Econometrica or the Quarterly Journal of Economics. If this survey were applied in, for example, a European or international context, German-language journals would be over-repre- sented and adjustments would have to be made. Second, the boundaries of the rating classes should be interpreted with care when used for decisions in the research system. We carried out significance tests to distinguish mean quality levels between classes, but it may be that journals are not statistically different from each other around the boundaries. Careful decisions have to be made here when using the results of the ratings. Third, and most importantly, publications in journals are only one aspect of scientific performance. It has to be borne in mind that agricultural economists contribute more widely through books, book chapters, unpublished reports and advice to policy makers. Ignoring these other aspects of their work would lead to a severe underestimation of the scientific performance of agricultural economists.

References Althouse, B. M., West, J. D., Bergstrom, C. T. and Bergstrom, T. ‘Differences in impact factor across fields and over time’, Journal of the American Society for Information Science and Technology, Vol. 60, (2009) pp. 27–34. Axarloglou, K. and Theoharakis, V. ‘Diversity in economics: An analysis of journal quality perceptions’, Journal of the European Economic Association, Vol. 1, (2003) pp. 1402–1423. Barrett, C. B., Olia, A. and Bailey, D. ‘Subdiscipline-specific journal rankings: Whither applied economics?’ Applied Economics, Vol. 32, (2000) pp. 239–252. Baumgartner, H. and Pieters, R. ‘The structural influence of marketing journals: A citation analysis of the discipline and its sub-areas over time’, Journal of Marketing, Vol. 67, (2003) pp. 123–129. Burton, M. and Phimister, E. ‘The ranking of agricultural economics journals’, Journal of Agricultural Economics, Vol. 47, (1996) pp. 109–114. Cameron, C. C. and Trivedi, P. K. Microeconometrics: Methods and Application (Cambridge: Cambridge University Press, 2005). Dabbert, S., Berg, E., Herrmann, R., Po¨chtrager, S. and Salhofer, K. Das GEWISOLA- O¨GA-Publikationsranking (Hohenheim: Arbeitspapier, 2009). Available at: http://www. gewisola.de/publikationsranking. Last accessed: 15 February 2011. Di Vaio, G. and Weisdorf, J. L. ‘Ranking economic history journals: A citation-based impact-adjusted analysis’, Cliometrica, Vol. 4, (2010) pp. 1–17. Garfield, E. ‘Citation analysis as a tool in journal evaluation’, Science, Vol. 178, (1972) pp. 471–479. Garfield, E. ‘The impact factor’, Current Contents, Vol. 25, (1994) pp. 3–7. Habibzadeh, F. and Yadollahie, M. ‘Journal weighted impact factor: A proposal’, Journal of Informetrics, Vol. 2, (2008) pp. 164–172.

2011 The Agricultural Economics Society. Ranking of Journals Relevant for Agricultural Economists 731

Hennig-Thurau, T., Walsh, G. and Schrader, U. ‘VHB-JOURQUAL: ‘Ein Ranking von betriebswirtschaftlich-relevanten Zeitschriften auf der Grundlage von Expertenurteilen’, Zeitschrift fu¨r betriebswirtschaftliche Forschung (ZfbF), Vol. 56, (2004) pp. 520–543. Herrmann, R., Jensen, U., Scha¨fer, A. and Terwitte, H. ‘A survey of views of agricultural economists in Europe’, European Review of Agricultural Economics, Vol. 12, (1985) pp. 295–311. Jemec, E. G. E. ‘Impact factor to assess academic output’, The Lancet, Vol. 358, (2001) p. 1373. Jokic, M. and Ball, R. Qualita¨t und Quantita¨t wissenschaftlicher Vero¨ffentlichungen – Biblio- metrische Aspekte der Wissenschaftskommunikation (Schriften des Forschungszentrums Ju¨lich, Vol. 15, 2006). Kalaitzidakis, P., Mamuneas, T. P. and Stengos, T. ‘Rankings of academic journals and institutions in economics’, Journal of the European Economic Association, Vol. 1, (2003) pp. 1346–1366. Kodrzycki, Y. K. and Yu, P. ‘New approaches to ranking economics journals’, Contributions to Economic Analysis & Policy, Vol. 5, (2006), Available at: http://www.bepress.com/be- jeap/contributions/vol5/iss1/art24. Last accessed: 20 January 2011. Laband, D. and Piette, M. ‘The relative impacts of economics journals: 1970–1990’, Journal of Economic Literature, Vol. 32, (1994) pp. 640–666. Liebowitz, S. and Palmer, J. ‘Assessing the relative impacts of economic journals’, Journal of Economic Literature, Vol. 22, (1984) pp. 77–88. Lusk, J. L. and Hudson, M. D. ‘Submission patterns, submission policies and revealed preferences’, Review of Agricultural Economics, Vol. 31, (2009) pp. 695–711. Mingers, J. and Harzing, A.-W. ‘Ranking journals in business and management: A statistical analysis of the Harzing data set’, European Journal of Information Systems, Vol. 16, (2007) pp. 303–316. Moed, H. F. Citation Analysis in Research Evaluation (Information Science and Knowledge Management, Vol. 9, Dordrecht, Netherlands: Springer, 2005). Palacios-Huerta, I. and Volij, O. ‘The measurement of intellectual inﬂuence’, Econometrica, Vol. 72, (2004) pp. 963–977. Ritzberger, K. ‘Eine invariante Bewertung wirtschaftswissenschaftlicher Fachzeitschriften’, Perspektiven der Wirtschaftspolitik, Vol. 9, (2007) pp. 267–285. Ritzberger, K. ‘A ranking of journals in economics and related ﬁelds’, German Economic Review, Vol. 9, (2008) pp. 402–430. Saha, S., Saint, S. and Christakis, D. A. ‘Impact factor: A valid measure of journal quality?’ Journal of the Medical Library Association, Vol. 91, (2003) pp. 42–46. Schlinghoff, A. and Backes-Gellner, U. ‘Publikationsindikatoren und die Stabilita¨t von wirtschaftswissenschaftlichen Zeitschriftenrankings’, Zeitschrift fu¨r betriebswirtschaftliche Forschung (ZfbF), Vol. 54, (2002) pp. 343–362. Schneider, F. and Ursprung, H. W. ‘The 2008 GEA journal ranking for the economic profession’, German Economic Review, Vol. 9, (2008) pp. 532–538. Schrader, U. and Hennig-Thurau, T. ‘VHB-JOURQAL 2: Method, results and implications of the German Academic Association for Business Research¢s Journal Ranking’, Business Research, Vol. 2, (2009) pp. 180–204. Schulze, G. G., Warning, S. and Weirmann, C. ‘Zeitschriftenrankings fur die Wirtschaftswis- senschaften: Konstruktion eines emfassenden Metaindexes’, Perspektiven der Wirtschaftspo- litik, Vol. 9, (2008) pp. 286–305. Seglen, P. O. ‘Why the impact factor of journals should not be used for evaluating research’, British Medical Journal, Vol. 314, (1997) pp. 497. Wooldridge, J. M. ‘Cluster-sample methods in applied econometrics’, American Economic Review, Vol. 93, (2003) pp. 133–138. Zapata, H. O. ‘The intellectual impact of agricultural economists’, Journal of Agricultural and Applied Economics, Vol. 41, (2009) pp. 293–314.

2011 The Agricultural Economics Society. 732 Roland Herrmann et al.

Appendix: Internet Sources for Journal Rankings (Retrieved on December 12, 2008) WU02: http://bach.wu-wien.ac.at/bachapp/cgi-bin/ﬁdes/ﬁdes.aspx?journal=true;rating= 2006;lang=DE. CNRS07: http://www.gate.cnrs.fr/spip.php?rubrique31&lang=en. ABDC08: http://www.abdc.edu.au/3.36.0.0.1.0.htm. ABS08: http://www.the-abs.org.uk/?id=257. KEELE: http://www.keele.ac.uk/depts/ec/cer/resources_journals.htm. VHB03: http://pbwi2www.uni-paderborn.de/WWW/VHB/VHB-Online.nsf/id/DE_ Ergebnisse_von_Jourqual_1_aus_dem_Jahre_2003. VHB08: http://pbwi2www.uni-paderborn.de/WWW/VHB/VHB-Online.nsf/id/DE_ Jourqual_2.

2011 The Agricultural Economics Society.