Research in Brazil: Funding excellence Analysis prepared on behalf of CAPES by the Web of Science Group Contents

5 How does Brazilian research perform?

8 International comparisons

18 Identifying excellence in Brazilian research

30 Appendix 1

40 Appendix 2

This analysis has been prepared for CAPES by the Web of Science Group.

It provides a high-level overview of Brazilian research performance describing recent trends.

We are grateful to Professor Carlos Henrique de Brito Cruz, the scientific director of the São Paulo Research Foundation (FAPESP) and Mr. Abel Packer, director and co- founder of the Scientific Electronic Library Online (SciELO) for their contributions.

2 Executive summary

This analysis provides a snapshot of Brazilian research in a global context and highlights the significance of international and industry collaborations on research impact and visibility. Using bibliometrics to analyze data on Brazilian research papers published between 2013 and 2018, illustrative examples of analyses and evidence describe the research landscape and provide an indication of the potential impact of Brazilian research and science policy. The data are drawn from the Web of Science – the world’s most trusted and largest publisher neutral citation index.

Productivity Collaboration: International Brazil ranks 13th in the world in terms of its Between 2013 and 2018, Brazilian output of research articles and reviews researchers collaborated with indexed in the Web of Science. researchers from 205 countries.

In 2018 alone, Brazilian researchers Approximately a third of Brazil’s published more than 50,000 papers. research papers are co-authored with researchers from other countries. The growth in output is 30% over this six year period observed and Growth in the % of papers with an is twice the global average. international co-author has slowed from and increase of 17.5% between 2013 and 2015 to There are pockets of excellence, in an increase of 1.8% between 2016 to 2018. terms of citation impact, in Brazilian research, in the life sciences, physical Collaboration: Industry sciences and engineering. Strategies to promote university-industry research collaborations are bearing fruit in Citation Impact terms of the number of papers published. The percentage of Brazilian papers in the world’s top one percent of most 81% of the joint university industry highly cited papers is consistently publications for the period 2015-2017 where more than one percent during the collaborations between public universities six-year period, 2013-2018. and industry.

The Category Normalized Citation Impact Universities and Research Institutes (CNCI) of this research is below the Public universities are the main source world average, but steadily improving. of research publications in Brazil.

Peer Comparisons The 15 universities with the highest Compared to other BRICS countries, research output, all public, produce Brazil’s research output, in terms of over 60% of the total research output. the number of papers indexed in the Web of Science is average.

Compared to neighboring countries in Latin America, Brazil’s research output has a lower CNCI.

3 Background The 2017 report documented the state In the last decade, Brazil has seen of research output indexed in the Web sizable growth in its research enterprise, of Science. In this analysis we explore maintaining its status as one of the Web of Science data4 with a focus on BRICS countries, alongside Russia, international and university-industry India, China, and South Africa, and as collaborations and the research output an emerging economy demonstrating of universities and research institutes to rapidly growing scientific productivity offer illustrative examples of how different and influence measured through analyses can generate evidence to help bibliometrics. Brazil is the fifth most inform policy and decision-making. populous country in the world, and remains in the top 10 world economies, We also examine some results from despite recent economic turmoil. the Scientific Electronic Library Online (SciELO), which was created and In this updated analysis the 2017 research implemented in Brazil in 1998 to strengthen profile of Brazil is revisited1 to examine research infrastructure by developing where Brazilian researchers continue to be scholarly communication capacity. SciELO active and to identify areas of excellence. currently includes 296 Brazilian journals, Over the last decade there has been a of which 99 are high quality, influential growing interest in Brazil’s research output journals indexed in the Web of Science. focused on both the national level2 and on specific fields of research.3 Remarking on When studying the scientific output of the rapidly changing research landscape a country such as Brazil, it is advisable in Brazil, Leta, Thijs, and Glänzel (2013) to focus bibliometric analysis on note that “The huge growth in the Brazilian documents within the Web of Science production of publications constitutes the that are classified as articles or reviews greatest potential that goes far beyond the as these are peer-reviewed publications Latin American region.” that reflect mature research findings.

1 https://www.capes.gov.br/images/stories/download/diversos/17012018-CAPES-InCitesReport-Final.pdf accessed August1, 2019. Adams J. and King C. (2009) Global Research Report: Brazil - Research and Collaboration in the New Geography of Science. https://www.slideshare.net/nielsleidecker/grr-brazil-jun09-1 2 Glänzel, W., Leta, J. Thijs, B. (2006) Science in Brazil Part 1: A macro-level comparative study. Scientometrics, 67 (1), 67–86. Leta, J, Thijs, B, & Glänzel, W. (2013) A macro level study of science in Brazil: seven years later. Bibli Encounters: Electronic Journal of Library and Information Science [Online], accessed 25 Aug. 2019 3 Leta, J., Glänzel, W., Thijs, B. (2006), Science in Brazil. Part 2: Sectorial and institutional research profile. Scientometrics, 61 (1), 87-105. Andrade Vargas, R.; de Souza Vanz, S.A; Chittó Stumpf, I.R. (2015) Brazilian agricultural research in the Web of Science: a bibliometric study of scientific output and collaboration (2000-2011). Em Questão, 21 (3), 296-318. 4 Throughout this document, the analysis is restricted to articles and reviews, indexed in the Science Citation Index Expanded (SCIE), the Social Sciences Citation Index (SSCI) and the Arts & Humanities Citation Index (AHCI) of the Web of Science. Whenever 4 mentioned, additional analyses are based on the Brazilian collection of SciELO Citation Index, hosted on the Web of Science platform. How does Brazilian research perform?

5 An overview of Brazilian research performance in recent years

How many research papers India (10th) and South Korea (12th), does Brazil produce? and ahead of Russia (15th) and South Africa (21st) (Figure 1). It is worth noting that despite changing economic Brazil was ranked 13th in the world in conditions, the growth in Brazilian terms of its output of research papers research output has been consistently between 2013 - 2018, just behind strong over this six-year period.

Figure 1

1. USA 2,658,867

2. China 1,810,725

3. UK 812,718

4. Germany 697,608

5. Japan 496,152

6. France 475,964

7. Canada 445,330

8. Italy 435,277

9. Australia 413,301

10. India 403,141

11. Spain 374,281

12. South Korea 352,551

13. Brazil 280,912

14. Netherlands 260,736

15. Russia 217,595

16. Switzerland 197,416

17. Iran 196,272

18. Turkey 178,300

19. Sweden 176,610

20. Poland 172,975

21. South Africa 86,992

0 500,000 1,000,000 1,500,000 2,000,000 2,500,000 3,000,000

Number of Papers

Papers added to Web of Science, 2013 – 2018.

6 This growth, 30% over this six-year percent of most highly cited papers, and period, has been twice that of the global the right panel shows the percentage of average (15%). In 2018 alone, Brazilian papers published by Brazil that are in the researchers published over 50,000 world’s top ten percent. When looking papers indexed in the Web of Science. at the top ten percent of highly cited work, Brazil is below the world average In addition to the 280,912 papers indexed and the decrease in recent years seems in the Web of Science (2013-2018), Brazilian more prominent. This seeming drop is authors published more than sixty thousand not unique to Brazil as larger drops in this articles and reviews indexed in SciELO measure are seen in recent data from journals that are not covered in the Web countries that have extensive research of Science. Bibliometric studies often outputs. It is important to emphasize that use the world’s top one and ten percent care be exercised in the interpretation of of most highly cited papers as proxy recent citation counts from 2018, which indicators for the production of excellent tend to be volatile and the apparent research. In Figure 2, the left panel shows deviations from the norm may be artefacts the percentage of papers published of recency of the data rather than dramatic by Brazil that are in the world’s top one departures from persistent trends.5

Figure 2

1.4 10 World Average

1.2 8 World Average 1.0

6 0.8

0.6 4 % Papers in world's top 1% top world's in % Papers % Papers in world's top 10% top world's in % Papers

0.4

2 0.2

Brazil Brazil 0.0 0 2013 2014 2015 2016 2017 2018 2013 2014 2015 2016 2017 2018

Percentage of Brazilian papers in the top one percent and top ten percent of most highly cited papers, 2013 – 2018 (world average denoted by the dotted line).

7 5 Adams, J. (2018) Information and misinformation in bibliometric time-trend analysis. Journal of Informetrics 12(4):1063-1071 International comparisons

8 What is the citation impact proportional increase in publications of Brazilian research? and high impact publications from China masks effects in other countries.

As Brazil’s research output grew between This downturn in 2017 is not due to a 2013-2016 the impact of its research, change in the coverage of Brazilian journals as indicated by the number of citations in the Web of Science. One potential received, also grew steadily. Figure 3 explanation for the decrease in CNCI presents a selective global perspective is the reduced growth in the number of on the distribution of the Category papers with one or more international Normalized Citation Impact (CNCI)6. co-authors. It is widely recognized that international collaborations enhance the Brazil’s annual CNCI reached a peak citation impact of an article10. In the three in 2016 at 0.91, compared to a world years, 2013 to 2015, the percentage of average of 1.0. There is an observed papers with an international co-author drop however, in Brazil’s CNCI in 2017. increased by 17.5%, in the most recent three years 2016 to 2018 there has only Other Latin American7 countries also exhibit been 1.8% growth. As mentioned above, this trend. This trends it is not replicated in a more likely explanation of this seeming developed8 countries. This trends it is not downturn in Brazilian CNCI is that it is an replicated in the BRICS9 as the dramatic artefact of the volatility of recent data.

Figure 3

World Average Brazil 1.3

1.2

1.1

1.0 CNCI 0.9

0.91 0.91 0.91 0.88 0.88 0.88 0.8 0.87 0.87 0.87

0.81 0.81 0.81 0.79 0.79 0.79 0.7

0.6 2013 2014 2015 2016 2017 2013 2014 2015 2016 2017 2013 2014 2015 2016 2017 Latin America Developed BRICS

Category Normalized Citation Impact (CNCI) for Brazil11 and comparator countries by groups, 2013 – 2017.

6 See the appendix for additional details regarding the computation of the CNCI. 7 Argentina, Chile, Colombia, Mexico 8 Canada, France, Germany, Japan, Spain 9 Russia, India, China, South Africa 10 Katz, J.S. & Hicks, D. Scientometrics (1997) 40: 541. https://doi.org/10.1007/BF02459299, accessed August 21, 2019. Adams, J. & Loach, T. (2015). Comment: A well-connected world. Nature. 527. S58-S59. 9 11 The diamonds and the numbers above the diamond indicate Brazil’s CNCI How does Brazilian impact across countries to gain insight research compare into the relationship between these factors and research outputs. For internationally? instance, China’s recent investments in higher education and research have led Several factors, such as the strength of to exponential growth in the quantity the economy, level and focus of research and quality, as measured by the number funding, international collaborations and of citations, of research output. the caliber of educational and research institutions, can affect the quantity In this analysis of Brazil and its comparator and influence of a country’s research countries, China, which has published output. Different countries vary in their at least three times more papers than approaches to science and technology Brazil or any other comparator country, policy, so although it is difficult to tease is an extreme outlier, which distorts the out the individual contribution of any visualization of the data. China data is of these factors, it could be instructive included in the analysis but excluded to compare productivity and research from the visualisation Figure 4.

Figure 4

1.6 Canada Germany South Africa 1.4 Colombia

France 1.2 Spain Chile

1.0 World Average Impact Japan

0.8 CNCI

Brazil Mexico India 0.6 Argentina Russia

0.4

0.2

0.0 0 100,000 200,000 300,000 400,000 500,000 600,000 700,000 800,000

PAPERS

Output and Category Normalized Citation Impact (CNCI) of all comparator countries excluding China. The diameter of the circles is proportional to the percentage of papers in the world’s one percent most highly-cited papers, 2013 - 2018.

10 Compared to the other BRICS countries Figure 4 also shows that Brazil publishes Brazil’s research output is average, with many more papers than other Latin more publications than those from American countries, though the other South Africa and Russia and a higher countries have higher citation impact citation impact than Russia and India. and proportion of papers in the top one China has higher citation impact, percent of papers worldwide. The higher above the world average, and vastly citation impact is probably derived from greater output of peer reviewed papers. international collaboration. Chile and Compared to developed countries Colombia produce most of their research, Brazil has a lower citation impact. Brazil’s 67% and 67.5% respectively, in collaboration research impact is only slightly behind with researchers from other nations while Japan’s which is the only comparator this percentage for Brazil is 36% over the developed country with a below world same period. Additional analyses of trends average citation impact (Figures 3, on would provide useful insights into Brazil’s page 9 and figure 4, on page 10). relative position and how it has evolved over time with respect to other countries.

International collaboration and its citation impact

Who does Brazil collaborate impacts are associated with publications with internationally? with partners in fellow BRICS countries, such as China and India, where research output has been growing rapidly in recent Brazilian researchers collaborate with years. Here again, a more detailed analysis scholars from all over the world. Over the of the papers with multiple co-authors six-year period, 2013-2018, they co-authored suggests a qualitative difference between papers with researchers from 205 countries, collaborations that arise out of work which represented approximately a third conducted at hyper-collaborative research of all the papers in the Web of Science with institutions such as CERN where papers with Brazilian authors. These collaborations hundreds of co-authors garner thousands of included established research-intensive citations versus papers that are the outcome nations such as the G7, neighboring of small groups of researchers working countries in Latin America, as well as collaboratively on relatively small projects. fellow BRICS countries. As mentioned earlier, international collaborations Brazil is also developing more regional usually have a higher citation impact. collaborations within Latin America as noted in the Global Research Report on The highest number of international Brazil (Adams & King, 2009)12. Investments co-authored papers are published with in projects such as the Brazilian synchrotron authors in the United States, although these light laboratory, the only source of papers do not deliver the highest citation synchrotron light in Latin America, enhance impact. Some of the highest citation opportunities for international collaborations.

12 Adams J. and King C. (2009) Global Research Report: Brazil - Research and Collaboration in the New Geography of Science. 11 https://www.slideshare.net/nielsleidecker/grr-brazil-jun09-1 This laboratory is the research home for Linguistics, Literature and Arts, Humanities approximately 1,200 Brazilian and foreign and Social Sciences, Applied Social researchers involved in hundreds of Sciences and Multidisciplinary are projects and co-authored publications13. excluded from this analysis. The distribution of the number of papers in each of the Detailed analyses of Brazilian research remaining five categories is shown by the output from international collaborations size of the bars in the middle. The flows is presented in our 2017 CAPES report14. from the middle bars to the countries on An alternative analysis and visualization of the right are proportional to the number of international collaborations is demonstrated papers with international co-authors from in Figure 5, showing the proportion the countries on the right-hand side. There of papers published betweeen 2013 is some double counting in these flows and 2018 in a select subset of the nine because several papers have co-authors CAPES research categories relative to from multiple countries. The size of the international co-authors countries of origin. rectangles is proportional to the number of co-authored papers, showing Spain with The bar on the left represents all papers the most co-authored papers, which is published in this six-year period in five notable as it has a smaller overall output of of the nine CAPES research categories papers compared to the other developed with the largest numbers of publications. countries and some of the BRICS.

Figure 5

Exact and Earth Sciences China

Russia BRICS India

South Africa Health Sciences

France

Germany Brazil Developed Biological Sciences Spain

Canada

Japan Engineering Argentina

Chile Latin America Columbia Agricultural Sciences Mexico

International collaboration with select countries by different research area

13 https://www.lnls.cnpem.br/the-lnls/about/ - accessed August 17, 2019. 12 14 https://www.capes.gov.br/images/stories/download/diversos/17012018-CAPES-InCitesReport-Final.pdf - accessed August 17, 2019 Among these comparator countries, Across all research areas, Japan has the developed countries have the largest number lowest number of papers in collaboration of collaborations with Brazil. However, the with Brazil. These low numbers are consistent number of collaborative papers does not with Japan’s lower overall percentage seem proportional to the partner country’s of papers with international co-authors total publication output. If this were so, (31.8%) compared to the other developed it would be expected that China would partners which have 50% or more. have the most collaborative papers. It appears that the number of collaborations Latin American countries have lower depends upon existing research networks numbers of collaborative papers compared and with researchers in partner countries to Brazil’s partner countries from other parts with a history of excellence in science. of the world. Argentina collaborates most frequently with Brazil in Agricultural Sciences In proportional terms, the fewest (439 papers) and Biological Sciences (1,762). collaborations are in Agricultural Sciences Across the region, the most collaborations and the most in Exact and Earth Sciences. are in Exact and Earth Sciences and Health For each partner country, Exact and Earth Sciences and the least in Engineering. Sciences is usually the most productive research area, with at least 1,000 papers In SciELO, international collaborations have per partnership. The highest number grown steadily over the past few years and of papers are with European countries, are close to 10% of all publications in 2018. with France collaborating on 6,381 The most frequent collaborations, indexed papers in Exact and Earth Sciences, the in SciELO, are with researchers in USA, most papers in a single discipline. Portugal, China, Argentina, Spain and Turkey.

A large proportion of collaborations in BRICS countries collaborate infrequently in Health Sciences and Biological Sciences Agricultural Sciences and Biological Science. tend to be with the US (40 to 50%) followed There is greater collaboration by China by European countries and Argentina, with and Russia in Exact and Earth Sciences and BRICS countries playing a much smaller role. with China and India in Health Sciences.

13 What is the citation impact of international collaborations?

The citation impact of papers with Sciences (Colombia, 0.72) and Engineering international co-authors is generally (Argentina, 0.91). The lowest CNCI higher than that of papers published by numbers are in Agricultural Sciences only Brazilian authors.15 For papers with where they range between 0.72 and 3.17 Brazilian and international co-authors (Russia). Even in Agricultural Sciences, this number is consistently high (Figure CNCI is extremely high for BRICS 6), and above the world average of 1.0. collaborations. Papers published in The few exceptions where the CNCI Health Sciences have the highest citation is below 1.0 are for collaborations with impact ranging from 4.13 (Spain) to the Latin American countries in Agricultural remarkably high score of 15.51 (Russia).

Figure 6

Agricultural Sciences Biological Sciences

3 4

2 3

2 1 1

0 0 Spain Columbia Columbia Germany Japan Germany Spain South Africa South Africa China Russia China Russia Japan

Chile

Chile Mexico Mexico Canada France Canada France Brazil Brazil Argentina Argentina India India

Engineering Exact and Earth Sciences

2 4

1.5 3

1 2

0.5 1

0 0 Columbia Columbia South Africa Spain South Africa China Russia Germany Spain China Russia Germany Japan Japan

Chile Chile Mexico Mexico Canada France Brazil Canada France Brazil India Argentina India Argentina

Health Sciences

15

10

5

0 Columbia Germany Spain South Africa China Russia Japan

Chile Mexico Canada France Brazil Argentina India

Category Normalized Citation Impact (CNCI) of Brazilian papers with co-authors from select countries by research area

These high citation impact numbers should guidelines or global impact of diseases. be treated with caution. The publications These papers are often cited thousands of underlying these numbers include several times. Brazilian citation impact tends not to hyper-collaborative papers with many be driven by regional collaboration; rather authors and many affiliated countries it is concentrated in papers with co-authors reporting on topics such as particle from further afield, such as the United physics experiments, astronomy, medical States, Europe, and other parts of the world.

15 The link between co-authorship and citation impact is a phenomenon that has been widely observed, 14 and is discussed in Moed, H. (2005) Citation Analysis in Research Evaluation. Dordrecht: Springer pp285-290. University – industry collaboration

How has university-industry The number of publications co-authored research collaboration in by researchers in universities and industry offers a window into ideas that were jointly Brazil changed in 37 years? created and developed by researchers in the two sectors indicating a higher level University-industry interaction has been of engagement than mere consultation, a central objective of Brazilian Science contract research and development or and Technology policy for many decades. research support and donations. Figure Paradoxically, very few measurements of 7 shows the evolution of these research the quantity and intensity of this interaction interactions since 1980, by counting have been performed. Here we contribute the number of Web of Science items (all an analysis based on the number of categories) that have at least one author scientific and technical publications from a university in Brazil and a co-author that display co-authorship between from industry, anywhere in the world. researchers in universities in Brazil and researchers in industry16 (for a discussion Figure 8 shows the evolution of co- of the methods, see the Appendix 2). In authorship between universities in Brazil order to provide a broader context for and industrial sector authors since 1980 for these university-industry collaborations, the ten universities with the largest number a longer-term view, going back to 1980 of joint publications. The ten universities is informative, rather than the six-year shown - all public - account for 81% of the window used in the rest of this anlysis. joint publications for the period 2015-2017.

Figure 7

1600

1400

1200

1000

800

600

400

Number of Items with University-Business Co-Authorship with University-Business Items of Number 200

0 2009 2017 1987 1989 1993 1997 2001 2003 2005 2007 2011 2015

2013 1983 1991 1995 1999 1980 1981 1982 1984 1986 1988 1900 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 1985

Number of Web of Science items with at least one author in a university in Brazil and at least one co-author from industry.

16 The data presented here on university-industry collaboration includes a set of corporate entities which is much larger than that presently considered at the Web of Science or Incites. These data were obtained by performing an enhanced search procedure which included reclassifying the nature of 15 entities in the list of publications with authors in Brazil and also searching for entities that have suffixes associated with industry. Which universities co-author more with industry?

Figure 8

450 Universidade de Sao Paulo 400 Universidade Federal do Rio de Janeiro Universidade Estadual de Campinas 350 Universidade Federal de Sao Paulo (UNIFESP)

300 Universidade Federal do Rio Grande do Sul Universidade Federal de Minas Gerais 250 Universidade Estadual Paulista Universidade Federal do Parana 200 Universidade Federal de Vicosa 150 Universidade Federal de Santa Catarina (UFSC)

100

50

0 2009 2017 1987 1989 1993 1997 2001 2003 2005 2007 2011 2015

2013 1983 1991 1995 1999 1980 1981 1982 1984 1986 1988 1900 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 1985

Number of Web of Science items with at least one author in a university in Brazil and at least one co-author from industry for the ten universities with the largest number of items in 2017.

Figure 9

60%

50%

40%

30%

20%

10%

0% 1985 - 1987 1995 - 1997 2005 - 2007 2015 - 2017

Agricultural Sciences Biological Sciences Health Sciences Engineering Exact and Earth Sciences

Classification of the publications in co-authorship between universities in Brazil and companies according to the ‘Fields of Research’ defined by CAPES.

The distribution of the university-industry Literature and Arts, Humanities and Social co-authored publications among five Sciences, Applied Social Sciences and of the nine fields of research defined by Multidisciplinary categories. The distribution CAPES is shown in Figure 9, for four three- which was strongly dominated by the year periods: 1985-87, 1995-97, 2005-07, Exact and Earth Sciences 30 years ago and 2015-17. Only five of the nine CAPES has evened out over time across the five categories are shown here because of the categories, especially, between the life and small number of collaborations in Linguistics, physical sciences for the 2015-17 period.

16 Which companies co-publish with universities in Brazil?

Petrobras, which has a strong program 14% for developing research collaborations Petrobas participated in 14% with academia, dominates the landscape of all university-industry of university-industry collaboration in Brazil (Figure 10). In the period collaborative output. 2015-2017, with 543 publications, the company participated in 14% of all university-industry collaborative output.

Figure 10

Petrobras Novartis Vale Pfizer 543 77 66 60

IBM Embraer Agilent Roche 54 37 37 36

Sanofi Dow Fibria Itaipu 36 Agrosciences 28 24 31

GSK 48 Monsanto Genentech Ericsson Eletrobras 24 22 22 35

Eli Lilly Suzano Braskem 24 20 19 AstraZeneca 39 Bayer 35 Merck IPEF Whirlpool / 23 20 Embraco 19

The 25 companies that had the largest number of publications in co-authorship with universities in Brazil from 2015-2017.

Of the 50 companies with the highest levels The creation of the Science and Technology of co-authorship with academia, 17 are of Sector Funds, to finance domestic Brazilian origin and 33 are multinationals. research, development, and innovation There is a predominance of companies spurred green economy developments from the pharmaceutical sector (18 of 50), and university-industry bio-ethanol and followed by the agricultural sector (12 of 50). bio-pharma research collaboration.17

17 Financial Times Confidential Research, (2015) Brazil – Biopharma seeking a pick-me-up, http://www.scienceforbrazil.com/wp-content/ uploads/2015/12/Brazil-%E2%80%93-Biopharma-seeking-a-pick-me-up.pdf accessed August 22, 2019. Corteza, LAB, Nogueirab, LAH, Lealc, MRVL, Junior, RB. 40 Years of the Brazilian Ethanol Program (Proálcool): Relevant Public Policies and Events Throughout Its Trajectory and 17 Future Perspectives http://bioenfapesp.org/gsb/lacaf/documents/papers/05_ISAF_2016_Cortez_et_al.pdf accessed August 22, 2019. Identifying excellence in Brazilian research

18 How does Brazilian Partly because of the small numbers, research in different Brazilian research output has a high citation impact, above world average, in Applied research areas compare? Social Sciences and Humanities and Social Sciences (Figure 11). Because the CNCI To focus on research excellence, is a weighted average, the highly cited between 2013-2018, analysis of the papers in sub-categories of Theology (145 nine categories defined by CAPES papers with an average CNCI of 2.37), is insightful. Excellence in this Archaeology (224 papers with a CNCI of context is measured by the Category 1.24) and Anthropology (393 papers with Normalized Citation Impact (CNCI). a CNCI of 1.36) raise the overall research impact above the world benchmark of Among these nine CAPES categories, 1.0. Theology is notable in another sense most papers by Brazilian researchers are in that only 18.6% of the papers have an published in life and physical sciences international co-author even though its and engineering. The main mode for CNCI is the highest of any of the CAPES the dissemination of research results 121 sub-categories. Thus, the excellence is in these fields are journals, which driven by domestic strength in the field. are well represented in the Web of Science database. In contrast, in the As shown in Figure 11, more than a third arts, humanities and social sciences of the papers in Biological Sciences and researchers are more likely to write almost half in Exact and Earth Sciences have books or policy documents, which are foreign collaborators, which, as previously less well represented in this database. noted, contributes to a higher CNCI.

Figure 11

% International Research Area Papers CNCI Collaborations

Health Sciences 82,406 0.96 34.7 Biological Sciences 75,717 0.74 37.0 Exact and Earth Sciences 71,214 0.90 45.1 Agricultural Sciences 46,222 0.71 21.7 Engineering 42,506 0.76 37.1 Multidisciplinary 30,190 0.82 39.5 Applied Social Sciences 14,229 1.03 30.1 Humanities and Social Sciences 9,581 1.00 30.5 Linguistics, Literature and Arts 953 0.68 19.1

Output and Category Normalized Citation Impact (CNCI) of Brazilian papers published between 2013 and 2018 in nine CAPES research categories

19 Delving deeper into the CAPES 121 sub- categories provides some insight into how Figure 12 this growth came about. For a long time within Engineering, Materials Sciences and Metallurgy has been a highly productive 16,000 area that has seen rapid growth in recent years. There has also been considerable growth in the closely aligned area of Exact 14,000 and Earth Sciences (chemistry, physics, etc.). Brazil’s wide range of expertise and skills in these two related areas, and 12,000 multidisciplinary research that bridges the two seem to have fueled this rapid growth.

10,000 Bibliometric analysis can be used to determine research areas that demonstrate strength, weakness, potential for 8,000 opportunity, or areas that are under threat in a research portfolio. Figure 13 maps Brazil’s research in the nine CAPES

6,000 categories along two indices benchmarked on global output and citation impact.

4,000 The horizontal axis measures relative proportional output which is a ratio representing the proportion of a nation’s research output relative to that research 2,000 2013 2014 2015 2016 2017 2018 area’s proportion of the world output. For instance, if Biology’s share of research Health Sciences Multidisciplinary output in Brazil is 4.5% and the global share Biological Sciences Applied Social Sciences Exact and Earth Sciences Humanities and Social Sciences of Biology research indexed in Web of Agricultural Sciences Linguistics, Literature and Arts Science is 1.5% then Biology’s score along Engineering the horizontal axis would be 3. The vertical axis is the Category Normalized Citation Impact (CNCI). Thus, research areas located Trends in the number of papers published annually in different CAPES research categories between 2013 and 2018. in the top right-hand quadrant where the relative output and citation impact are both above 1.0 are considered areas of strength. Brazilian research focus has evolved over six Research areas located in the top left-hand years (Figure 12). There has been increased quadrant where the output is relatively low, output in all research areas except for but citation impact is high are deemed Linguistic, Literature and Arts. The smallest potential opportunities. A research area is increases have been in Health Sciences, considered weak if it is in the bottom left (18.7%) and Agricultural Sciences (21.9%). quadrant where both relative output and Although not insubstantial, these increases citation impact are below world averages. seem small compared to Engineering, which Research areas in the bottom right-hand has grown by nearly two-thirds, surpassing quadrant with relatively large output but the output of Agricultural Sciences in 2018. low citation impact are under threat.

20 Areas of strength be considered weak research areas. and opportunity However, as already noted, Linguistics, Literature, and Arts its location in the bottom left might be an artefact of Health Science with its six-year relative how “counting” is done in the Web of output slightly above 1.0 and CNCI at Science. Much of this research may 0.91 can be an area of potential strength, be in Portuguese with a limited global especially because papers published readership and therefore, low coverage since 2015 have had above world average in the Web of Science. Thus, in global citation impact. In the bottom right, terms, this area is likely to appear threatened areas are Biological Sciences “weak” because of its specialized and Agricultural Sciences, as Brazil does nature and small research community. proportionately more research in these areas than the global average. Brazil is With Sirius18, the new Brazilian 3.5 times more productive than the world synchrotron light source, coming average in Agricultural Sciences though on line, more opportunities for these papers have below average citation Brazilian research exist in the Exact impact for the world and Brazil. However, and Earth Sciences, where citation agricultural research which is often of impact is above average for Brazil, but tremendous local importance does not internationally it does proportionally always garner international attention. less well than other research areas. A low CNCI, as a measure of global citation impact, does not account for the On the other hand, additional analysis impact of regionally relevant research. must be conducted to ascertain whether the scores associated with Applied Social Linguistic, Literature and Arts, and Sciences and Humanities and Social Engineering in the bottom left might Sciences are artefacts of their small size.

Figure 13

CNCI 1.1 Applied Social Sciences Humanities and Social Sciences

1.0 World Average Impact Health Sciences Exact and Earth Sciences

0.9

Multidisciplinary Relative Output

0.0 0.5 0.8 1.5 2.0 2.5 3.0

Engineering Biological 0.7 Sciences Agricultural Sciences Linguistics, Literature and Art

0.6

Strength, weakness, opportunity and threat analysis in Brazilian research aggregated by CAPES research area.

21 18 https://www.lnls.cnpem.br/sirius-en/ - accessed August 19, 2019. Further analyses of CAPES sub-category data could suggest strategies for Figure 14 enhancing research growth, citation impact or some combination of both. For

3,000 instance, the proportion of Agricultural Sciences research in Brazil is over three times the global proportion and continues to be a highly productive 2,500 area. Its citation impact is persistently below world averages, locating it under “threat” in the bottom right of Figure 13. However, a detailed analysis of the 2,000 sub-categories that make up the field could suggest a different story and different strategies for attaining Brazil’s research objectives for this field. 1,500

The Agricultural Sciences’ low CNCI can be attributed to Agronomy (CNCI of

1,000 0.46), which accounts for over a third of all papers in this research category (Figure 14). But the field is changing. There has been rapid growth in historically less 500 productive research areas that have achieved or are approaching world average citation impact. The number of papers has grown by 60% in Food 0 Science and Technology, which has 2013 2014 2015 2016 2017 2018 a CNCI of 1.11. There has also been Agronomy (0.55) a 78% growth in Fishing Resources Food Science and Technology (1.11) and Fishing Engineering papers Veterinary Medicine (0.72) whose citation impact has recently Fishing Resources and Fishing Engineering (0.91) Zootechnics (0.87) exceeded the world average. The Forest Resources and Forestry (0.49) level of international collaborations in Agricultural Engineering (0.47) Agricultural Sciences is low and below average for Brazil at 21.7%. Efforts to increase international collaborations Six-year trend in the number of papers published annually in each CAPES Agriculture sub-category. Category Normalized could enhance both international Citation Impact (CNCI) included in parentheses. visibility and potentially, the CNCI.

22 Where is this research conducted?

Which are the leading The relative intensity of research in research organizations different disciplines is similar across Brazil and most universities. Usually in Brazil and where do the most productive areas of research they excel? for a university are the areas with the highest output across Brazil; that is, Examining the five most productive Health Sciences, Biological Sciences, CAPES research categories comparisons and Exact and Earth Sciences. However, can be drawn between the institutional there appear to be universities that output and the citation impact of papers do proportionally more Agriculture published between 2013 and 2018. The Sciences and Engineering research. analysis is divided into two organizational categories: universities and specialized Figure 15 shows the numbers of papers research institutes. The most productive 15 in five CAPES categories published universities and 10 research institutes are from these 15 Brazilian universities examined. While the universities have areas with the largest output in the Web of of emphasis, most of them conduct research Science between 2013 and 2018. The in each of five CAPES research categories. shading indicates level of research In contrast, the research institutions, with output, that is, darker shades indicate two notable exceptions, were established to institutions with higher output within focus on specific research areas, which are a CAPES research category. These evident from the distribution of their research 15 universities produce more than output across the five CAPES categories. half of all research output in Brazil.

23 Universidad de São Paulo dominates Science, and perhaps to a lesser extent in among the universities, producing Engineering is the distribution of research more than twice the total number of output more evenly distributed across publications compared to Universidade other universities producing papers in Estadual Paulista, which is the second numbers that are comparable to the most productive university in terms of number of papers published by researchers research output. Only in Agricultural affiliated with Universidade de São Paulo.

Figure 15 Health Sciences Health Sciences Biological Sciences Earth and Exact Sciences Agricultural Engineering All Research

Universidade de Sao Paulo 21,912 17,025 14,536 6,476 6,819 58,899 Universidade Estadual Paulista 5,283 6,948 5,336 5,908 2,914 22,868 Universidade Estadual de Campinas 5,719 4,416 6,571 1,989 3,941 19,317 Universidade Federal do Rio de Janeiro 4,672 5,351 5,503 981 3,038 17,484 Universidade Federal do Rio Grande do Sul 5,199 4,009 3,960 2,168 2,599 15,860 Universidade Federal de Minas Gerais 5,233 4,349 3,293 1,809 2,108 14,904 Universidade Federal de Sao Paulo (UNIFESP) 7,372 3,186 1,212 358 724 11,228 Universidade Federal do Parana 2,133 3,333 2,486 2,190 1,628 9,995 Universidade Federal de Santa Catarina (UFSC) 2,473 1,974 2,468 1,358 2,284 9,162 Universidade Federal de Pernambuco 1,778 2,302 2,391 662 1,082 7,098 Universidade de Brasilia 1,756 2,039 2,023 895 892 7,056 Universidade do Estado do Rio de Janeiro 2,110 1,315 3,046 281 1,030 7,039 Universidade Federal de Sao Carlos 977 1,727 2,643 670 2,072 6,980 Universidade Federal de Vicosa 602 2,726 940 3,064 441 6,893 Universidade Federal de Santa Maria (UFSM) 1,247 1,809 1,425 2,522 782 6,670

Research output of 15 top universities in five CAPES categories and All Research accross the 9 categories.

24 Figure 16 Health Sciences Health Sciences Biological Sciences Earth and Exact Sciences Agricultural Engineering All Research

Universidade Federal do ABC (UFABC) 1.06 0.74 1.95 - 0.95 1.68 Universidade Federal de Sao Joao del-Rei 0.65 0.61 2.53 1.29 0.95 1.54 Universidade Federal de Juiz de Fora 0.96 0.66 1.89 1.03 0.69 1.30 Universidade Federal de Sergipe 2.68 0.74 0.70 0.71 0.73 1.28 Universidade Federal de Pelotas 1.72 0.68 1.59 0.58 0.85 1.15 Universidade Federal de Santa Catarina (UFSC) 1.98 0.84 0.87 0.98 0.72 1.13 Universidade do Estado do Rio de Janeiro 0.91 0.72 1.37 0.63 1.09 1.06 Universidade Federal de Sao Paulo (UNIFESP) 1.17 0.94 0.73 0.68 0.87 1.06 Universidade Federal de Minas Gerais 1.56 0.90 0.80 0.93 0.79 1.03 Universidade Estadual de Campinas 0.86 0.94 1.23 0.87 0.74 1.03 Universidade de Sao Paulo 1.18 0.91 1.10 0.59 0.67 1.02 Universidade Federal do Rio Grande do Sul 1.50 0.78 0.97 0.88 0.82 1.02 Universidade Federal do Rio Grande do Norte 0.81 0.81 1.43 0.87 0.85 1.02 Universidade Federal do Rio de Janeiro 0.96 0.87 1.24 0.55 0.81 0.98 Universidade de Brasilia 1.50 0.86 0.64 0.59 0.87 0.90

Category Normalized Citation Impact of the 15 universities with highest impact in 5 subject categories and All Research Impact accross the 9 CAPES categories.

Figure 16 with citation impact shows which are the highest impact fields across that the most productive universities Brazil. The exceptions are above world do not always produce the most highly average impact in Agricultural Sciences cited research. Darker shades in Figure at Universidade Federal de São Joao 16 indicate a field with higher CNCI and del-Rei and Engineering at Universidade lighter shades indicate lower CNCI at that do Estado do Rio de Janeiro. university. A university with a negligible number of papers in an area is indicated Smaller to medium sized universities are by a “–”. Universidade Federal do ABC doing well in terms of citation impact. (UFABC) is the only one in this list that Universities in São Paulo State do not has too few publications in Agricultural dominate this list of universities ranked Sciences to compute a CNCI. by overall CNCI shown on the right in Figure 16. As mentioned above, high Generally, universities have one or two research output is not associated with high areas of high citation impact that is citation impact. The top three universities above the Brazilian or world average. in research output (Figure 15) are in the Most frequently these areas are Health bottom half of the universities in this Science and Exact and Earth Sciences citation impact ranking (Figure 16).

25 Figure 17 Health Sciences Health Sciences Biological Sciences Earth and Exact Sciences Agricultural Engineering All Research

Empresa Brasileira de Pesquisa Agropecuaria (Embrapa) 325 3,737 1,089 5,451 504 9,598 Fundacao Oswaldo Cruz 5,264 4,983 663 560 138 9,195 Centro Brasileiro de Pesquisas Fisicas 16 26 1,969 4 233 2,097 Institute Nacional de Pesquisas da Amazonia 84 1,438 205 387 28 1,898 Instituto Nacional de Pesquisas Espaciais (INPE) 25 161 1,406 95 338 1,817 Comissao Nacional de Energia Nuclear (CNEN) 272 153 738 37 801 1,435 Hospital Israelita Albert Einstein 1,142 278 23 19 28 1,345 Instituto Butantan 450 1,046 90 99 31 1,294 Comando-Geral de Tecnologia Aeroespacial (CTA) 30 17 656 4 570 1,063 Instituto Tecnologico de Aeronautica (ITA) 25 14 648 4 555 1,039

Research output of 10 Research Institutes in five CAPES categories and All Research accross 9 categories.

Figure 17 shows the numbers of papers Nacional de Pesquisas Espaciais (INPE). in five CAPES categories obtained Physics and engineering are the focus from the Web of Science published by of ComisSão Nacional de Energia researchers affiliated with these 10 research Nuclear (CNEN), Comando-Geral de institutes between 2013 and 2018. In Tecnologia Aeroespacial (CTA), and spite of their specialization in one or at Instituto Tecnologico de Aeronautica most two research areas, the research (ITA) (Figure 17). While Agricultural capacity of the research institutes is much Sciences papers account for more than smaller than that of the universities. The half the output of Empresa Brasileira de research output of the most productive Pesquisa Agropecuaria (Embrapa), it is research institution, Empresa Brasileira also a substantial producer of research de Pesquisa Agropecuaria (Embrapa), in Biological Sciences and to a lesser is less than one sixth the research extent in Earth and Exact Sciences. No output of Universidade de São Paulo. other research institute seems to have a major focus Agricultural Sciences, As might be expected, physics dominates except for the Instituto Nacional de the research output of Centro Brasileiro Pesquisas da Amazonia, where it is the de Pesquisas Fisicas and Instituto second largest research category.

26 Figure 18 Health Sciences Health Sciences Biological Sciences Earth and Exact Sciences Agricultural Engineering All Research

Centro Brasileiro de Pesquisas Fisicas - - 2.23 - 0.80 2.21 Instituto Nacional de Pesquisas Espaciais (INPE) - 1.49 2.52 0.54 0.95 2.19 Hospital Israelita Albert Einstein 1.75 1.09 - - - 1.62 Fundacao Oswaldo Cruz 1.21 0.98 0.75 0.68 0.86 1.07 Instituto Nacional de Pesquisas da Amazonia 0.70 0.83 1.62 0.71 - 0.91 Empresa Brasileira de Pesquisa Agropecuaria (Embrapa) 1.17 0.75 1.06 0.62 1.08 0.72 Instituto Butantan 0.80 0.70 0.58 0.66 - 0.68 Instituto Tecnologico de Aeronautica (ITA) - - 0.78 - 0.60 0.65 Comando-Geral de Tecnologia Aeroespacial (CTA) - - 0.78 - 0.60 0.65 Comissao Nacional de Energia Nuclear (CNEN) 0.47 0.67 0.62 - 0.87 0.58

Category Normalized Citation Impact (CNCI) of 10 top research institutions in five CAPES categories

In Figure 18 the research institutions are where the highest CNCI is 1.68. However, in ranked by their overall CNCI. Both of the two terms of citation impact, the universities have most productive research institutes, Centro greater depth, where 13 of the 15 universities Brasileiro de Pesquisas Fisicas and Instituto have overall CNCI above the world average Nacional de Pesquisas Espaciais (INPE), of 1.0, while only four of the ten research focus on Exact and Earth Sciences with institutions score above 1.0. In common CNCI of 2.23 and 2.52 respectively. Their with the universities, the research institutes overall CNCI of 2.24 and 2.1919 is significantly with the highest number of publications higher than that of any of the universities, do not have the highest citation impacts.

19 It is likely that these high citation impacts are distorted because they include several massively multi-authored papers. 27 For instance, there are 95 papers published with authors from INPE whose CNCI is 24. How does Brazil support research infrastructure?

How does Brazil enhance Coordination for the Improvement of research communication? Higher Education Personnel (CAPES) and the National Council for Scientific and Technological Development (CNPq). In addition to contributing to the expansion of the arts, humanities, With multilingualism21 as a key feature, engineering and scientific knowledge the SciELO network covers and indexes base and training and educating the research output in Portuguese, Spanish, workforce for the next generation, and English from Latin America, Portugal, institution building and supporting the Spain, and South Africa. A distinguishing scientific infrastructure are also valuable feature of SciELO is its emphasis on contributions to the scientific enterprise. open access and that that all journals in In 1998, Brazil launched the Scientific its collection be indexed in the Directory Electronic Library Online (SciELO)20, of Open Access Journals (DOAJ). developed by São Paulo Research Foundation (FAPESP) in partnership While there is some overlap between with BIREME, the Latin American and SciELO and the Web of Science Core Caribbean Center on Health Sciences Collection, access to SciELO can Information. It continues to flourish also be obtained through the Web of with additional support from the Science regional hosted collections.

20 https://scielo.org/, http://www.scielo.br/ accessed August 21, 2019 21 Meneghini, R, Packer, A.L. (2007). Is there science beyond English? Initiatives to increase the quality and visibility of non-English publications might help 28 to break down language barriers in scientific communication. EMBO Reports, 8(2): 112–116. Available at http://bit.ly/2Mn43I5, accessed August 21, 2019 Findings and conclusions

During the six-year window, 2013-2018, projects such as Sirius, the new Brazilian the volume of Brazil’s research output has synchrotron light source, not only promote continued to grow. Brazil has maintained international collaboration but can also its position as the 13th largest producer attract international research funding. of research publications globally. Among the BRICS countries, this level of output is Policies that foster university-industry below that of China and India and above collaborations have borne fruit, resulting that of Russia and South Africa. Gross in exponential growth in papers co- Domestic Expenditures on Research authored with researchers from industry. and Development (GERD) as a percent A third of these collaborations are of Brazil’s Gross Domestic Product with Brazilian companies and the rest (GDP) has ranged between 1.1 and 1.4% with multinationals. Petrobras is the according to the most recent (2012- strongest collaborator with universities 2016) data22 available, which has not kept and the pharmaceutical industry also up with expectations of a decade ago has a long history of collaboration. that it might reach two percent. Except Public universities are at the forefront for the potentially volatile numbers of these collaborations with industry. for the most recent two years, the percentage of Brazilian papers among Public organizations are also doing well in the top one percent of cited papers in terms of research output and its citation the world has exceeded one percent. impact (CNCI). The 15 most productive - all of them public - organizations include The make-up of Brazilian research reveals 13 universities and two specialized activity and excellence concentrated in research institutes. Of these 15, 11 have a fields which have received targeted sector CNCI above the Brazil average and eight investment. For instance, although the have a CNCI above the world average. Agricultural Sciences as a research field has low citation impact, disaggregating In addition to promoting scholarly research that into its constituent components shows and education, Brazilian public policy also rapid growth in sub-categories that have supports scholarly communication, which above world-average citation impact. has yielded SciELO, the electronic library Additional investments in such areas that has grown beyond Brazil to 13 other would enhance international visibility and countries and will soon include scholarly citation impact. Similarly, investments in publications from a total of 17 countries.

29 22 http://data.uis.unesco.org/index.aspx?queryid=74 accessed on August 21, 2019 Appendix 1

Bibliometrics and ‘cite’ earlier work relevant to the material citation data being reported. New papers are cited in their turn. Papers that accumulate more citations are thought of as having Bibliometrics are about publications greater ‘impact’, which is interpreted as and their citations. The academic field significance or influence on their field. emerged from ‘information science’ and Citation counts are therefore recognized as now usually refers to the methods used a measure of impact, which can be used to to study and index texts and information. index the excellence of the research from a particular group, institution or country. Publications cite other publications. These citation links grow into networks, and their The origins of citation analysis as a tool that numbers are likely to be related to the could be applied to research performance significance or impact of the publication. can be traced to the mid-1950s, when The meaning of the publication is Eugene Garfield proposed the concept determined from keywords and content. of citation indexing and introduced Citation analysis and content analysis the Science Citation Index, the Social have therefore become a common part Sciences Citation Index and the Arts & of bibliometric methodology. Historically, Humanities Citation Index, produced bibliometric methods were used to trace by the Institute of Scientific Information relationships amongst academic journal (now part of Clarivate Analytics). citations. Now, bibliometrics are important in indexing research performance. We can count citations, but they are only ‘indicators’ of impact or quality Bibliometric data have particular – not metrics. Most impact indicators characteristics of which the user should use average citation counts from be aware, and these are considered here. groups of papers, because some individual papers may have unusual Journal papers (publications, sources) or misleading citation profiles. These report research work. Papers refer to or outliers are diluted in larger samples.

30 Data source The Web of Science Group has extensive experience with databases on research The data used in this analysis came from inputs, activities and outputs and the Web of Science database which give has developed innovative analytical access not only to journals but also to approaches for benchmarking and conference proceedings, books, patents, interpreting international, national websites, and chemical structures, and institutional research impact. compounds and reactions. Web of Science has a unified structure that integrates all data and search terms together and therefore provides a level of comparability not found in other databases. It is widely acknowledged to be the world’s leading source of citation and bibliometric data. The Web of Science focuses on research published in journals, conferences 22 and books in science, medicine, arts, There are 22 fields in Essential humanities and social sciences. Science Indicators and 254 The Web of Science was originally created fields in Web of Science. as an awareness and information retrieval tool, but it has acquired an important primary use as a tool for research evaluation, using citation analysis and Database categories bibliometrics. Data coverage is both current and retrospective in the sciences, The source data can be grouped in various social sciences, arts and humanities, in classification systems. Most of these are based some cases back to 1900. Within the on groups of journals that have a relatively high research community this data source was cross-citation linkage and naturally cluster previously referred to by the acronym ‘ISI’. together. Custom classifications use subject maps in third-party data such as the OECD Unlike other databases, the Web of categories set out in the Frascati manual. Science and underlying databases are selective, that is: the journals abstracted The Web of Science Group frequently are selected using rigorous editorial uses the broader field categories in the and quality criteria. The authoritative, Essential Science Indicators system multidisciplinary content covers over and the finer journal categories in the 18,000 of the highest impact journals Web of Science. There are 22 fields in worldwide, including Open Access Essential Science Indicators and 254 journals, and over 180,000 conference fields in Web of Science. In either case, proceedings. The abstracted journals our bibliometric analyses draw on the full encompass the majority of significant, range of data available in the underlying frequently cited scientific reports and, database, so analyses in our reports will more importantly, an even greater differ slightly from anything created ‘on proportion of the scientific research the fly’ from data in the web interface. output which is cited. This selective process ensures that the citation counts Most analyses start with an overall view remain relatively stable in given research across the data, then move to a view across fields and do not fluctuate unduly from broad categories and only then focus in at year to year, which increases the usability a finer level in the areas of greatest interest of such data for performance evaluation. to policy, program or institutional purpose.

31 Assigning papers to addresses

A paper is assigned to each country and each institution whose address appears at least once for any author on that paper. One paper counts once and only once for each assignment, however many address variants may occur for the country or institution. No weighting is applied.

For example, a paper has five authors, thus:

Author Institution Country

Gurney, KA Univ Leeds UK Counts for Univ Leeds Counts for UK

Adams, J Univ Leeds UK No gain for Univ Leeds No gain for UK

Kochalko, D Univ C San Diego USA Counts for UCSD Counts for USA

Munshi, S Gujarat Univ India Counts for Gujarat Univ Counts for India

Pendlebury, D Univ Oregon USA Counts for Univ Oregon No gain for USA

So this one paper with five authors by institution and country makes little would be included once in the tallies difference to the conclusions of an analysis for each of four universities and once in at an aggregate level. Such fractional the tallies for each of three countries. analysis can introduce unforeseen errors in the attempt to create a detailed but Work carried out within the Web of uncertain assignment. Partitioning credit Science Group, and research published would make a greater difference at a elsewhere, indicates that fractional detailed, group level but the analysis weighting based on the balance of authors can then be manually validated.

32 Citation counts Other papers can be very highly cited: hundreds, even thousands of times. A publication accumulates citation Again, there are multiple reasons for counts when it is referred to by more this. Most frequently cited work is being recent publications. Some papers get recognized for its innovative significance cited frequently and many get cited and impact on the research field of rarely or never, so the distribution which it speaks. Impact here is a good of citations is highly skewed. reflection of quality: it is an indicator of excellence. But there are other papers Why are many papers never cited? which are frequently cited because their significance is slightly different: they Certainly, some papers remain uncited describe key methodology; they are a because their content is of little or no thoughtful and wide-ranging review of impact, but that is not the only reason. a field; or they represent contentious It might be because they have been views which others seek to refute. published in a journal not read by researchers to whom the paper might be Citation analysis cannot make value interesting. It might be that they represent judgments about why an article is important but ‘negative’ work reporting uncited nor about why it is highly a blind alley to be avoided by others. The cited. The analysis can only report the publication may be a commentary in an citation impact that the publication editorial, rather than a normal journal has achieved. We normally assume, article and thus of general rather than based on many other studies linking research interest, or it might be that the bibliometric and peer judgments, that work is a ‘sleeping beauty’ that has yet high citation counts correlate on average to be recognized for its significance. with the quality of the research.

700

600

500

400 Frequency

200

100

0

Uncited Citation at end of 2013 for UK cell biology papers published in 2009 Maximum

The figure shows the skewed distribution of more or less frequently cited papers from a sample of UK authored publications in cell biology. The skew in the distribution varies from field to field. It is to compensate for such factors that actual citation counts must be normalized, or rebased, against a world baseline.

33 We do not seek to account separately for the journal category Materials Science, the effect of self-citation. If the citation Biomaterials. Papers less than eight years count is significantly affected by self- old are, on average, still accumulating citation then the paper is likely to have additional citations. The citation count goes been infrequently cited. This is therefore on to reach a plateau for older sources. only of consequence for low impact activity. Studies show that for large samples The graph shows that the percentage at national and institutional level, the of papers that have never been cited effect of self-citation has little or no effect drops over about five years. Beyond on the analytical outcomes and would five years, between 5% and 10% or not alter interpretation of the results. more of papers remain uncited.

Account must be taken of these time factors in comparing current research Time factors with historical patterns. For these reasons, it is sometimes more appropriate to Citations accumulate over time. Older use a fixed five-year window of papers papers therefore have, on average, more and citations to compare two periods citations than more recent work. The than to look at the longer-term profile graph below shows the pattern of citation of citations and lack of citations for a accumulation for a set of 33 journals in recent year and an historical year.

40 60

30 45

20 30

10 15 Percentage of papers uncited at the end 2013 at papers uncited of Percentage Impact (citations end 2013/number of papers) of end 2013/number Impact (citations

0 0 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2003 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Year of publication Citations/paper Uncited papers

34 Discipline factors Some journals may be considered to be part of the publication record Citation rates vary between disciplines for more than one research field. and fields. For the UK science base as As the example below illustrates, a whole, ten years produces a general the journal Acta Biomaterialia is plateau beyond which few additional assigned to two journal categories: citations would be expected. On the Materials Science, Biomaterials whole, citations accumulate more rapidly and Engineering, Biomedical. and plateau at a higher level in biological sciences than physical sciences, and Very few papers are not assigned to natural sciences generally cite at a any research field and as such will higher rate than social sciences. not be included in specific analyses using normalized citation impact data. Papers are assigned to disciplines (journal The journals included in the Web of categories or research fields) by the Web Science databases and how they are of Science, bringing cognate research selected are detailed here: clarivate. areas together. Before 2007, journals were com/webofsciencegroup/solutions/ assigned to the older, well established webofscience-core-collection- Current Contents categories which were editorial-selection-process/ informed by extensive work by Thomson and with the research community since Some journals with a very diverse the early 1960s. This scheme has been content, including the prestigious journals superseded by the 254 Web of Science Nature and Science were classified as journal categories which allow for greater Multidisciplinary in databases created disaggregation for the growing volume of prior to 2007. The papers from these research which is published and abstracted. Multidisciplinary journals are now re- assigned to more specific research fields Papers are allocated according to the using an algorithm based on the research journal in which the paper is published. area(s) of the references cited by the article.

35 Normalized Citation Impact cited more frequently than an article in the same field, but editorials and meeting Because citations accumulate over time abstracts are rarely cited and citation rates at a rate that is dependent upon the field for conference proceedings are extremely of research, all analyses must take both variable. The most common normalization field and year into account. In other words, factors are the average citations per because the absolute citation count paper for (1) the year and (2) either the for a specific article is influenced by its field or the journal in which the paper field and by the year it was published, was published. This normalization is also we can only make comparisons of referred to as ‘rebasing’ the citation count. indexed data after normalizing with reference to these two variables. Impact is therefore most commonly analysed in terms of ‘normalized citation We only use citation counts for reviews and impact’, or nci. The following schematic articles in calculations of impact, because illustrates how the normalized citation document type influences the citation impact is calculated at the paper count. For example, a review will often be level and journal category level.

Design of scaffolds for blood vessel tissue engineering using a multi-layering electrospinning technique (2005) Acta Biomaterialia 1: 575-582

Materials Science, Biomaterials Engineering, Biomedical Impact normalized to world average Impact normalized to world average citations/paper in the Materials Science, citations/paper in the Engineering, Biomaterials in 2005 = 5.6 Biomedical journal category in 2005 = 6.5

This article in the journal Acta Biomaterialia Most papers (nearly two-thirds) are assigned is assigned to two journal categories: to a single journal category while a minority Materials Science, Biomaterials and of them are assigned to more than five. Engineering, Biomedical. The world average baselines for, as an example, Materials The average (normalized) citation impact science, Biomaterials are calculated by can be calculated at an individual paper summing the citations to all the articles level where it can be associated with and reviews published worldwide in the more than one journal category. It can journal Acta Biomaterialia and the other 32 also be calculated for a set of papers journals assigned to this category for each at any level from a single country to an year and dividing this by the total number individual researcher’s output. In the of articles and reviews published in the example above, the average citation journal category. This gives the category- impact of the Acta Biomaterialia paper specific normalized citation impact (in can be expressed as ((5.6 + 6.5)/2) = 6.1. the above example the category-specific nciF for Materials Science, Biomaterials World average impact data are sourced is 5.6 and the category-specific nciF for from the Web of Science National Science Engineering, Biomedical is higher at 6.5). Indicators baseline data for 2015.

36 Mean Normalized of relative citation rates, normalized Citation Impact (rebased) to world average.23 An Impact Profile enables an examination and analysis of the strengths and weaknesses Research performance has historically been of published outputs relative to world indexed by using average citation impact, average and relative to a reference profile. usually compared to a world average that This provides much more information accounts for time and discipline. As noted, about the basis and structure of research however, the distribution of citations performance than conventionally amongst papers is highly skewed because reported averages in citation indices. many papers are never cited while a few papers accumulate very large citation Papers which are “highly cited” are often counts. That means that an average may be defined in our reports as those with an misleading if assumptions are made about average citation impact greater than the distribution of the underlying data. or equal to 4.0, i.e. those papers which have received greater than or equal to In fact, almost all research activity metrics are four times the world average number skewed: for research income, PhD numbers of citations for papers in that subject and publications there are many low activity published in that year. This differs from the values and a few exceptionally high values. database of global highly cited papers, In reality, therefore, the skewed distribution produced by the Web of Science which means that average impact tends to be are the top one percent most frequently greater than and often significantly different cited for their field and year. The top from either the median or mode in the percentile is a powerful indicator of distribution. This should be borne in mind leading performance but is too stringent a when reviewing analytical outcomes. threshold for most management analyses.

The proportion of uncited papers in a dataset can be compared to the Impact Profiles benchmark for the UK, the USA or any other country. Overall, in a typical ten-year We have developed a bibliometric sample, around one-quarter of papers methodology that shows the proportion of have not been cited within the 10-year papers that are uncited and the proportion period; the majority of these are, of course, that lie in each of eight categories those that are most recently published.

37 23 Adams J, Gurney K & Marshall S (2007) Profiling citation impact: A new methodology. Scientometrics 72: 325-344 The Impact Profile histogram can be presented in several ways which are illustrated below.

A B

25 25

20 20

15 15

10 10

5 5

0 0 RBI ≥ 0.5 < 1 RBI ≥ 0.5 < 1 RBI > 0 < 0.125 < 0 > RBI RBI > 0 < 0.125 < 0 > RBI RBI ≥ 4 < 8 RBI ≥ 4 < 8 uncited RBI ≥ 0.125 < 0.25 RBI ≥ 0.25 < 0.5 uncited RBI ≥ 0.125 < 0.25 RBI ≥ 0.25 < 0.5 RBI ≥ 2 < 4 RBI ≥ 2 < 4 RBI ≥ 1 < 2 RBI ≥ 8 RBI ≥ 1 < 2 RBI ≥ 8 Percentage of output [insert years to – from] – to years output [insert of Percentage Percentage of output [insert years to – from] – to years output [insert of Percentage

C D

25 25

20 20

15 15

10 10

5 5

0 0 RBI ≥ 0.5 < 1 RBI ≥ 0.5 < 1 RBI > 0 < 0.125 < 0 > RBI RBI > 0 < 0.125 < 0 > RBI RBI ≥ 4 < 8 RBI ≥ 4 < 8 uncited RBI ≥ 0.125 < 0.25 RBI ≥ 0.25 < 0.5 uncited RBI ≥ 0.125 < 0.25 RBI ≥ 0.25 < 0.5 RBI ≥ 2 < 4 RBI ≥ 2 < 4 RBI ≥ 1 < 2 RBI ≥ 8 RBI ≥ 1 < 2 RBI ≥ 8 Percentage of output [insert years to – from] – to years output [insert of Percentage Percentage of output [insert years to – from] – to years output [insert of Percentage

A: is used to represent the total output histograms where the eye is drawn to the of an individual country, institution or data most offset to the right, but can be researcher with no benchmark data. less easy to interpret as categorical data. Visually it highlights the numbers of uncited papers (weaknesses) and D: illustrates the complexity of data highly cited papers (strengths). which can be displayed using an Impact Profile. These data show research B & C: are used to represent the output in defined journal categories total output of an individual country, against appropriate benchmarks: institution or researcher (client) against an appropriate benchmark dataset client, research field X; (benchmark). The data are displayed as client, research field Y; either histograms (B) or a combination client, research field Z; of histogram and profile (C). Version benchmark, research field X+Y; C prevents the ‘travel’ which occurs in benchmark, research field, Z.

38 Impact Profiles enable an examination and We cannot tell why papers are not cited. It is analysis of the balance of published outputs likely that a significant proportion of papers relative to world average and relative to a remain uncited because they are reporting reference profile. This provides much more negative results which are an essential information about the basis and structure of matter of record in their field but make research performance than conventionally the content less likely to be referenced reported averages in citation indices. in other papers. Inevitably, other papers are uncited because their content is An Impact Profile shows what proportion trivial or marginal to the mainstream. of papers are uncited and what proportion However, it should not be assumed that are in each of eight categories of relative this is the case for all such papers. citation rates, normalized to world average (which becomes 1.0 in this graph). There is variation in non-citation between Normalized citation rates above 1.0 indicate countries and between fields. For example, papers cited more often than world relatively more engineering papers tend average for the field in which that journal is to remain uncited than papers in other categorized and in their year of publication. sciences, indicative of a disciplinary factor but not a quality factor. While Attention should be paid to: there is also an obvious increase in the likelihood of citation over time, most • The proportion of uncited papers papers that are going to be cited will be on the left of the chart cited within a few years of publication.

• The proportion of cited papers either side of world average (1.0) What is the threshold • The location of the most common for “highly cited”? (modal) group near the center

• The proportion of papers in the The Web of Science Group has most highly cited categories to the traditionally used the term “Highly right, (≥4 x world, ≥8 x world). Cited Paper” to refer to the world’s one percent of most frequently cited papers, considering year of publication and field. After reviewing the outcomes of a What are uncited papers? number of analyses, we have chosen a more relaxed definition for our descriptive It may be a surprise that some journal and analytical work. We deem papers papers are never subsequently cited after that are in the world’s top 10% of most publication, even by their authors. This frequently cited papers, considering year accounts for about half the total global of publication and field, to be relatively output for a typical, recent 10-year period. highly cited for national comparisons.

39 Appendix 2

University- industry (>300,000 records), then analyzing the collaboration methodology organizations to which the authors were affiliated (>22,000), and then classifying among these the ones which were in the The analysis presented here uses business sector. In the end we had more data from the Web of Science, than four thousand organizations. At this obtained through searches point we ran a search looking for items in performed at the normal WoS which the authors were in one of the 4,000+ interface available to researchers. business sector organizations and each and any university (obtained in a separate list). While the database InCites carries data for the percentage of articles with industry co- Obtaining complete data is challenging. authorship, their data is incomplete as the The Web of Science (and Scopus) has an database is not yet able to classify correctly incomplete classification of business sector the nature of a large number of business organizations, so that their indicator on organizations in Brazil (and elsewhere, for co-authorships between universities and that matter). To obtain the data shown here industry undercounts the real size of the we devised a search routine especially collaborations. The presently accessible built to unveil the business sector in Brazil. database categorizes as “industry” The procedure involved obtaining the mostly multinational companies and has data for all scientific documents in the weak (or no) categorization of medium database with at least one author in Brazil and small Brazilian companies24.

24 This fact should not be construed as a criticism to the bibliometric bases. If anything, we might have criticism towards the way the databases are frequently used. These databases were initially developed to assist the Science Community in finding useful references to use in their work. They are heirs to the tradition initiated by the Science Citation Index, created by Eugene Garfield, as a database of information on publications, classified by sector, author, topic. Many of us scientists will remember the trips to the library to consult the Science Citation Index (and the Current Abstracts) to try and keep abreast of the new publications in our fields. Part of the challenge here is that the databases were not built to be ranking instruments. Still, they can assist 40 ranking or science policy studies, if used with due care. The Web of Science offers, however, search (e.g. companies funded by FAPESP’s tools that can be used to compensate Small Business Innovative Research for the lack of categorization. We Program, companies listed on the used the following methodology: newspaper Valor Econômico’s list of most innovative companies in Brazil). 1. We started with a search for all This allowed us to add more than 200 items with addresses in “Brazil” organization names to the search string. (or “Brasil”), obtaining a list of approximately 700,000 items. 6. Finally, we added the names of all entities considered by InCites as companies. 2. Then we used the “Analysis Tools” to obtain the list of institutions mentioned in 7. The list of entities had more than the “Address” field of each publication. 4,000 items at this point. This list had 22,000 entities. 8. Then it was necessary to eliminate 3. The list of entities was analyzed line by redundancies to comply with the line to identify the ones belonging to restriction of the search tool for the Web the business sector that were then used of Science that limits the number of to compose a search string applied search terms to about 6,000, apparently to the field “Organization Name”. considering the boolian operators in the calculation. Many companies appear 4. In addition to the list of search terms under similar names. In other cases, it obtained in (3) we added to the search was necessary to consolidate names: string items ending in terminations that for example, works from Petrobras characterize business sector entities CENPES appear under several entity such as LTDA, LLC, SpA, LTD, BV, INC names such as ‘Leopoldo Miguez’, and others, using wildcards for the initial Cenpes Petrobras, ‘Centro Leopoldo portion of the organization name. M’, and other combinations.

5. We added to the resulting list of search 9. Then the search string was built terms the names of organizations in with the companies’ names and the Brazil known for having R&D activities names of the universities in Brazil.

41 About Web of Science Group

Web of Science Group, organizes the world’s research information to enable academia, corporations, publishers and governments to accelerate the pace of research. It is powered by the Web of Science – the world’s largest publisher-neutral citation index and research intelligence platform. Its many well-known brands also include Converis, EndNote, Kopernio, Publons, ScholarOne and the Institute for Scientific Information (ISI). The ‘university’ of Web of Science Group, ISI maintains the knowledge corpus upon which the index and related information and analytical content and services are built; it disseminates that knowledge externally through events, conferences and publications and it carries out research to sustain, extend and improve the knowledge base. Web of Science Group is a Clarivate Analytics company.

Contact our experts today:

+1 215 386 0100 (U.S.) +44 (0) 20 7433 4000 (Europe) webofsciencegroup.com

© 2019 Clarivate Analytics. Web of Science Group and its logo, as well as all other trademarks used herein are trademarks of their respective owners and used under license.