STRATEGIC REPORT on HFSPO

by the

INDEPENDENT SCIENTIFIC REVIEW COMMITTEE (ISRC)

September 29, 2018

The views expressed herein are those of the authors and do not necessarily reflect the position of their employers or affiliated organizations.

Short Bio Sketches of the ISRC members

Dame Bridget Ogilvie, Chair Former Director of the Wellcome Trust Chair of International Advisory Board, ARC Centre of Excellence for Electromaterials Science, University of Wollongong Deputy Chair, Sense About Science Council Member, St George's House Trust, Windsor Castle

Selected accomplishments: Fellow of the Royal Society Dame of the British Empire Fellow of Academy of Medical Sciences Fellow of the Australian Academy of Science Companion of the Order of Australia Establishment of Wellcome Trust Sanger Institute

Professor Nancy Ip Vice-President for Research and Graduate Studies, Morningside Professor of Life Science, and the Director of the State Key Laboratory of Molecular Neuroscience at The Hong Kong University of Science and Technology (HKUST).

Selected accomplishments: Member of the Chinese Academy of Sciences Member of the US National Academy of Sciences Member of the World Academy of Sciences, Member of the American Academy of Arts and Sciences. Councillor for the Society for Neuroscience, Member of the World Economic Forum Global Agenda Council National Natural Science Awards L’OREAL-UNESCO for Women in Science Award 10 Science Stars of China by Nature.

1 Dr. Jill Heemskerk Acting Director, National Institute of Biomedical Imaging and Bioengineering (NIBIB), National Institutes of Health.

Deputy Director, NIBIB, NIH

Selected accomplishments: Former Director for Research Administration, NIBIB, NIH Former Deputy Director for the Division of Adult Translational Research, National Institute of Mental Health, NIH Former Acting Director of the Office of Translational Research at the National Institute of Neurological Disorders and Stroke, NIH. Former Advisory Board member of the ALS Association, the Spinal Muscular Atrophy Foundation, and the Huntington’s Disease Society of America.

Professor Mats Ulfendahl Professor, Director of Research, Regional Council of Östergötland Chair, Swedish Society for Medical Research Chair, Delegation for Research of the Swedish Society of Medicine

Selected accomplishments Former Secretary-General for medicine and health, Swedish Research Council Former Director of the Center for Hearing and Communication Research, Karolinska Institutet. Chair of international expert panel for interim evaluation of EDCTP2 (European Commission) Former Chair of the management board of Joint Programming Initiative on Antimicrobial Resistance Honorary Doctor (Dr h.c.) of Odontology, Malmö University The Petrus and Augusta Hedlund 40-year anniversary award

HFSP involvement Program Grant 2003

2 Eric Westhof

Emeritus Professor, Université de Strasbourg (France). Institut de biologie moléculaire et cellulaire du CNRS, Strasbourg, France.

Selected accomplishments: Former Director of the Unit and the Institute (2005-2016). Former Vice-President for Research and Doctoral Studies at the University of Strasbourg (2007-2012). Member of European Molecular Biology Organization (EMBO) (1998), Deutsche Akademie der Naturforscher (LEOPOLDINA) (2000), the French Academy of Sciences (2011). Lifetime Achievement Award, International RNA Society, Kyoto (2016).

HFSP involvement: HFSP Research Grants 1997 and 2005.

3

EXECUTIVE SUMMARY

Main conclusions of the ISRC

 HFSP grants and fellowships are highly recognized and a real mark of distinction.  HFSPO is successful in its strategy to enhance and synergize national science funding programs.  HFSPO is international at every level.  The current policy of funding curiosity-driven, daring, interdisciplinary and international research, without any priority given to a scientific topic or area, has been particularly successful.

Main results of the Science Metrix Analysis

. All the indicators point to the high impact level of the scientific outcomes. . Bibliometric comparisons with other national or international funders, all much larger than HFSP, are stunningly convincing that the scientific returns on investment are significantly above world average. . The analysis demonstrates the very high quality of the review and selection processes. . The post-doctoral fellowships are an undoubted “success story”. . The analysis demonstrates that HFSP Research Grants are unique in combining impact, multidisciplinary collaboration and interdisciplinary research at the same time. . The funding amount and duration of the awards are no longer competitive relative to other funding programs.

Major recommendations of the ISRC

 Restore attractiveness of the awards by increasing their amount and duration. This will require an increase in the budget.  Focus on Grants and Fellowships.  Encourage repatriation of HFSP fellows. The Career Development Awards should cease and the salient features as well as the funds should be made available as an option for the long-term and cross-disciplinary fellows from the start of these awards.

Supporting comments

 Continue to build up the Alumni Network of HFSP worldwide to promote international understanding, scientific integration, and increase connectivity.  Continue to exploit Awardees Meetings to promote internationality. Invite board members and alumni.  Continue to make efforts to attract other countries that are key to the global scientific endeavor to join HFSPO.

HFSP grants and fellowships are highly recognized, very competitive and a real mark of 4 distinction.

 The main strategic objective of HFSP is to enhance and synergize the national science Preamble

This strategic report is intended for the Board of HFSPO in order to prepare the Intergovernmental Conference in 2019, which coincides with the 30th anniversary of the Program. The Secretary-General, Warwick Anderson, originated the idea and the organization of the ISRC. The Board of Trustees validated the process on 25 October 2016. Both the members of the ISRC and the reviewing organization were chosen among a list furnished by the Board of Trustees and the final choice and terms of reference approved by the Board by email on 13 April 2017. The Committee members met several times in person and during teleconferences (see The installation of the Independent Scientific Review Committee in the Appendix). At the outset the committee members would like to acknowledge the imaginative and visionary perspective of Japan about three decades ago when Hon. Yasuhiro Nakasone proposed HFSPO at the 1987 G7 summit in Venice. The idea of former Japanese Prime Minister Yasuhiro Nakasone was to establish a funding organization for basic research in the life sciences. The following words pronounced by Hon. Yasuhiro Nakasone in 1987 were discerning and far-sighted: “HFSP was meant to be a 21st century, long-term global research program fostering international cooperation for the benefit of mankind, under which courageous researchers from throughout the world and the spirit of challenge would form teams to analyze and discover various complex mechanisms of living organisms”.1

This Report is not meant primarily as an evaluation report, but instead aims at presenting a strategic analysis setting a vision that should go beyond maturation following three decades of very successful and pioneering activities.

In support of this effort, an independent analysis was commissioned from Science Metrix Inc.2 by the Secretary-General of HFSPO upon the advice of the ISRC. The analysis is quantitative and based on various criteria and data sets including interviews, case studies, surveys and extensive bibliometrics. The Independent Scientific Review Committee validated the methodology and the indicators used after several exchanges with Science Metrix agents. The staff at HFSPO participated in those exchanges, prepared and sent the necessary datasets and files from the HFSPO archives. We aim to answer the following questions:

 What is truly unique and distinctive about HFSP funding?  How successful and impactful is the HFSP funding?  What aspects of the funding schemes should be maintained, improved, amended or discontinued?

In this Final Report, the evaluation performed by Science Metrix provides the observational data on which the Independent Scientific Review Committee (ISRC) elaborated and built a forward- looking strategic vision with clear recommendations to the Board. In the present Report, results from the Science Metrix analysis will not be systematically considered or discussed, but references to it will be given when judged appropriate.

With a funding budget of HFSPO around $55 million/year, it is obvious that scientists worldwide depend essentially on national funding throughout their careers. HFSPO incorporates, therefore, as a main axiom that its central objective is to enhance and synergise the science promotion and funding strategies of the national funders. Consequently, we stress HFSP’s added value compared to what

1 From HFSP Strategic outlook Final.pdf 2 See Science-Metrix Inc 5 can be done at a national level. A second foundation is that the core element of the uniqueness of HFSPO is its in-built international character at every level, starting with its funders and members, to the peer review committee members, the awardees, and the scientific and managing staff of the organization. We will use this common thread throughout the document.

The Independent Scientific Review Committee (ISRC) is indebted to HFSPO staff for their very constructive help throughout. The members of the ISRC particularly would like to thank HSFPO staff members Guntram Bauer, Barbara Pauly, Geoff Richards and Jill Husser for their expert help during the whole evaluation process.

International recognition

In this regard, the main conclusions that are drawn from the interviews of high-level scientists3 are the following.

 Scientists concur to state strongly that HFSP grants and fellowships are highly recognized and a real mark of distinction4. When they have the choice, young scientists have chosen the HFSP fellowship because of the international recognition and fame associated with a HFSP Award. Steven Chu, Nobel Prize laureate in physics, who acknowledged the critical role of HFSP support in his early work on single molecule demonstrations using optical tweezers, gave an important example. Dr. Chu stressed the on-going need for support for science that has not yet become mainstream. As remarked by Eric Miska, “HFSP provides the financial glue to enable very different researchers, who might otherwise not have the opportunity to collaborate, to work on a creative project together.”  Scientists praised the thorough and well-conceived international peer-review processes set up by HFSPO. They all agreed that peer-review based on written reports followed by committee discussions and final evaluations is the most widely accepted process for making awards.  Scientists shared two perceived points regarding science funding worldwide: (i) funds are decreasing for daring, potentially paradigm shifting, and high impact research; (ii) national funds are moving away from basic and fundamental science to translational and applied research.

In short: “a premier program…among the very few funders of blue-sky, high-risk research” (Jennifer Doudna); “the most prestigious worldwide” (Elaine Fuchs).

Scientists emphasized also that the projects should remain curiosity-driven, daring, international and without any scientific priority. They acknowledged the well-conceived peer review processes maintained by the staff and the minimal administrative burden prior, during the award and afterwards5. HFSPO distinguishes itself by being an investment for the future and a way for national agencies to outsource risk.

3 See the document « Interviews with distinguished scientists 11-8 ». 4 On average, more than 800 letters of intent are received annually either for the grants or the fellowships with final success rates around 3% and 10%, respectively (see Figure 1 in Appendix). 5 A note on cost-benefit and efficiency of the programme: the permanent staff is 15 persons and the overhead personnel and running cost is less than 7% of the total budget. 6 Plurality in funding strategies strengthens science nationally and internationally. Plurality contributes to quality through competition between funders and applicants while allowing for a greater diversity in funder objectives and applicant perspectives. This ultimately leads to improved complementarity between funders and applicants and overall efficiency. HFSPO challenges scientists to change the way they think. Even when they fail to get the HFSPO award, the experience is constructive and rewarding. Although HFSPO also funds already well-endowed laboratories, this allows scientists from less prestigious laboratories or institutions to collaborate with the best.

The committee would like to add that science by nature is international and that there is a clear, albeit not easily measurable, value in diversity of thought to justify an international effort6.

Main results of the Science Metrix analysis

. All the indicators point to the high impact level of the scientific outcomes.

. Bibliometric comparisons with other national or international funders, all much larger than HFSPO, are stunningly convincing that the scientific returns on investment are significantly above world average.

Those two statements follow findings No 4 to 7 in the Science Metrix Analysis. See also the Comments on the Science Metrix Analysis in the Appendix.

Finding #4: Overall, evidence from the survey and bibliometric analysis pointed to a high degree of implementation of the proposed research by HFSPO awardees. Those who failed to achieve their initial objectives—primarily fellowship awardees—predominantly experienced a major change in research direction during the course of their project.

Finding #5: Overall, HFSPO was one among several funders supporting research that was highly impactful, especially in the areas most relevant to HFSP. HFSPO contributed to an increase in the scientific impact of fellows and Long-term fellows in particular. HFSPO awardees primarily contributed to advances in the field through the identification of new research questions, new concepts and novel research approaches or methods. HFSPO funding helped them to achieve major scientific advances such as shifts in paradigms and creation of new research fields/subfields. Long-term fellows generated translational (innovation) outcomes at a rate notably above the global average.

Finding #6: HFSP support contributed to an increase in the rate of international and intercontinental collaborations and the formation of new partnerships. However, only Research Grants and Long-term fellows saw the increase in their international collaborations materialize in the form of a growth in international publications (i.e., co-authored by researchers from at least two countries). Collaborations set up through Research Grants and Career development awards were largely centred on activities related to the planning and execution of research projects and did not involve many on-site interactions. Research partnerships provided awardees critical access to novel ideas, methods and techniques, as well as complementary expertise.

6 A recent Nature paper shows a correlation between openness of scientist exchanges and international collaborations with publication impact; Australia and Switzerland have a proportionally high impact because they are very open and internationally connected (Nature 550, 32 (2017)). 7 Finding #7: Research supported by HFSPO scored strongly in multidisciplinary collaboration and especially in interdisciplinary integration. HFSP contributed to an increase in the multidisciplinarity7 of the Long-term fellows and in the interdisciplinarity7 of the Cross- disciplinary fellowships development awards. Most Cross-disciplinary awardees changed discipline or research area; however, this change was most likely not driven by HFSPO because such a change was also observed with unsuccessful applicants. HFSPO-supported projects had an impact in research disciplines other than the life sciences, as well as on research areas within the life sciences other than those of the awardees. In fact, it was among the top funders in terms of impact in areas outside the life sciences.

We conclude that the preparation of an application for a Career development award seemed to be sufficient stimulus to change direction in research whether or not the applicant won the award.

. The analysis demonstrates the very high quality of the review and selection processes.

This statement follows from Conclusion #1 of the Science Metrix analysis: “HFSP reviewers and awardees were highly satisfied with the award selection process and the overall design of the program, which both compared well with those of other programs on most aspects.”

. The post-doctoral fellowships are an undoubted “success story”. Conclusions #5 and #6 state: “Among all awardees, the Long-term fellows were those who benefited most from the program and led to innovation outcomes notably above the global average. The Career development awardees were all leading or planning to lead their own lab, either in their home country or in the Career development awardee’s host country. HFSPO funding contributes, within the present research ecosystem, to filling an important funding gap for early career researchers who wish to carry out interdisciplinary research. Thus, Tables 9 and 10 show that 23% of the papers published by Cross-disciplinary awardees belong to the top 10% of the most multidisciplinary papers (much higher than any other funder). Further, Tables 11 and 12 show that 26% of the papers published by Cross- disciplinary awardees belong to the top 10% of the most interdisciplinary papers published (much higher than the other funders).

. The analysis demonstrates that HFSPO Research Grants (RG) uniquely combine impact, multidisciplinary collaboration and interdisciplinary research at the same time.

This is illustrated in tables 5, 6, 9, 10, 11 and 12 and on page 19 of the Science Metrix analysis, it is stated “very few of the other funders received high scores across all these dimensions—impact, multidisciplinary collaboration and interdisciplinary integration. The

7 From the Science Metrix Analysis:…”the term multidisciplinarity is used to refer to research teams that integrate members from various disciplines, while the term interdisciplinarity is used to refer specifically to the integration of expertise and approaches from different disciplines into a research paper (tracked by the diversity of scientific subfields represented among its references). These two dimensions may be independent: a multidisciplinary team may yet produce quite monodisciplinary research, while a monodisciplinary team may yet produce highly interdisciplinary research.” (Wagner, C. S. et al. (2011). Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature. Journal of Informetrics 5, 14–26).

8 Research Grant scheme was the only HFSP scheme that required all these dimensions at the same time. Therefore, HFSP may have represented an exclusive opportunity for RG awardees to carry out this unique kind of research; this could explain why RG projects were basically not executed in the absence of HFSP funding.” For example, Tables 9 and 10 show that 20% of the papers published by RG awardees belong to the top 10% of the most multidisciplinary papers as high as several funders like HHMI, CIHR and NHMRC). Tables 11 and 12 show that 16% of the papers published by Research Grant awardees belong to the top 10% of the most interdisciplinary papers published (much higher than the other funders).

. The funding amount and duration of the awards might not be competitive relative to other funding programs.

This statement follows finding #2 and conclusion #2 of the Science Metrix analysis. Although it is not easy to compare fellowship awards between programs because the organizations have particular conditions and availability, a quick survey shows that most international awards offer higher amounts (See Table 1 in the document submitted by Secretary-General and HFSPO Directors to the ISRC).

Scientific leadership

HFSP funding is a long-term investment for member countries and generates fundamental knowledge with potential economic value in the hosting countries. The committee would like to emphasize that although the expected output of HFSP funding is fundamental research, commercial impact will follow8 with return to science generally, to the host institution and the hosting country. HFSPO does not get involved in Intellectual Property and potential patents. We endorse fully the current policy and guidelines on IP as one of the benefits that HFSP provides to the host country and the awardee9.

To achieve high impact, it is key to be well connected, with frequent exchanges, visits or seminars between scientists at an international level. HFSP gives valuable and strong opportunities for setting and maintaining international exchanges and networking. HFSPO funding over three decades has established an international community of frontier researchers that have emerged as scientific leaders. Through the Awardees Meeting and the continued development of the alumni network, HFSPO should persist in building up this international cohort or community of frontier researchers with strong ties across scientific and national frontiers.

Thus, the role of HFSPO in science leadership is not to become a leader in science policy (which is properly among the roles of the national agencies), but to build up an international cohort of interconnected and innovative scientists that can assume academic leadership to promote the best science. Exchange visits and work abroad contribute enormously to the learning process of communicating and sharing scientific results, whatever the original culture6. HFSP builds up science internationally.

8 See, for example, « Patents, Pasteur, and Productivity | A Model for Promoting Scientific and Economic Growth at the National Institutes of Health » by M.J. Kalutkiewicz & R.L. Ehlman, Report 2017. 9 In this respect, it is worth noting that close to 10% of the papers published by HFSP fellows were cited in patents between 2009 and 2011 (see Table 7 of the Science Metrix report), almost three times the percentage of papers published in the life sciences. 9 The ISRC encourages HFSPO very strongly to pursue efforts to attract additional countries in the organization. The ISRC is pleased to note that Israel has agreed to join HFSPO. Because of its scientific successes, the participation of China would contribute to the international community of scientists and to the full recognition of Chinese science within the international networks. China has been mentioned by several of the leading scientists. South America countries, like Argentina, Chile, Uruguay or Brazil, as members would also be valuable members of the program. The ISRC suggests that countries could join as an international consortium, as the non-G7 EU members have done.

Recommendations

1. HFSPO is at a critical point in its history; such a successful organization is difficult to create but easy to destroy. Increasingly, successful applicants are declining HFSP awards in favour of more lucrative awards from other funders. Fifteen percent of fellows declined HFSP awards last year, indicating that the program has reached a tipping point. (See Table 1 in Appendix). Award amounts must be on par with the best national schemes to stay competitive. HFSP’s high reputation is not sufficient to attract applicants if the award amounts lag behind other funders. Achieving an increase in the amount and duration of awards by decreasing the total number of awards is not a viable solution because the success rate is already very low (around 3% for research grants and 10% for fellowships)4. For this reason, reducing the number of grants (now around 30) or fellowships (now around 75-80) would risk attrition and a collapse of the program.

Thus, in order to restore attractiveness of the awards an increase in the amount and duration of the awards is strongly recommended. This will require an increase in the budget.

The ISRC asked the HFSPO Secretariat to prepare simulations of the associated costs depending on the chosen trajectories. For example, assuming 30 grants and 80 fellowships an extension of LTFs to 4 years, CDFs to 5 years (including a “repatriation incentive”, see below), and Research Grants to 4 years would lead to a progressive budget increase up to a steady state level about 40% above the current contribution by 2025 (See the detailed document submitted by Secretary-General and HFSPO Directors to the ISRC). The ISRC considered the modeled durations and amounts to be reasonable and in the expected range. Besides, we feel strongly that it is essential that an increase in operational costs over the 6-year period be also included in the simulations.

In this critical moment, the Board faces the challenge to invest in HFSPO for its long-term survival. The ISRC urges the Board to take account of the fact that the international science community has increased greatly during the lifetime of the HFSPO, without evidence of diminution of quality, making it necessary for HFSPO to grow.

2. HFSPO should focus on grants and fellowships, keeping the schemes simple with as few constraints as possible in terms of nationality or country to support global science. However, some limitations could provide greater benefits to and maintain the commitment of member nations.

3. Career development is not a central mission of HFSPO. Governmental agencies and national funders follow their own guidelines to offer attractive conditions to encourage researcher careers. However, the ISRC acknowledges the understandable wish of HFSPO members that, having invested in the training of their most talented young researchers by sending them abroad, they capitalise on this investment by ensuring their return to the home country (such training is necessary as international experience and training in the best of laboratories is the key to success).

10 Acknowledging this wish, the preference would be to eliminate specific programs for career development and instead integrate the possibility of repatriation within the general fellowship scheme: a 3-year fellowship for frontier research with the possibility (not automatic and on a basis of an original project) of an additional 2 years of support in the home country (if it is a member) or another member country10.

Brain gain is in accord with the mission of HFSPO to promote excellence in science globally; repatriation support and incentive is a way of redressing a potential imbalance. Such a scheme would make the HFSPO fellowship more valuable, opening the repatriation component to all fellowship applicants from member countries. It would also provide an element of assurance to young post-docs that essential ties will be maintained at home. Governmental agencies and national funders follow their own guidelines to offer attractive conditions to encourage researchers.

Supporting comments

The committee would like to add some supporting comments and suggestions. HFSPO is encouraged to:

 continue to build up the Alumni Network of HFSP worldwide to promote international understanding, scientific integration, and increase connectivity;  continue to exploit Awardees Meetings to promote internationality. Invite board members and alumni;  continue to make efforts to attract other countries that are key to the global scientific endeavor to join HFSPO.

Concerning the eligibility criteria for grants around new collaborations, it was agreed that it was a distinctive feature of HFSP to promote new collaborations and that this should be reinforced.

Finally, the committee would like to praise HFSPO for the quality, thoroughness, fairness and balance of the selection processes leading to the awards. The ISRC encourages an already successful organisation to be thoughtful of innovative in scoring schemes and techniques. Already, the process may serve as a model for other funders. Along such lines, HFSPO may provide added value to funders as a pioneer organisation in the development of novel processes of innovative funding and evaluation instruments. In other words, because of its size and targeted objectives, HFSPO could exploit opportunities to be experimental in science funding11.

10 Along the lines of Option 2 in the « CDA working group report to the Board December 2017 ». 11 See « Fund ideas, not pedigree » in Nature 555, 143 (2018) for suggestions. Uri Alon from the Weizmann Institute suggested very convincingly this opportunity during a visit in Rehovot in March 2018. 11

Appendix

The installation of the Independent Scientific Review Committee After agreement to go ahead with the review in October 2016, the Secretary-General presented the Terms of Reference at the Board meeting March 2017 and these were approved in an email in April. The Board was asked for suggestions as to reviewing organisations in the HFSPO Members (Science Metrix was among the suggestions received) and for suggestions as to panel members. After consulting with the Steering Committee, the Secretary-General then appointed the four panel members (originally Roderick Pettigrew was appointed but he was later replaced by Jill Heemskerk who succeeded him at the NIBIB).

The Terms of Reference, approved by the Board at its 52nd meeting (March 2017), are: To report to the Board of Trustees as follows:  Assessment of the outcomes of the Program’s activities  Assessment of whether HFSPO is achieving its mission and stated aims, specifically in regard to: o the overall aims of the Program o the aims and mission of each scheme of the Program  Provision of any additional advice to the Board of HFSPO on the Program.

The ISRC met on several occasions:

9-11 October 2017, Paris

18-19 April 2018, HFSPO Secretariat, Strasbourg

17 September 2018 by teleconference

21-22 September 2018, Paris

12 Figure 1: Number of applications and awards (2008-2017) for research grants and fellowships. Data from HFSPO.

13 Comments on the bibliometric impact in the Science Metrix analysis

Here, we would like to explain and discuss the methodology that led to the main results from the bibliometric impact. For Tables 5, 6, 9, 10, 11, and 12 two data sets were built. The first one is called “life sciences in general” and the second one “life sciences most relevant to HFSP”. What is the reason for creating two data sets and how best to interpret what those statistical parameters measure? The general life sciences dataset is built by randomly sampling papers from the subfields in which HFSP papers are published. For example, very few HFSP papers were published in journals classified in physics. Accordingly, the constructed dataset will include a random number of physics papers (with a proportion representative of their occurrence within HFSP papers). In reality, the physics papers of HFSP likely represent interdisciplinary papers that have something to do with the life sciences (perhaps they were authored by Cross-disciplinary fellows). Of course, by randomly sampling physics papers, it is unlikely to capture similar papers from those funding more disciplinary projects. This is why another dataset was built (i.e., life sciences most relevant to HFSP). By picking the papers that referenced at least two HFSP papers, Science Metrix was again able to reproduce the distribution of HFSP papers across subfields. But this time, the retrieved papers are not random. They directly relate to the subject matter dealt with by HFSP grantees and, in this respect, it was called “the most relevant” dataset for comparison. Note that papers co-citing HFSP permits the retrieval of papers highly relevant to HFSP work since they use HFSP knowledge. The method does not, a priori, create a bias towards highly cited papers. Such an analysis looks at the research that uses HFSP outputs by using a strong link (i.e. co-citing HFSP funded papers). Within the ‘life sciences in general’, HFSPO ranked in the top 3 out of 28 funders based on the CDI indicator, which provides an overall score of citation impact (Table 5). Within the ‘life sciences most relevant to HFSP’, HFSPO research ranked 5th based on the 10% most highly cited papers (the impact metric more relevant to scientific excellence) with five other funders and 7th (tied with three national funders) among the 28 selected comparators based on the CDI indicator (Table 6). HFSPO had very good scores, but there are other funders performing well in this context. But the two tables do not report on the same ensemble of observations. Table 6 shows how other scientists funded by other organizations use HFSP production. In other words, the analysis shows that other funders do support similar work, that the funded scientists publish their results with high impact and cite already published research supported by HFSP. Thus, one can also view this as a means to assess what is the impact of research relying strongly on HFSP outputs. Indeed, when the scientists supported by these funders publish and co-cite papers of HFSP, they publish in very high- ranking journals. These findings may result from the relationships between novelty, publication ranking, and delay in citations as reported in the literature (see Figure 2 below)12. First conclusion, some of the excellent research supported by the most highly regarded funders relies on previous work initiated and supported by HFSP. The amplitude of the sampling bias discussed above is not easily quantifiable. Importantly, that sampling bias is a direct consequence from the very foundational objective of HFSP awards (interdisciplinarity and multidisciplinarity). This last point is clearly highlighted by comparing HFSPO rankings between Tables 9 and 10 on Multidisciplinarity and between Tables 11 and 12 on

12 See Novelty needs time in Nature 544, 411 (2017). First, there is a delay for risk research in citations. While novelty and risk increase the citation impact of papers, it does so in the longer run. Secondly, the citation distribution of “high risk/high gain” papers presents a higher mean but also a higher variance (thus covering the tails of high as well as low impact). Often highly novel papers are published in lower impact journals. 14 Interdisciplinarity assessments. As stated in the Science Metrix report “With respect to overall performance in multidisciplinarity, Tables 9 and 10 show that HFSP awardees outranked researchers supported by other comparable funders, placing within the top five for both overall multidisciplinary profile (multidisciplinarity distribution index, MDI) and for contributions to the most multidisciplinarity literature (highly multidisciplinary papers, HMP).” Further, the reports states “HFSP-supported research was also found to be the most interdisciplinary among the funders analyzed (Tables 11 and 12). HFSP ranked first, tied with the US NSF and the Australian Research Council for interdisciplinarity among the 28 comparators.

Second conclusion, for multidisciplinary and interdisciplinary, the comparisons show that the HFSPO rankings are both remarkable and independent of the data set chosen.

Figure 2: Novelty needs time. Figure from Nature 544, 411 (2017)12. The proxy used for evaluating risky research is the combination of cited journals and the likelihood of their co-occurrence. For example, an article in biological chemistry is more likely to cite a journal in biochemistry than in clinical psychiatry or soft matter physics. The authors found that almost 90% of the papers made no new combinations and among the 10% left more than half made only a single new combination of cited journals.

15 Table 1: Number of declined fellowships (2007-2017). BOT stands for Board of Trustees. Data from HFSPO.

% of awardees Total Total BOT Total Total no. of YEAR who declined applicants awardees awardees declines the fellowship

2007 668 100 100 8 8,00%

2008 636 107 100 12 11,21%

2009 672 120 119 14 11,67%

2010 647 86 86 10 11,63%

2011 762 85 85 5 5,88%

2012 747 85 85 7 8,24%

2013 750 70 83 9 12,86%

2014 747 90 88 10 11,11%

2015 788 75 75 9 12,00%

2016 697 75 75 8 10,67%

2017 776 85 72 13 15,29%

Total 7890 978 968 105 10,74% (2007-2017)

16