July 2007 Assessment of Research Quality Institute for Logic, Language and Computation Institute for Logic, Language and Computation Address: Plantage Muidergracht 24 1018 TV Amsterdam The Netherlands Phone: +31 20 525 6051 Fax: +31 20 525 5206 Email: [email protected] Website: http://www.illc.uva.nl ILLC evaluatierapport.indd 1 01-10-2007 15:12:33 Evaluation report Institute for Logic, Language and Computation Research 2000–2005 Universiteit van Amsterdam July 2007 2 Table of contents page Introduction 4 Evaluation procedure 4 The Institute 6 The Research Programmes - Logic and Language 10 - Language and Computation 11 - Logic and Computation 13 Overview quantitative assessment 15 Conclusions and Recommendations 16 Appendix 1 Review Committee 17 Appendix 2 Programme of the Site Visit 20 Appendix 3 Bibliometric analysis 21 List of Abbreviations 22 3 Introduction The Institute for Logic, Language and Computation of the Universiteit van Amsterdam is an interdisciplinary institute. It is part of two faculties: Humanities and Science. For the first time a research assessment was made of ILLC as a whole, under the authority of the Uni- versity Board. The review committee, installed by the University Board, consisted of Prof. Dr. M.J. Steedman, University of Edinburgh, UK (chair) Prof. Dr. G. Chierchia, University of Milan, Italy, and Harvard University, USA Prof. Dr. J.W. Klop, Vrije Universiteit, and CWI Amsterdam, The Netherlands Prof. Dr. W. Pohlers, University of Münster, Germany. Drs. J. Heijn has been appointed as the committee’s secretary. In this document, the committee reports on its findings. The report reflects the collective opinion of the whole committee. Evaluation procedure The evaluation was performed according to the Standard Evaluation Protocol SEP 2003– 2009 for Public Research Organisations, as defined by VSNU, KNAW, and NWO. A comprehensive ILLC self-evaluation and outcome report had been circulated ahead of time, as prescribed in the SEP. Although not prescribed in the SEP, often a bibliometric analysis is made providing complementary information to the review committee. In the pre- sent evaluation procedure it was decided not to produce a bibliometric analysis. The reasons are given in Appendix 3. Following the instructions of the Standard Evaluation Protocol (SEP), ratings had to be given for the ‘Quality’, ‘Productivity’, ‘Relevance’ and ‘Vitality/Prospects’ of the groups and the research performed, and of the Institute as a whole. The five point numerical SEP scale for judging the ‘Quality’ of the work evaluated is as fol- lows. 5 – Excellent Work that is at the forefront internationally, and which most likely will have an important and substantial impact in the field. Institute is considered an international leader. 4 – Very good Work that is internationally competitive and is expected to make a significant contribution; nationally speaking at the forefront in the field. Institute is considered international player, national leader. 3 – Good Work that is competitive at the national level and will probably make a valuable contribu- tion in the international field. Institute is considered internationally visible and a national player. 4 2 – Satisfactory Work that is solid but not exciting, will add to our understanding and is in principle worthy of support. It is considered of less priority than work in the above categories. Institute is na- tionally visible. 1 – Unsatisfactory Work that is neither solid nor exciting, flawed in the scientific and or technical approach, repetitions of other work, etc. Work not worthy of pursuing. The descriptions of the five point SEP scale for ‘Productivity’, ‘Relevance’ and ‘Prospects’ are similar. The members of the review committee have confirmed their independence via an Independ- ence Form according to the usual SEP procedure. The site visit started on Sunday October 29, 2006 and ended on Wednesday November 1, 2006. Unfortunately, Prof. Chierchia could not attend for reasons of health. The committee spent two full days meeting with research groups, management and with PhD and MoL (Master of Logic) students. The committee also met with the Rector Prof. Mr. Dr. Paul van der Heijden, with Prof. Dr. Karel Gaemers (dean Faculty of Science), and with Prof. Dr. Willem Weststeijn (vice-dean Faculty of Humanities). A number of survey reports and in-depth reports on highlighted research were presented by the research groups. At the end of the site visit – for the full programme see Appendix 2 – the preliminary findings, still without the quantitative assessments, of the committee were orally communicated to the ILLC community by the committee’s chairman. His remarks on the self-evaluation document and the choice of presentation form of the re- search groups are repeated below. The Institute and its members are to be congratulated on the general presentation of the self-evaluation document. By and large, the achievements of the past and present pro- grammes were clearly articulated and accessible. We note as minor points for future self- evaluations that the presentation of the publications in terms of the old five-part pro- gramme and the body of the report in terms of the new structure made quantitative evaluation arduous for the reviewers. Moreover, the presentation of those same publica- tions for the five groups by year, subdivided into types of publication, exacerbated this difficulty. In future reports of this kind, publications should be presented for each group by category such as journal article, etc., for the entire period of review, rather than by year. [The committee received after the site visit a helpful revision of these data accord- ing to these specifications.] The document was also less than helpful in identifying ex- actly who is currently contributing exactly what to the programme: the listing of current staff on p. 12 does not distinguish students from academics, and includes members who appear to have left the Institute and are therefore at most adjuncts. The tables of past in- stitute staff are rather impenetrable, though the presentations helped to clarify much of this. In other respects the survey presentations by the individual groups tended to reca- pitulate the self-evaluation document. On the other hand, the research case studies were interesting and illuminating: in future, it may be a good idea to increase the time avail- able for the latter at the expense of the former. 5 THE INSTITUTE The Institute for Logic, Language, and Computation is a world-renowned centre for inter- disciplinary research and graduate education in its eponymous fields. It is unique in taking logic as the backbone of the broader field. It has created an ideal atmosphere for research, and has mounted a highly innovative Masters’ programme which attracts students from all over the world, many of whom continue to PhD. The committee’s impression of the calibre of the graduate students and the inspiring environment for graduate study offered by ILLC was extremely positive. The researchers of the Institute have been highly productive in terms of new results, high-quality publications, and grant income. Many are accounted lead- ers in their field. This productivity includes the graduate students, many of whom have achieved substantial publications and prizes in their own right on the basis of their disserta- tions. Any criticisms offered below should be interpreted in the context of this overall ex- tremely positive view, and are intended to constructively assist in the continuation of this excellence in the face of possible or actual threats. Quantitative Assessment of the Institute as a Whole Quality 5 Productivity 5 Relevance 5 Prospects 4 Leadership, Strategy and Policy Leadership has been very strong throughout the history of the Institute. The committee gained a very strong impression of harmony and satisfaction with the current leadership of Frank Veltman from all constituencies, despite the fact that the Institute bridges two facul- ties with very different traditions and cultures. Quality of the Resources, Funding Policies and Facilities ILLC runs on a mixture of UvA direct funding and external research grants and contracts. Over the past six years of the review period, the ratio of UvA to external funding has aver- aged about 55% to 45% (p. 19 of the self-evaluation document) – slightly better than the av- erage across UvA departments, which we are told is around 60/40. However, this average conceals some important changes over the relevant period. Firstly, there was a dip in new grants secured in 2002. The reasons for this are unclear. Secondly, the transfer of the Lan- guage and Inference Technology (LIT) group led by Maarten de Rijke from ILLC to the In- formatics Institute affected the funding balance in 2004. The loss of this high-earning group led to a temporary fall in newly secured grant income, and to a worsening of the ratio of di- rect to indirect funding. However, over the period from 2002 to the present, the rate at which new grants have been secured has more than replaced the missing LIT income. In particular, in 2005, grant income was almost 4M euros, of which around half came from NWO (already exceeding the previous level of income from that source, and including a 6 large VICI grant to Yde Venema), and half came from an EU Marie Curie RST award (the EU had not previously been a source of funding for ILLC). As a result, it is anticipated that the ratio of direct to indirect funding will be around 40/60 over the next cycle. Thus funding appears to be secure for the foreseeable future. Resources also seem fairly secure, in the sense that the committee received a clear impres- sion from its talks with the Rector, the dean of Science, and the vice-dean of Humanities that UvA and the faculties were supportive of the ILLC. However, we noted a number of concerns arising from allocation of teaching resource credit across faculty boundaries.