Assessment of Research Quality

July 2007

Institute for Logic, Language and Computation

Address: Plantage Muidergracht 24 1018 TV Amsterdam The Netherlands

Phone: +31 20 525 6051 Fax: +31 20 525 5206 Email: [email protected] Website: http://www.illc.uva.nl

ILLC evaluatierapport.indd 1 01-10-2007 15:12:33

Evaluation report Institute for Logic, Language and Computation Research 2000–2005 Universiteit van Amsterdam

July 2007

Table of contents

page

Introduction 4 Evaluation procedure 4

The Institute 6

The Research Programmes - Logic and Language 10 - Language and Computation 11 - Logic and Computation 13

Overview quantitative assessment 15

Conclusions and Recommendations 16

Appendix 1 Review Committee 17 Appendix 2 Programme of the Site Visit 20 Appendix 3 Bibliometric analysis 21

List of Abbreviations 22

Introduction

The Institute for Logic, Language and Computation of the Universiteit van Amsterdam is an interdisciplinary institute. It is part of two faculties: Humanities and Science. For the first time a research assessment was made of ILLC as a whole, under the authority of the Uni- versity Board.

The review committee, installed by the University Board, consisted of Prof. Dr. M.J. Steedman, University of Edinburgh, UK (chair) Prof. Dr. G. Chierchia, University of Milan, Italy, and Harvard University, USA Prof. Dr. J.W. Klop, Vrije Universiteit, and CWI Amsterdam, The Netherlands Prof. Dr. W. Pohlers, University of Münster, Germany.

Drs. J. Heijn has been appointed as the committee’s secretary.

In this document, the committee reports on its findings. The report reflects the collective opinion of the whole committee.

Evaluation procedure

The evaluation was performed according to the Standard Evaluation Protocol SEP 2003– 2009 for Public Research Organisations, as defined by VSNU, KNAW, and NWO. A comprehensive ILLC self-evaluation and outcome report had been circulated ahead of time, as prescribed in the SEP. Although not prescribed in the SEP, often a bibliometric analysis is made providing complementary information to the review committee. In the present evaluation procedure it was decided not to produce a bibliometric analysis. The reasons are given in Appendix 3.

Following the instructions of the Standard Evaluation Protocol (SEP), ratings had to be given for the ‘Quality’, ‘Productivity’, ‘Relevance’ and ‘Vitality/Prospects’ of the groups and the research performed, and of the Institute as a whole. The five point numerical SEP scale for judging the ‘Quality’ of the work evaluated is as fol- lows.

5 – Excellent Work that is at the forefront internationally, and which most likely will have an important and substantial impact in the field. Institute is considered an international leader. 4 – Very good Work that is internationally competitive and is expected to make a significant contribution; nationally speaking at the forefront in the field. Institute is considered international player, national leader. 3 – Good Work that is competitive at the national level and will probably make a valuable contribution in the international field. Institute is considered internationally visible and a national player.

2 – Satisfactory Work that is solid but not exciting, will add to our understanding and is in principle worthy of support. It is considered of less priority than work in the above categories. Institute is nationally visible. 1 – Unsatisfactory Work that is neither solid nor exciting, flawed in the scientific and or technical approach, repetitions of other work, etc. Work not worthy of pursuing.

The descriptions of the five point SEP scale for ‘Productivity’, ‘Relevance’ and ‘Prospects’ are similar.

The members of the review committee have confirmed their independence via an Independ- ence Form according to the usual SEP procedure.

The site visit started on Sunday October 29, 2006 and ended on Wednesday November 1, 2006. Unfortunately, Prof. Chierchia could not attend for reasons of health. The committee spent two full days meeting with research groups, management and with PhD and MoL (Master of Logic) students. The committee also met with the Rector Prof. Mr. Dr. Paul van der Heijden, with Prof. Dr. Karel Gaemers (dean Faculty of Science), and with Prof. Dr. Willem Weststeijn (vice-dean Faculty of Humanities).

A number of survey reports and in-depth reports on highlighted research were presented by the research groups. At the end of the site visit – for the full programme see Appendix 2 – the preliminary findings, still without the quantitative assessments, of the committee were orally communicated to the ILLC community by the committee’s chairman. His remarks on the self-evaluation document and the choice of presentation form of the research groups are repeated below.

The Institute and its members are to be congratulated on the general presentation of the self-evaluation document. By and large, the achievements of the past and present programmes were clearly articulated and accessible. We note as minor points for future self- evaluations that the presentation of the publications in terms of the old five-part programme and the body of the report in terms of the new structure made quantitative evaluation arduous for the reviewers. Moreover, the presentation of those same publications for the five groups by year, subdivided into types of publication, exacerbated this difficulty. In future reports of this kind, publications should be presented for each group by category such as journal article, etc., for the entire period of review, rather than by year. [The committee received after the site visit a helpful revision of these data according to these specifications.] The document was also less than helpful in identifying exactly who is currently contributing exactly what to the programme: the listing of current staff on p. 12 does not distinguish students from academics, and includes members who appear to have left the Institute and are therefore at most adjuncts. The tables of past institute staff are rather impenetrable, though the presentations helped to clarify much of this. In other respects the survey presentations by the individual groups tended to reca- pitulate the self-evaluation document. On the other hand, the research case studies were interesting and illuminating: in future, it may be a good idea to increase the time avail- able for the latter at the expense of the former.

THE INSTITUTE

The Institute for Logic, Language, and Computation is a world-renowned centre for interdisciplinary research and graduate education in its eponymous fields. It is unique in taking logic as the backbone of the broader field. It has created an ideal atmosphere for research, and has mounted a highly innovative Masters’ programme which attracts students from all over the world, many of whom continue to PhD. The committee’s impression of the calibre of the graduate students and the inspiring environment for graduate study offered by ILLC was extremely positive. The researchers of the Institute have been highly productive in terms of new results, high-quality publications, and grant income. Many are accounted leaders in their field. This productivity includes the graduate students, many of whom have achieved substantial publications and prizes in their own right on the basis of their disserta- tions. Any criticisms offered below should be interpreted in the context of this overall extremely positive view, and are intended to constructively assist in the continuation of this excellence in the face of possible or actual threats.

Quantitative Assessment of the Institute as a Whole

Quality 5 Productivity 5 Relevance 5 Prospects 4

Leadership, Strategy and Policy Leadership has been very strong throughout the history of the Institute. The committee gained a very strong impression of harmony and satisfaction with the current leadership of Frank Veltman from all constituencies, despite the fact that the Institute bridges two faculties with very different traditions and cultures.

Quality of the Resources, Funding Policies and Facilities

ILLC runs on a mixture of UvA direct funding and external research grants and contracts. Over the past six years of the review period, the ratio of UvA to external funding has aver- aged about 55% to 45% (p. 19 of the self-evaluation document) – slightly better than the average across UvA departments, which we are told is around 60/40. However, this average conceals some important changes over the relevant period. Firstly, there was a dip in new grants secured in 2002. The reasons for this are unclear. Secondly, the transfer of the Lan- guage and Inference Technology (LIT) group led by Maarten de Rijke from ILLC to the In- formatics Institute affected the funding balance in 2004. The loss of this high-earning group led to a temporary fall in newly secured grant income, and to a worsening of the ratio of direct to indirect funding. However, over the period from 2002 to the present, the rate at which new grants have been secured has more than replaced the missing LIT income. In particular, in 2005, grant income was almost 4M euros, of which around half came from NWO (already exceeding the previous level of income from that source, and including a

large VICI grant to Yde Venema), and half came from an EU Marie Curie RST award (the EU had not previously been a source of funding for ILLC). As a result, it is anticipated that the ratio of direct to indirect funding will be around 40/60 over the next cycle. Thus funding appears to be secure for the foreseeable future.

Resources also seem fairly secure, in the sense that the committee received a clear impression from its talks with the Rector, the dean of Science, and the vice-dean of Humanities that UvA and the faculties were supportive of the ILLC. However, we noted a number of concerns arising from allocation of teaching resource credit across faculty boundaries. We discuss these concerns at other places in this report. Facilities (which we take to refer to the buildings and other infrastructure provided for ILLC) are of a high standard, but are currently split over a number of different sites, creat- ing some practical difficulties for the collaboration across disciplines to which the Institute is dedicated. The current proposal for a satellite Science Campus out of the centre of Am- sterdam (Watergraafsmeer) threatens to exacerbate these problems, a point to which we also return in section Strengths and Weaknesses below, and in the sections concerning individual programmes.

Academic Reputation

The distinguished mathematical tradition of Brouwer, Beth, and Heyting in Amsterdam was continued by the founding of ILLC and by scholars such as Troelstra and van Benthem and has been recognized by many awards, including membership of the Academy of Europe and a Spinoza award to van Benthem. The current leadership are similarly widely recognized: they and many other researchers in the Institute are accounted leaders in their field. The academic reputation of the graduate programmes is similarly high. The quality of the Am- sterdam MoL is widely acknowledged, and around 80% of graduates go on to PhD. Around two-thirds of graduating PhDs in the review period obtained academic positions, of which around half were outside the Netherlands.

There have been complaints that not enough courses in mathematical logic had been offered. A possible reason is the time of vacancy of the chair, last held by Professor de Jongh. We expect, however, that the appointment of Jouko Väänänen will remedy this deficiency.

Social Relevance

ILLC pursues basic research and graduate education in fields that are highly relevant to progress and innovation in information technology. As such, its social relevance is hard to quantify, and is mainly evinced by placement of students and citations (or other use) of the outputs, discussed in the foregoing and in the sections dealing with individual programmes. In this connection, it is noteworthy that around 20% of students graduating in 2005 from the Master’s of Logic programme went directly to industry, mainly in IT, while most of the rest go on to PhD. Over the review period a roughly similar proportion of PhD graduates from ILLC have gone into industry, while the remainder have mainly gone to academic positions. Times to completion of PhD are low by Dutch standards, and are comparable to those at

leading US institutions such as Stanford, MIT, U. Penn, etc. The social relevance of ILLC goes considerably beyond graduate teaching and research. ILLC teaching faculty contribute to a wide range of undergraduate courses across the curriculum. There is considerable out- reach beyond the confines of UvA to the community at large, exemplified by van Ben- them’s lectures for children on Logic at NEMO museum, among other lectures for the general public. Stokhof contributes considerable public service as chairman of the board of the Division of Humanities of the Netherlands Organisation for Scientific Research (NWO).

Strengths and Weaknesses as Identified in the Self-Evaluation

A number of threats that may weaken the Institute were identified in the self-evaluation, and were remarked on during the review by a number of groups, including the graduate students. These concern problems created by the financial structure and in particular those concerning teaching credit, the plan to abolish the chair in computational linguistics, and the implications for a cross-faculty institute of the proposed move of the science faculty to a satellite campus. The problem with the chair in computational linguistics is in part a conse- quence of the first problem, since the problem stems from allocation of teaching resource credit across departmental/faculty boundaries. A solution might therefore be sought in terms of more explicit transfer of teaching income. If that is impossible, then the central role that computational linguistics plays in the Institute, both in terms of linkage to contiguous departments and in terms of its contribution to the empirical and probabilistic modelling ele- ments of its research skills, suggests to this committee that a solution must be found elsewhere, either in terms of the Science wing of ILLC taking on the responsibility for an activ- ity it already benefits from by providing a position, or by consciousness-raising on the Hu- manities side, since computational linguistics seems to this committee to be a vital component of a modern linguistic education. The problem posed by the proposed Science campus varies by programme. The strongest threat is to the groups in the Humanities, and there is a danger that moving ILLC to join Informatics there will cause it to split, destroying its unique interdisciplinary culture, and with it much of its value to UvA. The optimal solution would appear to be to locate Informatics and the entire ILLC at some location more accessible to the concerned Humanities departments, perhaps at Plantage Muidergracht.

The committee noted a striking under-representation of women among the faculty of the In- stitute. This should be a matter of concern in recruiting personnel.

Logic

LoLa TI LoCo ACT

CIL CSIP

Language Computation LaCo

Starting in 2006, the Institute has reorganized its old four programme structure into a new tripolar structure as indicated in the Figure. (Actually before 2004 there were five programmes. The fifth was LIT. ) The four programmes TI, CSIP, ACT and CIL were reconfigured as LoLa (= TI with part of CSIP), LaCo (= remainder of CSIP), and LoCo (= ACT together with CIL). The transition is in the opinion of the committee very well-motivated, and its execution seems flawless.

Research programme LOGIC and LANGUAGE

This new programme comprises a subset of the previous Cognitive Systems and Information Processing (CSIP) group and the previous Theory of Interpretation Group (TI). The Cognitive Systems component was in the past led by Michiel van Lambalgen, while the TI component was led by Jeroen Groenendijk and Martin Stokhof. The new programme is concerned with human reasoning and interpretation of natural language, and is led by Jeroen Groenendijk and Paul Dekker.

Quantitative Assessment Quality 5 Productivity 5 Relevance 4 Prospects 4

Leadership, Strategy and Policy The component groups have in the past been excellently led. The strategy has been outstandingly successful in terms of publications and academic recognition. There is good continuity, with Groenendijk staying in place and Dekker stepping up.

Quality of the Resources, Funding Policies and Facilities We heard complaints about excessive administrative loads from this group.

Publications and Publication Strategy There are several superb publications. Occasionally in the past there may have been a tendency to disperse results in somewhat unambitious journals, but this tendency has disappeared by now, and the publication strategy is sound.

Academic Reputation of the Group The group has a tremendous academic reputation, recently witnessed by Stokhof’s selection as member of the Royal Academy (KNAW).

Academic and Social Relevance Academic and social relevance of this research programme is very high – this holds in fact for all three programmes. See the elaboration on this issue above, for the whole institute.

The Future Notwithstanding the excellence of this work, the high quality and impact of the many publications, a question is raised in the self-evaluation about the consequences of the mainly philoso- phical turn of this new programme, concerning its connection to the field at large. The involvement of van Lambalgen’s cognitive neuroscience programme, which aims to link logic programming and neural computation will, as is claimed, address this potential problem to some extent. The committee appreciates the originality of this promising venture. However, it is not clear at present how the main body of work in semantics by the group as a whole can link to this representation. In particular, in view of developments elsewhere in semantics and artificial intelligence, it is surprising that there seems to be no involvement of probability in the core dy- namic semantic theories. This may be a problem that the group will have to take on board in future. 10

Research programme LANGUAGE and COMPUTATION

This programme under the new structure comprises the remainder of the previous Cognitive Systems and Information Processing (CSIP) programme, which was led by Remko Scha and Michiel van Lambalgen. The current programme, led by Scha and Khalil Sima’an, is concerned with wide coverage natural language parsing and music perception. (The cognitive systems component led by van Lambalgen has migrated to the new programme Logic and Language.)

Quantitative Assessment Quality 5 Productivity 5 Relevance 5 Prospects 4

Leadership, Strategy and Policy The programme has been excellently led to this point by Remko Scha. The strategy of en- gaging with the difficult practical task of parsing realistically-sized corpora has paid off in terms of funding and recognition. As a research policy, it represents a new and adventurous departure for ILLC in that it incorporates a strong element of statistical modeling that may well have future implications and linkage with other programmes within ILLC discussed below. The current joint leadership with Sima’an is a strong team for the future. However, we understand that Prof. Scha is close to mandatory retirement, and that renewal of his post, which is in the faculty of Humanities, is not certain. Despite the obvious strengths of the junior director of the programme, this must raise a question about its future. We regard this as a serious threat to the life of the Institute as a whole. We will return to this question below.

Quality of the Resources, Funding Policies and Facilities For these concerns we refer to the homonymous section on the whole Institute.

Publications and Publication Strategy There are many first-class papers. There is strong engagement with the competitions and bake-offs that drive this field, including TREC and CLEF. In the next phase, the strong CL group should concentrate on consolidating its reputation as a leading European group by concentrating on appearing more in top journals such as Computational Linguistics and Ar- tificial Intelligence and first-rank conferences such as ACL.

Academic Reputation of the Group The leaders Sima’an and Scha have a very high international standing as researchers, and the other members of the group are similarly highly-regarded internationally, in proportion to seniority.

Academic and Social Relevance The work on wide-coverage parsing, information retrieval, discourse semantics, and musical perception, among other topics covered, is at the leading-edge of international research.

Data oriented parsing (DOP) is recognized as a major parsing paradigm. The work of this group is also the most directly related to commercially and socially significant applications in natural language processing, information processing, and music processing. As such, it is of very considerable social relevance, above and beyond the less tangible but nonetheless real social importance of basic research and scientific training common to all of ILLC.

The Future Strong though it is, there is a question about the future viability of this programme. The senior leader Scha is approaching mandatory retirement, and there is a question about the post being replaced. We understand that the question arises because the post is in the faculty of Humanities, but much of the teaching of the incumbent has been to the benefit of the faculty of Science. We think that the institute as a whole has to take this problem on board. Compu- tational Linguistics is currently one of the strongest links between ILLC and the cognitive sciences, and one of the most internationally visible of its strengths. If it is indeed the case that Science has benefitted in logistic terms from Humanities’ investment in this chair, then the Science faculty-supported component of ILLC should consider taking on the burden. Alternatively, Humanities should consider whether they have correctly assessed the real value of the link to Science that the Computational Linguistics chair affords, which we be- lieve to be considerable. If the post does pass to Science, then the Linguists should ask themselves how they are going to sustain the linkage, say by making a new appointment in some related area, such as formal syntax or semantics. This is a vital strategic decision that may affect the life of the institute very profoundly, and will call for co-operation across a number of interest groups, and levels of governance in UvA. Notwithstanding the above concerns, we see this as a very strong group, and have immense confidence in the junior staff Sima’an, Honing and Bod.

Research programme LOGIC and COMPUTATION

This new programme subsumes the previous programmes Constructive and Intensional Logic (CIL) and Algorithms and Complexity Theory (ACT). CIL was led by Johan van Benthem and Dick de Jongh, ACT was led by Peter van Emde Boas and Leen Torenvliet. The new programme, led by Jouko Väänänen and Leen Torenvliet, has made a shift in emphasis from Constructivism to a diverse range of topics in set theory and model theory, including determinacy, infinitary combinatorics, co-algebras, generalized quantifiers, and abstract logics.

Quantitative Assessment Quality 5 Productivity 5 Relevance 5 Prospects 5

Leadership, Strategy and Policy Past leadership has been outstandingly successful in terms of publications and international academic recognition. The transition caused by the retirement of Anne Troelstra and Dick de Jongh, and van Benthem’s standing down as director, has been exceptionally well- handled, with a number of excellent new appointments, and leaves this group continuing to look very well prepared for the future.

Quality of the Resources, Funding Policies and Facilities For these concerns we refer to the homonymous section on the whole Institute.

Publications and Publication Strategy Especially in the first years of the evaluation period there are several publications in the most prestigious journals. A decrease in the productivity can be observed in the middle of the evaluation period. An improvement can be observed in the very near past. This is especially due to the younger scientists in the group. The journals are well selected. Overall the publication strategy is fine.

Academic Reputation of the Group The group, as a successor of the former programmes Constructive and Intensional Logic and Algorithms and Complexity Theory, owes much of its excellent academic reputation world- wide to their former, now retired leaders Anne Troelstra and Dick de Jongh as well as Johan van Benthem who is, though freed from explicit duties, apparently still contributing to the new group. With the choice of Jouko Väänänen as the successor of de Jongh again a world- wide renowned scientist entered the group. Vitányi, Apt, Buhrman are linked to CWI, and are world class researchers in their field. Vitányi is an authority on algorithmic complexity, with spectacular recent applications. Apt is an established leader since long (he joined the Academia Europaea recently). Buhrman is making a name in quantum computation. Even though they occupy only part-time positions at ILLC, they are a precious asset to ILLC. Venema is about to enter the group of top researchers in the area of coalgebras.

Academic and Social Relevance The relevance in all respects of this programme is undisputed. From academic perspective the programme carries on its great historical inheritance, with a vigorous renewing of its subjects. This programme is the part of the ILLC programme with the biggest mathematical emphasis. From the mathematical point of view it is doubtlessly excellent. The connection to the other parts of the research programme, however, is a matter of careful consideration. The influence of Apt and Vitányi in particular contributes much to its social relevance, with a number of promising practical applications.

The Future The programme has received a number of recent awards, including the prestigious VICI award to Venema. The linkage of the research in algorithms and complexity, including that on Kolmogorov complexity, to computer science and quantum computation, is impressive and promising. If there is a question about the direction of the group, it might concern the claims that are made concerning linkage of their very strong work on mathematical foundations to the broader field of cognitive sciences, which seem at times quite tenuous. But viewed from the more specific computational and foundational perspective, the committee recognizes the excellent quality which is also residing in this work.

OVERVIEW OF THE QUANTITATIVE ASSESSMENT

Quality: 5 LOGIC and LANGUAGE Productivity: 5 Jeroen Groenendijk Relevance: 4 Paul Dekker Prospects: 4

Quality: 5 LANGUAGE and COMPUTATION Productivity: 5 Remko Scha Relevance: 5 Khalil Sima’an Prospects: 4

Quality: 5 LOGIC and COMPUTATION Productivity: 5 Jouko Väänänen Relevance: 5 Leen Torenvliet Prospects: 5

Quality: 5 INSTITUTE FOR LOGIC, Productivity: 5 LANGUAGE AND COMPUTATION Relevance: 5 Frank Veltman Prospects: 4

CONCLUSIONS AND RECOMMENDATIONS

ILLC is world-renowned as a centre for interdisciplinary research and training, with an il- lustrious history behind it, and a very high level of current productivity in basic research outputs, and educational and social impact. This committee feels very confident that it will continue these functions at its present high standard, and to add to the lustre and reputation of the Universiteit van Amsterdam by so doing. We are particularly impressed by the way the Institute leadership has avoided any temptation to rest upon these laurels, and has moved with the times, making a number of strong appointments bringing in new ideas and ap- proaches. We see a number of threats to the Institute, arising from the relocation of the Sci- ence faculty, and the conflicting resource demands associated with the Chair of Computa- tional Linguistics. We urge that a solution be found that is compatible with the interdisciplinary nature of the Institute mission, and the central role that computational linguistics plays in linking sciences and humanities in that work, and have tried to suggest some possible avenues. We are confident that the University will act to preserve this unique institution, and make the following recommendations in the hope that they will assist this process:

The Chair in Computational Linguistics should be continued, either under support from the Science faculty, or by greater engagement by the Humanities, or by rationalizing the allocation of teaching credit, or by some combination of the above.

The new research programmes to some extent represent a step in the direction of a return to a traditional division between Science and Humanities. The Institute should be on guard in the coming period against further fragmentation and isolation, paying particular attention to connections between the Logic and Language and Language and Computation programmes (perhaps by developing shared interests in probabilistic semantics), and between Logic and Computation and the cognitive science interests of the Institute as a whole.

To help in this process, strong consideration should be give to the possibility of locating the Institute at a site intermediate between the new science campus and the Humanities in the Center, such as the Plantage Muidergracht site.

While recruitment of new faculty has been highly successful and creative, there is a striking under-representation of women on the faculty in proportion to the field as a whole, which should be a matter of concern in future recruitment.

Appendix 1 REVIEW COMMITTEE

The Review Committee consisted of

Prof. Dr. Gennaro Chierchia General Linguistics, Department of Linguistics, Harvard University, USA

Prof. Dr. Jan Willem Klop Theoretical Computer Science, Department of Computer Science, Vrije Universiteit, Amsterdam, The Netherlands; CWI (Centrum voor Wiskunde en Informatica), Amsterdam, The Netherlands

Prof. Dr. Wolfram Pohlers Mathematical Logic, Proof Theory, Generalized Recursion Theory University of Münster, Germany

Prof. Dr. Mark Steedman (chair) Cognitive Science, School of Informatics, University of Edinburgh, Scotland, UK

As secretary acted Drs. Jan Heijn, self-employed physicist and former executive secretary of several national and international evaluation committees.

CURRICULA VITAE

Gennaro Chierchia Haas Foundations Professor of Linguistics, Harvard, USA

Academic education and degrees: 1978 Laurea in Filosofia, University of Rome 1984 Ph.D. in Linguistics, University of Massachusetts at Amherst

Recent Academic positions: 1992–1999 Associate then Full Professor, Department of Philosophy, Univ. of Milan 1999–2005 Professor, Department of Psychology, University of Milan-Bicocca 2005–present Professor, Department of Linguistics, Harvard University

Research interests: Semantics, syntax-semantics interface, pragmatics, philosophy of language, psycholinguistics

Membership of editorial boards: Linguistic Inquiry, Linguistics and Philosophy, Natural Language Semantics, Syntax

Jan Willem Klop Professor of Computer Science at the Vrije Universiteit Amsterdam

Academic education and degrees: 1975 M.Sc. Mathematical Logic, University of Utrecht 1980 Ph.D. Mathematical Logic, University of Utrecht

Academic positions: 1980–1984 Researcher at CWI, Amsterdam 1984–2007 Senior Researcher at CWI, Amsterdam 1986–2007 Professor of Applied Logic at the Vrije Universiteit Amsterdam 1998–2007 Research Professor (0.2 fte), Radboud University Nijmegen

Research interests: Process algebra, term rewriting systems, foundations of functional programming, abstract data types, lambda calculus, formal methods, applied logic, verification of systems

Professional activities: 1998–2004 Chairman of Dutch Association for Theoretical Computer Science (NVTI) 1999–2004 Head of Section Theoretical Computer Science, Vrije Universiteit 1999–2003 Member of editorial board of ACM Transactions on Computational Logic (TOCL) 2002–2004 Head of Department of Computer Science, Vrije Universiteit Amsterdam 2002–2007 Member of Board of Directors of Dutch Research School IPA

Awards: 2002 Honorary Doctorate Degree, University of East Anglia 2003 Member of Royal Dutch Academy of Sciences (KNAW) 2003 Fellow of CWI Amsterdam

Wolfram Pohlers Professor of Mathematics at the University of Münster Director of the Institute for Mathematical Logic and Foundational Research

Academic education and degrees: 1971 Diplom in Mathematics at the Ludwigs Maximilian University in Munich 1973 Dr. rer. nat. at the Ludwigs Maximilian University in Munich 1978 Dr. rer. nat. habil. at the Ludwigs Maximilian University in Munich

Academic positions: 1971–1972 Scientific collaborator (Ludwigs Maximilian University Munich) 1972–1978 wissenschaftlicher Assistent (Ludwigs Maximilian University Munich) 1978–1980 Pivatdozent (Ludwigs Maximilian University Munich) 1980–1985 Assistant Professor, tenured (Ludwigs Maximilian University Munich) 1985– Full Professor at the Westfälische Wilhelms-University in Münster

Research interests: Mathematical logic, proof theory with emphasis on ordinal analysis, generalized recursion theory

Professional activities: Member of the editorial board of ‘Mathematical Logic Quarterly’ (Wiley_VHC) Member of the editorial board of ‘Journal of Applied Logic’ (Elsevier) Member or the editorial board of ‘Series in Mathematical Logic’ (Ontos Verlag) Managing editor of ‘Archive for Mathematical Logic’ (Springer) Review editor for the Bulletin of Symbolic Logic until January 2006 Chairman of the committee on Logic in Europe (until January 2007) Treasurer of the German Association for Mathematical Logic and the Foundations of Sciences Member of the Academic Senate of the Westfälische Wilhelms-University Spokesperson of the professors in the Academic Senate

Mark Steedman Professor of Cognitive Science in the School of Informatics, University of Edinburgh.

Academic education and degrees: 1968 B.Sc. Experimental Psychology, University of Sussex 1973 Ph.D. Artificial Intelligence, University of Edinburgh

Recent Academic positions: 1988–1998 Associate then Full Professor, Computer and Information Science, University of Pennsylvania 1998– Professor, School of Informatics, University of Edinburgh

Research interests: Computational linguistics, wide-coverage parsing, syntax and semantics of natural language, prosody and intonation, artificial intelligence, musical cognition

Current professional activities: President-elect for 2008 of the Association for Computational Linguistics (ACL) and member of the ACL executive committee Membership of editorial boards: Computational Linguistics; Journal of Semantics; Research in Language and Computation.

Awards: Fellow of the British Academy; Fellow of the Royal Society of Edinburgh; Member of the European Academy; Fellow of the American Association for Artificial Intelligence

Appendix 2 SITE VISIT PROGRAMME

Sunday October 29

Arrival of the committee members Committee Dinner at the Hotel

Monday October 30

9.00 – 9.30 Closed session: working procedure, division of tasks 9.30 – 10.30 Introduction Veltman Participants: all (former) programme leaders 10.30 – 11.00 Coffee break 11.00 – 12.00 Meeting with the ILLC (former) management Participants: Veltman, Stokhof, Loewe, Van Loon 12.00 – 12.45 Closed committee session 13.00 – 14.30 Lunch 14.45 – 17.15 LANGUAGE AND COMPUTATION Participants: Scha, Sima’an, Honing 17.15 – 18.30 Closed session and meeting with Rector Paul van der Heijden

Tuesday October 31

9.30 – 12.00 LOGIC AND LANGUAGE Participants: Stokhof, Groenendijk, Van Lambalgen, Dekker 12.00 – 12.30 Closed committee session 13.00 – 14.30 Lunch with PhD and MoL students 14.45 – 17.15 LOGIC AND COMPUTATION Participants: Van Benthem, Väänänen, Van Emde Boas, Torenvliet 17.15 – 17.45 Closed committee session 19.00 – 22.00 Boat trip through the canals and working dinner aboard, together with Karel Gaemers, dean Faculty of Science, Willem Weststeijn, vice-dean Faculty of Humanities, and Johan van Benthem, University Professor and Chair of Computer Science department

Wednesday November 1

Location Bungehuis, room 1.01 10.00 – 13.30 Closed committee session 13.30 – 13.30 Lunch 14.00 – 14.30 Plenary session: ORAL PRESENTATION of the preliminary results 14.30 Get-together Departure committee members

Appendix 3 BIBLIOMETRIC ANALYSIS

In consultation with CWTS* the UvA decided not to order a bibliometric analysis of the scientific output of ILLC, for the following reasons.

ILLC is composed of scientists from the humanities and the computer sciences. As for the humanities, CWTS does not apply its standard bibliometric instruments on research(ers) in this field of research, as the scientific communication between scientists in the humanities is not adequately covered by the databases CWTS uses for its bibliometric analyses. As these researchers mainly publish in books and book chapters, and do publish in (often non- English language) journals not covered in the Arts & Humanities Citation Index (A&HCI), such a performance analysis would not do right to the people involved, and their research efforts.

As for the computer sciences in ILLC, researchers in this field do publish often in conference papers, covered by proceedings books. As these proceedings papers are not adequately covered in the Web of Science (WoS), a performance study of their research would only of- fer you a partial insight into their research performance. CWTS has been involved in a fea- sibility study, initiated by NWO, on the applicability of bibliometrics in the field of computer sciences, and although the findings are not yet public, one conclusion is clear: bibliometrics could be applied in this field, but only if the universe on which such an analysis would be based, is set up much broader than the current WoS coverage for the computer sciences. As such a database exists currently only in a proto-type version at CWTS, a bibliometric analysis for ILLC would be premature.

* The Centre for Science and Technology Studies (CWTS), part of Leiden University, spe- cializes in advanced quantitative analysis of science and technology performance and the cognitive and organizational structure of science and technology.

LIST OF ABBREVIATIONS

AE Academia Europaea ACL Association for Computational Linguistics ACT Algorithms and Complexity Theory (former research programme) CIL Constructive and Intensional Logic (former research programme) CL Computational Linguistics CLEF Cross-Language Evaluation Forum CSIP Cognitive Systems and Information Processing (former programme) CWI Centrum voor Wiskunde en Informatica Center for Mathematics and Computer Science CWTS Centrum voor Wetenschaps- en Technologiestudies Centre for Science and Technology Studies EU European Union ILLC Institute for Logic, Language and Computation KNAW Koninklijke Nederlandse Akademie van Wetenschappen Royal Netherlands Academy of Arts and Sciences LIT Language and Inference Technology MoL Master of Logic NWO Nederlandse Organisatie voor Wetenschappelijk Onderzoek Netherlands Organisation for Scientific Research TREC Text Retrieval Conference QANU Quality Assurance Netherlands Universities RST Research Training Site SEP Standard Evaluation Protocol TI Theory of Interpretation UvA Universiteit van Amsterdam VSNU Vereniging van Samenwerkende Nederlandse Universiteiten Association of Universities in the Netherlands

24 July 2007

Assessment of Research Quality

Institute for Logic, Language and Computation

Address: Plantage Muidergracht 24 1018 TV Amsterdam The Netherlands

Phone: +31 20 525 6051 Fax: +31 20 525 5206 Email: [email protected] Website: http://www.illc.uva.nl

ILLC evaluatierapport.indd 1 01-10-2007 15:12:33