Final Report ALCOM-FT Algorithms and Complexity Future Technologies Project No. IST-1999-14186 January 2004 Summary This is the final report for the ALCOM-FT project, supported by the European Commission as project number IST-1999-14186 under the Future and Emerging Technologies part of the IST programme of the Fifth Framework. The report covers the entire project period from June 1, 2000 to November 30, 2003. ALCOM-FT has brought together eleven of the leading groups in algorithms research in Eu- rope in a project aiming at discovering new algorithmic concepts, identifying key algorithmic problems in applications, and contributing to the transfer of advanced algorithmic techniques into commercial systems. The eleven participants of the project are listed in Table 1. No. Full Name Short Name 1 BRICS, Department of Computer Science, University of Aarhus Aarhus (coordinating site) 2 Department of Software, Polytechnic University of Catalunya, Barcelona Barcelona 3 Department of Computer Science, University of Cologne Cologne 4 INRIA, Rocquencourt INRIA 5 Max-Planck-Institut fur¨ Informatik, Saarbruc¨ ken MPI 6 Department of Mathematics and Computer Science, University of Paderborn Paderborn 7 Computer Technology Institute, Patras, Greece. CTI 8 Department of Computer and System Science, University of Rome Rome \La Sapienza" 9 Department of Computer Science, University of Utrecht Utrecht 10 Department of Computer Science, University of Warwick Warwick 11 Department of Computer Science, University of Cyprus Cyprus Table 1: The participants of the ALCOM-FT project. The focus of the project has been a combination of application oriented research in three important areas { massive data sets, massive and complex communication, and complex problems in production and planning { with innovative methodological work on experimental algorithmics and generic algorithmic methods. The project has been divided into work packages, one for each of the five areas above, and one additional dealing with management and dissemination efforts. The scientific advances made by the project consortium in each of these areas have been very notable in terms of quality as well as quantity, and the goals set forth in the project description have been reached. Part I of this document gives an overview of these achievements. All deliverables have been delivered1, and the large majority of these were delivered on schedule. Table 2 lists all deliverables and their delivery times. Further descriptions of their contents are given in Part II of this document. The largest deliverable is D1, which constitutes a total of 538 scientific reports produced during the project period. This high productivity has not been achieved at the cost of quality, as can be seen by the fact that at least 410 of the 538 reports have already been published in the scientific community via key conferences and journals, including the most prestigious in the world. The set of deliverables also contains a number of software projects contributing to the trans- fer of advanced algorithmic methods into practice. These are: A library for external memory 1Except the Technology Implementation Plan, which is to be handed in by February 1, 2004 ii Summary computation with a functionality similar to the widely used STL C++ library (D6, D15, D24); A demonstration package for methods in data mining (D16); A LEDA Extension Package for distributed algorithmic engineering, to facilitating simulation of large, hierarchical, or mobile net- works and development of network optimization methods for resource allocation (D7, D17, D25); A BSP-style library which allows efficient implementation of BSP-style programs in dynamic dis- tributed environments (D8, D18, D26); Software for production and transportation planning, along with a database containing real-life problem instances from the same area (D19, D27, D28); A completed software system (SCIL) for structured combinatorial optimization problems, including high-level constraints and their separation routines (D20, D23, D29); A set of tools for executing, documenting, visualizing, and reproducing algorithmic experiments (D5, D13, D21 D30). Further dissemination efforts of the ALCOM-FT project include arrangement of eight sum- merschools, arrangement of several conferences and workshops (among these the 2000, 2001, and 2002 versions of ALGO, the premier algorithmic event in Europa), creation of a project web site (D2) and two web sites disseminating algorithmic knowledge (D11, D14), and industrial talks on algorithmic issues, The project has been lead by a Consortium Board consisting of the scientific leader from each site. The following two adjustments was decided upon by the Board during the project: The algorithmics group at the University of Cyprus, headed by Prof. Marios Mavronicolas was added as an ALCOM-FT site for further strengthening of Work Packages 2 and 4, and the project was extended by six months to ensure proper finalization of some delayed deliverables (D14, D15, D19, and D21). The rest of this document is organized into three parts. The first describes the progress achieved within the six work packages of the project. The second lists details of each deliverable of the project. The third lists all scientific reports published as part of the project. iii Summary No. Deliverable Month Month Planned Delivered D1 Research reports Cont. Cont. D2 Project Presentation 3 3 D3 Dissemination and Use Plan 6 6 D4 Guidelines for algorithmic experiments (internal release) 6 12 D5 Testbed for experimental algorithmics (specification) 6 6 D6 External memory experimental platform (design) 12 12 D7 Distributed algorithmic engineering software package (prototype) 12 12 D8 BSP-style library for dynamic distributed environments (proto- 12 12 type) D9 Production and transportation planning modeling report 12 12 D10 Guidelines for algorithmic experiments (external release) 12 36 D11 Algorithm Forum web site 12 12 D12 First progress report 12 12 D13 Testbed for experimental algorithmics (internal release) 18 18 D14 PR web-pages for algorithmics 18/36 36 D15 External memory experimental platform (prototype) 24/36 36 D16 Data mining demonstration package 24 24 D17 Distributed algorithmic engineering software package (beta ver- 24 24 sion) D18 BSP-style library for dynamic distributed environments (beta ver- 24 24 sion) D19 Production and transportation planning software prototype 24/36 36 D20 Description language for high level constraints in optimization 24 24 D21 Testbed for experimental algorithmics (external release) 24 26 D22 Second progress report 24 24 D23 Library of separation routines for ABACUS 30/36 36 D24 External memory experimental platform (final release) 36/42 42 D25 Distributed algorithmic engineering software package (final re- 36/42 42 lease) D26 BSP-style library for dynamic distributed environments (final re- 36/42 42 lease) D27 Production and transportation planning software user evaluation 36/42 42 report D28 Production and transportation planning problem instance 36/42 42 database D29 Complete software system for structured combinatorial optimiza- 36/42 42 tion problems (SCIL) D30 Testbed for experimental algorithmics (final release) 36/42 42 D31 Technology implementation plan 36/42 { D32 Third progress report 36/42 42 D33 Final report 36/42 42 Table 2: The deliverables of the project. The notation 18/36 denotes the planned delivery month before and after the extension. iv Contents I Work Packages 1 WP 1 Massive Data Sets . 1 WP 2 Networks and Communication . 5 WP 3 Production and Transportation Planning . 15 WP 4 Generic Methods . 17 WP 5 Experimental Algorithmics . 24 WP 6 Project Management, Dissemination, Evaluation . 26 II Deliverable Overview 29 III Scientific Reports 35 v Part I Work Packages WP 1: Massive Data Sets Participants: Aarhus, Barcelona, INRIA, MPI, Paderborn, CTI, Rome, Warwick Coordinating site: Barcelona Work Package Leader: Jos´e L. Balc´azar Number of technical reports: 80, of which 61 also appear in other work packages Presentation Traditional algorithm design is based on the von Neumann model, which assumes uniform memory access costs. However, our current state-of-the-art machines increasingly deviate from this model. From the registers, through very fast caches and a bit less fast internal memories, to increasingly complex secondary or even tertiary storage structures such as systems of parallel hard drives or robotized racks of CD-ROM units, algorithms that process large datasets need an awareness of where the accessed data is stored, and must take into account that the cost of memory accesses depends on the characteristics of the memory support. Typically, each level of this complex memory hierarchy, compared to its preceding level, has slower access time, larger capacity, and cheaper per-bit costs. For instance, an operation on registers in a modern but humble machine, operating at around a gigahertz as clock speed, takes a few cycles, whereas a memory access takes microseconds, or about 1000 times longer, and a disk access takes milliseconds, or one million times longer. These rough estimates give a hint of the problem faced, but actual computations with actual times of actual machines give even more daunting differences. Even the constantly growing capacity of disk drives cannot satisfy the ever growing demand for flexible and efficient storage systems. Increasingly, commercial database systems offer support for including, as fields of the records in their tables, so-called BLOBS (Binary Large OBjectS) corresponding to e.g. images