International Review of Research Using HPC in the UK December 2005

This document represents the conclusions of an international Review Panel of experts in computational science and engineering. The views expressed are entirely those of the members of that Panel.

Engineering and Physical Sciences Research Council Polaris House, Northstar Avenue Swindon, SN2 1ET Wiltshire, UK

http://www.epsrc.ac.uk/

ISBN 1-904425-54-2

CONTENTS

1. Executive Summary ...... 1

2. Introduction ...... 2 2.1. The Importance of Research Using HPC for the UK ...... 2 2.2. Context, Scope, and Methodology of the Review ...... 2

3. Findings on the UK HPC Research Environment ...... 4 3.1. Delivery of Science through the Use of HPC ...... 4 General Observations ...... 4 Specific Accomplishments ...... 5 3.2. Standing and Research Potential in Comparison to the International Field ...... 6 3.3. Developing and Retaining Talent ...... 6 3.4. Impact on the Knowledge Economy ...... 8 Technology Transfer Activities ...... 8 Research Collaborations between Academic and Industrial Partners ...... 8 3.5. Strengths ...... 9 3.6. Opportunities ...... 10 Research Consortia ...... 10 Collaboration with Computer Science and Applied Mathematics ...... 11

4. HPC Resources ...... 12 4.1. National Leading-Edge Computational Systems ...... 12 4.2. University-Based Computational Systems ...... 12 4.3. Data Resources ...... 13 4.4. Visualization Resources ...... 15 4.5. User Support ...... 15

5. Conclusions and Recommendations ...... 16

6. Acknowledgements ...... 18

7. Appendices ...... 19 7.1. Steering Group ...... 19 7.2. Panel Membership ...... 19 7.3. Terms of Reference ...... 19 7.4. Research Groups Reviewed ...... 19 7.5. Review Sheet ...... 20

1. Executive Summary Because of the complexities of large-scale hardware systems and the increasing sophistication of modelling software, including multi-physics and multiscale simulation, CSE Computational Science, the scientific investigation increasingly became a team science. The most of physical processes through modelling and successful practitioners of CSE today are simulation on computers, has become generally multidisciplinary teams that include accepted as the third pillar of science, mathematicians and computer scientists. These complementing and extending theory and teams have set up a software infrastructure, experimentation. This view was probably first including a support infrastructure, for large codes expressed in the mid-1980s. It grew out of an that are well maintained and extensible beyond impressive list of accomplishments in such diverse the set of original developers. areas as astrophysics, aeronautics, chemistry, climate modelling, combustion, cosmology, The importance of CSE for the future of research earthquake prediction, imaging, materials, accomplishments and economic growth has been neuroscience, oil exploration, and weather well established. “Computational science is now forecasting. Today, in the middle of the first indispensable to the solution of complex problems decade of the 21st century, the pace of innovation in every sector, from traditional science and in information technology is accelerating, and engineering domains to such key areas as national consequently the opportunities for computational security, public health, and economic innovation,” science and engineering abound. Computational is the principal finding of the recent report of the Science and Engineering (CSE) today serves to President’s Information Technology Advisory 1 advance all of science and engineering, and many Committee (PITAC) in the U.S. Hence the areas of research in the future will be only EPSRC’s decision to organize a review of the state accessible to those with access to advanced of research using HPC in the UK was timely. The computational technology and platforms. International Review Panel found that research using HPC in many areas is of the highest Progress in research using high performance standing and competitive at the international level. computing platforms has been tightly linked to However, in a dynamically changing and rapidly progress in computer hardware on one side and evolving field such as CSE, one cannot afford to progress in software and algorithms on the other, stand still, and therefore in this report the Review with both sides generally acknowledged to Panel also point out areas for further contribute equally to the advances made by improvements and refinements of the HPC researchers using these technologies. With the research strategy. It is our hope that building on arrival of highly parallel compute platforms in the this report, the UK computational science and mid-1990s, several subtle changes occurred that engineering community will be able to continue to changed the face of CSE in the last decade. maintain its position at the forefront of research using HPC.

1 President’s Information Technology Advisory Committee, Computational Science: Ensuring America’s Competitiveness (Arlington, Virginia: National Coordination Office for Information Technology Research and Development, 2005), p. 2.

1 2. Introduction advantage a company (or a nation) will enjoy will be its process of innovation—combining market and technology know-how with the creative talents of knowledge workers to solve a constant stream of competitive problems—and its ability to 2.1 THE IMPORTANCE OF derive value from information. RESEARCH USING HPC FOR THE UK The UK Research Councils recognize the importance of HPC and have made significant For the UK to compete in the global economy, it investments in HPC facilities and the research must take advantage of its excellent track record they support. To assess the returns on those in scientific discovery and technological investments, an international panel of ten innovation. Both science and technology are computational scientists spent a week in becoming increasingly dependent on high September 2005 visiting universities, talking with performance computing (HPC)—indeed, researchers and students, discussing their computation has emerged as a distinct branch of observations, and formulating findings and research, with an importance matching those of recommendations. This report is the culmination theory and experiment. Computation has now of their efforts. become essential for the advancement of all research across science and engineering. HPC “aims to satisfy the most demanding scientific 2.2 CONTEXT, SCOPE, AND goals, to push the boundaries of researchers’ METHODOLOGY OF THE ambitions and to stretch the development of REVIEW hardware and software technologies in dimensions that often prove beneficial outside the [HPC] In 1999, the Engineering and Physical Sciences research arena.”2 Research Council (EPSRC)—in cooperation with relevant learned societies—began a cycle of HPC enables science and engineering to tackle a reviews by international panels of the state of UK class of problems and opportunities that cannot be science in fields corresponding to its major approached in any other way. These opportunities programmes. In these reviews, a panel of include genome sequencing, biomedicine, and internationally leading researchers benchmark the drug design; molecular engineering of new strength of the UK research activity against world materials and devices; engineering efficient and competitors and highlight any gaps or missed environmentally sound technologies and energy opportunities. The panel visit a number of UK sources; forecasting global climate change and research groups and have access to a wide pool of finding strategies to deal with it; and experts and supporting data to help them reach understanding the fundamental interactions of their conclusions. Each research discipline is elementary particles and the origin and fate of the scheduled to be reviewed every five years. universe. The 2005 International Review of Research Research using HPC is widely expected to be a Using High Performance Computing in the major contributor to the UK’s “knowledge UK was coordinated by the Deutsche economy.” The concept of the knowledge Forschungsgemeinschaft (German Research economy is based on the assumption that Foundation or DFG) and was the first to review knowledge and innovation will be crucial to this area of research. The review was overseen future economic success in the industrialised by a Steering Group (see Appendix 7.1) who world. Information and knowledge are replacing appointed the Chair of the Panel, selected the capital and energy as the primary wealth-creating other Panel members (Appendix 7.2), and set the assets, just as the latter two replaced land and terms of reference as follows: labour 200 years ago. The only comparative

2 High End Computing Terascale Resources (HECToR) Scientific Case (unrestricted version, dated 26 April 2004), p. 4.

2 The International Review Panel is requested to visit universities and computing centres. to: Discussions between sub-groups and researchers were guided by a standardised review sheet • report on the calibre, standing and (Appendix 7.5), on which panellists recorded their research potential in research using High observations and comments. The leader of each Performance Computing in the UK sub-group compiled the group’s comments onto a single review sheet, and all of these were • discuss the potential impact of university- distributed to the entire Panel. The Panel based research using High Performance reconvened in London on the evening of Computing on the UK’s knowledge 7 September and all day on 9 September to economy discuss their findings. During the following weeks, • provide comparisons with international Panel members drafted sections of this report, research using High Performance which were compiled and edited by the Scientific Computing Secretary.3 Drafts were circulated for review and comments, and the report was then finalised • make recommendations on future actions under the supervision of the Panel Chair and and/or priorities accepted by the Steering Group.

The research groups to be reviewed (Appendix This review was particularly challenging for all 7.4) were selected by the Steering Group from a participants, not only because it was the first on list proposed by EPSRC in consultation with the this topic, but even more because of the diversity Biotechnology and Biological Sciences Research of the research under consideration. Council (BBSRC), the Natural Environment Computational Science and Engineering (CSE) is Research Council (NERC), and the Particle Physics interdisciplinary by nature and is a relative and Astronomy Research Council (PPARC). newcomer on the academic scene. There are few Criteria included usage of the CSAR and HPCx degree programs in CSE and few university computing facilities, Research Assessment departments dedicated to this field; as a result, Exercise (RAE) scores, and scientific diversity. there is a scarcity of objective data on the state of the discipline in the UK, in comparison with data Panel members were sent background that can readily be obtained for more traditional documentation about UK science policy and disciplines. The Review Panel were keenly aware funding structures, basic information about of the anecdotal nature of much of the research using HPC in the UK, and information information presented to them, and were careful about the research groups to be reviewed in their deliberations not to draw overly broad (provided by the groups themselves in answers to conclusions from too few examples. Nevertheless, a questionnaire). The Panel convened in London several common themes did emerge from the on 4 September 2005 to review roles and reviews of the individual research groups, and responsibilities before splitting up into sub-groups those themes form the basis for this report.

3 In a few cases, additional information was provided to the Review Panel by research groups to resolve questions that remained open after the site visits.

3 3. Findings on the The collective strength of effective consortia formed in a number of research areas has greatly UK HPC Research aided the delivery of new scientific results through computational models with high fidelity Environment physics developed with significantly less abstraction than in the past. Effectively harvesting the great potential for scientific discovery made possible by HPC has also involved making progress on the formidable challenge of obtaining 3.1 DELIVERY OF SCIENCE meaningful information from the tremendous THROUGH THE USE OF HPC amounts of data generated at dramatically higher rates than ever before. As emphasized with specific recommendations in Section 3.6 below, General Observations the consortium model is an organizational asset of During the course of this review, the panel HPC in the UK which should not only be observed a wide range in the quality of science maintained but further expanded. delivered through the use of HPC in the UK but overall was very impressed. It is now widely Specific examples of the delivery of science recognized that the rapid advances in HPC have through the use of HPC in the UK are highlighted enabled realistic computer-based modelling and in the following section. These examples of simulation to accelerate progress in solving the impressive advances enabled by HPC come formidable mathematical models that govern the largely from those groups (including UKQCD, behaviour of complex natural and engineered Mineral Physics at UCL, and UK Turbulence systems, including the challenge to compute Consortium) which are very proficient in both phenomena over wide ranges of temporal and using and enhancing computational technologies spatial scales. For example, the challenge in for the accelerated achievement of new scientific materials research is to deduce macroscopic results. Many other research groups use behaviour directly from the quantum-mechanical computational technology effectively but only in behaviour of the constituent atoms or molecules. the restricted sense of delivering the scientific For atmospheric flows, global patterns need to be results using computational tools and codes resolved, as well as the intricate details of the developed elsewhere. heat, mass, and momentum exchange at the surface of the sea. In plasma physics, new As a whole, it was observed that the level of insights are needed to address how large-scale integration of computational science with magnetic confinement of high-temperature computer science in the UK has not yet reached plasmas can be significantly improved via control the highest international standards. Perhaps due of micro-scale turbulent losses. In particle in some measure to present funding practices, physics, high-precision computer simulations of many research groups in the UK appeared to be QCD are expected to provide detailed tests of the deficient in adapting to modern programming Standard Model and better understanding of practices, anticipating the need for data quark confinement and the phase structure of management and visualization, and developing strongly interacting matter. For astrophysics and interoperable software and algorithm cosmology, the task is to bring together the environments. If these deficiencies were detailed interaction of gas with electromagnetic remedied, we should witness the accelerated radiation and the gravitational potential of dark emergence of dramatically improved capabilities matter. In engineering applications, industrial for making critical discoveries in many scientific competitiveness depends critically on the domains which in the past have been considered continuous development of advanced intractable due to their extreme complexity and/or computational techniques for design optimization lack of available data analysis capability. of complex systems. UK scientists are effectively utilizing the advances in HPC in many of these As we look toward the future, the examples of areas with results that compare favourably with significant scientific progress observed in the international standards.

4 current review indicate that the needs of exciting UK Turbulence Consortium (UKTC): scientific applications will further stimulate the Significant progress in the understanding of development of new computer science capabilities laminar-turbulent transition (Neil Sandham’s and innovative mathematical methods and group at Southampton) and new insights into key algorithms. These advances will help harness the topological issues in the physics of premixed underlying exponential growth in technology and combustion (Stewart Cant’s group at Cambridge) enable new possibilities for investigation. have been attained by the UKTC’s direct numerical simulations using HPC resources. In Specific Accomplishments particular, direct numerical simulations of In the course of this review, several examples of premixed turbulent flames utilizing the full the fruitful delivery of science through the use of available power of the HPCx system revealed that HPC in the UK stood out. These included: the topology is largely cylindrical in shape. This is an important finding which can be cited in Institute for Computational Cosmology justifying the application of simpler geometric (ICC), University of Durham: Carlos Frenk is models. the Director of ICC and co-PI of the Virgo Consortium, together with Simon White, Director Multiphoton, Electron Collision, and Bose of the Max Planck Institute for Astrophysics in Einstein Condensates (MECBEC) HPC Garching, Germany. The Virgo group has carried Consortium: In the area of atomic, molecular, out the largest, most detailed cosmological and optical physics, Ken Taylor’s group has gained simulations: Hubble Volume and Millennium. important insights into the limits of photon With 10 billion particles, the Millennium N-body absorption in time-dependent Schrödinger simulation of the growth of structure in the equation calculations of multi-photon ionization. standard cold dark matter model is about one They have also been the first to calculate the full order of magnitude larger than the previous two-electron response of helium to intense free largest. Science magazine called the combination electron laser light at vacuum ultraviolet (VUV) of 2dF galaxy survey data interpreted via wavelengths. extensive use of cosmological simulations with Wilkinson Microwave Anisotropy Probe (WMAP) HPC Materials Chemistry Consortium: In data the Breakthrough of the Year for 2003, the area of HPC simulations of materials and “Illuminating the Dark Universe.” nanoscience, Richard Catlow’s group has achieved major progress in studies of heterogeneous UKQCD: Effective utilization of HPC by the catalytic processes. In the course of their UKQCD consortium has achieved major advances development of the most advanced molecular in the understanding of observations from dynamics code (DL_POLY3), simulations of worldwide particle physics experiments. In several million atoms have been carried out—a particular, a very important and highly visible world record for simulations of materials with result obtained in partnership with the HPQCD, charged ions. MILC, and Fermilab Lattice Collaborations is the comparison of physical results obtained in the Mineral Physics at University College quenched approximation with results from full London: David Price’s group at UCL have dynamical simulations including the three lightest carried out HPC ab initio simulations with novel quarks. The discrepancies between results from results (published in Nature) which provide tight experiments and the quenched approximation are constraints on the composition and thermal completely eliminated in the much more realistic structure of the Earth’s core. dynamical simulations. Although there is still an unknown systematic error in these results that UK Astrophysical Fluids Facility (UKAFF): ought to be clarified, this result has been widely UKAFF has performed a highly visible simulation heralded as a breakthrough in the field of lattice of star formation that appeared on the cover page quantum chromodynamics. of Nature.

5 These examples of impressive advances enabled the need for data management and by HPC come largely from those groups which are visualization, and developing interoperable very proficient in both using and enhancing software and algorithm environments. computational technologies for the accelerated delivery of new scientific results. • The issue of developing algorithms, tools, and methods that are required in common by diverse projects in computational 3.2 STANDING AND RESEARCH sciences has not been adequately addressed POTENTIAL IN COMPARISON in the UK. There appears to be no strong TO THE INTERNATIONAL FIELD national computational science community to lobby for such a programme. The Review Panel judged that in many of the consortia and research groups visited, the • The training of computational scientists in scientific results compared well with the highest the UK is typically too narrow (problem- international standards. These groups included the and code-specific) and too short for HPC Materials Chemistry Consortium, the interdisciplinary training and internships, as Institute for Computational Cosmology, the discussed in Section 3.3 below. MECBEC HPC Consortium, the Mineral Physics • The traditionally strong assets in some fields Programme at UCL, the UK Astrophysical Fluids of UK science are theory and observation Facility, the UK Car-Parinello Consortium, the rather than computation. UKQCD Collaboration, and the UK Turbulence Consortium. In some cases—ICC, Mineral Physics, The EPSRC’s recognition of the importance of UKQCD, UKTC—the UK groups are playing a lead HPC and the research community’s already strong role in setting the international standards. demand for HECToR indicate that the UK Continued leadership and realization of the latent research environment is now ripe for fostering potential will depend on continuity in getting scientific progress through HPC. During the access to adequate computational resources, as course of our site visits, the Panel recognized both discussed in Section 4 below. the significant achievements and the high potential of computational science and Panel members are also aware of other engineering in the UK. Of course, the progress of internationally leading institutions in the UK that computational science depends strongly on the were not part of this review but should not be available computer facilities. A new HPC system overlooked, including the European Bioinformatics would undoubtedly attract good students and Institute, the Hadley Centre for Climate Prediction provide a boost for both existing and new and Research, and the European Centre for computational research projects in the UK, thus Medium-Range Weather Forecasts. contributing substantially to science and engineering worldwide. In view of the recognition that “computation has now become essential for the advancement of all research across science and engineering,”4 it is 3.3 DEVELOPING AND clear that HPC in the UK has not achieved its full RETAINING TALENT potential in every area of research. The Panel found a variety of possible reasons for this With increasing emphasis on computer and shortcoming: information sciences in international scientific research, the Panel assessed whether the • Many computational scientists are not education and training of computational scientists working closely with computer scientists and engineers in the UK reflected the need for and HPC manufacturers to maximize the scientists who have high interdisciplinary training. effectiveness of computer systems for The metrics employed included training and specific scientific applications. As noted competency in numerical algorithms, expertise in above, many research groups in the UK software engineering, ability to adapt to modern appeared to be deficient in adapting to programming practices, and efficient use of high modern programming practices, anticipating performance computational resources.

4 High End Computing Terascale Resources (HECToR) Scientific Case (unrestricted version, dated 26 April 2004), p. 4.

6 The Panel probed the plans evolved by the various their specific disciplinary codes. Algorithmic consortia for training students and retaining them sophistication was found lacking in several cases. in the computational science area. The UK The majority of the application codes were funding agencies have instituted High End extensions of legacy codes, i.e., codes written in Computing Studentships that provide four-year Fortran that had grown over decades, and funding, including one year equivalent of training therefore a modern software engineering in the computational sciences over the four years perspective was lacking, even though the trainees for M.Sc.-Ph.D. students enrolled in this program. using the codes were proficient in their respective In Warwick and Edinburgh, students work on programs. Also, in the area of visualization, computational science modules and thus obtain modern client-server programming paradigms additional training. Even though small in scale were not deployed. Overall, the culture of best- (about ten students per year), this was found to be practices software engineering has not percolated a good mechanism for interdisciplinary training by effectively into the UK scientific community. Still, several consortia. The Integrated Biology group the panel observed that several of the consortia had evolved computational life science modules effectively used the high performance where they trained students for six weeks. This computational resources. Several codes were MPI- program was widely subscribed to and was capable and were scaling well on multi-processor regarded as a success by life science trainees. machines. In the area of data and data-driven Several of the groups, however, had no formal computing, a number of groups recognized its training mechanism for their students and relied growing importance; some are already developing on “during the studentship training.” The Data Grid tools, but others acknowledged their Materials Chemistry and UKCP consortia unpreparedness for data organization, handling, expressed the desire to have more extensive and query and analysis. training for their students, but were limited by the three-year degree constraints. There was a The Panel tried to understand the reasons behind significant paucity of students in some consortia, the above findings. There appear to be two major where the absence of students despite the criteria: first, the culture in computational science presence of resources was a barrier for research training in the UK has historically been only productivity. To some extent this reflects the problem-specific. For instance, students are culture of education and training in UK trained in specialized codes such as CASTEP institutions. The Panel also were told that the during their studentship. Second, the UK difficulty of obtaining UK funding for non- educational and funding system mandates that domestic students imposes constraints on each student complete his or her Ph.D. program attracting the best talent. in three years. This leaves little or no time for formal interdisciplinary computational training. Students trained in the various programs either Further, it inhibits the opportunities for internship continued to stay in the program with fellowships training in industry during graduate studentship. or moved on, in several cases to other computational jobs (mostly in the same discipline). The Panel makes the following specific In some areas, such as life sciences and materials recommendations: simulations, the students found jobs in the 1. The funding agencies should explore respective industry. There was some attrition of training fellowships that mandate formal students to non-science job opportunities, and this curriculum in computational science. This is was more common in physics than engineering or not just to offer training to a new generation life science disciplines. of scientists and engineers, but also to give a stronger identity to the computational The Panel observed that in the majority of the science community in the UK. sites visited, the personnel were trained in the specific code they were using and in some cases 2. Institutions like Daresbury Laboratory and contributed significantly to the development of the efforts like Collaborative Computational code. However, the Panel observed significant Projects (CCPs) should be given the shortcomings based on the metrics stated above. resources required to educate the science The Panel observed that the training of most of consortia and university research groups in the students in the consortia was centred around best-practices software engineering.

7 3. Institutions should encourage industrial • providing dedicated training in problem internships for students and explore joint solving through the use of HPC. funding opportunities. From the sites visited and reviewed, it appears that 4. CCPs and other consortia should generate the university research groups are the main source tools and methods that are required in of training in modelling, simulation, and HPC in common by diverse projects in the UK, and that the majority of students after computational sciences. graduating continue working on aspects of HPC either in universities or in industry. Currently, 5. Funding should be provided on a longer much of the demand for trained professionals in time scale for computational science Ph.D. computing and modelling in the booming financial students. industry is satisfied by new graduates. The knowledge base on HPC related technologies is 3.4 IMPACT ON THE widely present within the UK research KNOWLEDGE ECONOMY community; the good placement record illustrates this finding. However, there appears to be no Two elements can be considered as measures of global strategy towards a consistent the HPC contribution to the knowledge economy: implementation of the action lines mentioned (1) technology transfer activities and (2) research above. collaborations between academic and industrial partners. It should be noted, however, that the Computational Science and Engineering Department (CSED) at Technology Transfer Activities Daresbury Laboratory has developed the expertise The transfer of HPC expertise and know-how to support code porting to HPC architectures, even from research universities to industry provides a though the EPSRC programme does not seem to competitive edge in the knowledge economy. fund dedicated efforts toward technology transfers Mechanisms for this technology transfer can be to industry. Although the CCPs supported by the any of the following: CSED have links with industry, including industrial members on working groups and the steering panel, • supporting industry in extending and it is unclear whether the actions mentioned above deploying their high-end technology through are being effectively implemented through these the use of HPC measures. • supporting industrial efforts to port in- Research Collaborations between Academic house, proprietary codes to dedicated HPC and Industrial Partners hardware inside the companies Some of the research consortia, such as UKAA, • stimulating access of industry to high-end UKTC, UKCP, and UKQCD, have strong industrial HPC facilities and intellectual resources collaborations and provide specialized expertise through adequate channels, taking into for the growth of UK knowledge economy. These account the issue of security and industrial collaborations make industry aware of confidentiality critical for industry- the benefits of HPC. proprietary software • The UK Applied Aerodynamics Consortium • training experts in HPC software (UKAA) is focusing its activities towards engineering, at the graduate or postgraduate high-end industrial applications in level, as a basis for transferring know-how aerodynamics, for external flows (helicopter to industry applications) as well as internal flows (gas turbine engine components), pushing HPC • providing dedicated training in HPC, simulations of industrial applications beyond including hardware and software the state of the art. engineering, databases, post-processing of data, and visualization • Similarly, the UK Turbulence Consortium (UKTC) is oriented towards fundamental applications in combustion and turbulence

8 but having a direct relevance for industrial In general, the research groups reviewed are using applications. computational technology effectively to deliver scientific results, and several groups are actively • The UK Car-Parinello Consortium (UKCP) developing and enhancing computational has developed a software system that is used technologies for the delivery of science. In worldwide and marketed to industry particular, UK researchers’ leadership in the through a commercial joint venture. development of scientific application codes is contributing to scientific progress worldwide. • The UKQCD consortium has a collaboration Examples of such codes (and the research groups with IBM in the development of a involved in their development) include: specialized supercomputer for lattice QCD simulations. This collaboration and the • GAMESS-UK, a general purpose ab initio emerging computer technology laid the molecular electronic structure program ground for IBM’s very successful BlueGeneL (Computational Science and Engineering systems. Department [CSED] at Daresbury Laboratory) Other consortia, oriented toward fundamental physics, biology, or chemistry, might not have • CASTEP, which uses density functional direct links with industry. However, the theory to compute the forces on the atoms experience gained within these areas might be and to simulate the time evolution made available to industry if more extensive (“dynamics”) of molecular systems (UK interdisciplinary cross exchanges between the Car-Parrinello Consortium) different consortia would be implemented. In addition, the universities should implement or • DL_POLY, a general purpose molecular strengthen such interdisciplinary cross training dynamics simulation package (CSED) between different fields in HPC, so that students • MOLPRO, a system of ab initio programs for could learn how solutions developed for a molecular electronic structure calculations particular problem might also be applied to with extensive treatment of the electron different domains. correlation problem (University of Birmingham) In general, the application oriented consortia have strong links with industry and provide a channel • Chroma, a software system for lattice QCD for industry to become knowledgeable about and calculations which is portable and efficient reap the benefits of HPC. on a wide range of architectures (UKQCD Collaboration)

3.5 STRENGTHS • HiGEM, a new high-resolution integrated climate modelling code which includes As stated in Section 3.1 above, the panel in atmospheric chemical influences (NERC general were very impressed with the quality of Centres for Atmospheric Science). science delivered through the use of HPC. The panel considered the formation of research The Computational Science and Engineering consortia (discussed in more detail in Section 3.6 Department at Daresbury Laboratory plays an below) to be one of the most important factors important role in supporting the research contributing to this quality. The creation of consortia by assisting in porting and optimising communities with a similar agenda facilitates the users’ codes, developing new applications and flow of scientific and technical know-how among algorithms, and evaluating new programming consortia members. The consortia also seem to be methodologies. Most of the consortia in close the main source of training in modelling, contact with the CSED expressed their satisfaction simulation, and data analysis using HPC. And the and sometimes enthusiasm about the support collaborations of several consortia with industry obtained from this group in areas such as are pathways for research using HPC to have a e-science. In Section 3.6 below, the Review Panel positive impact on the UK’s knowledge economy. recommend more opportunities for the kind of interdisciplinary communication and interactions that CSED exemplifies.

9 The UK has been very well served by the leading- The Panel observed that the allocation and edge HPCx and CSAR computational systems, by redistribution of resources within a consortium the special-purpose QCDOC machine, and by generally appears to be a smooth process; no many university-based computational systems. complaints were recorded. The HECToR program promises to build on this excellent tradition. HPC resources are discussed in Whereas some consortia live a life as pure more detail in Section 4 below. resource owners and redistributors (e.g., ChemReact), others have much more elaborate EPSRC’s recently initiated High End Computing agendas, including regular meetings to discuss Studentships combine training at one of the UK’s scientific and technical issues. In at least one case leading computation centres with a research (Materials Chemistry), we encountered a project at the student’s host institution, leading to consortium with a designated internal a Master’s degree or Doctorate in Computational communicator funded through the grant. Overall, Science and Engineering. The Review Panel see the Panel observed substantial differences among the studentships as a good strategy for training the consortia with respect to their presentations at the next generation of computational scientists and review: some consortia considered the review was engineers and for overcoming the boundaries and a major event, whereas for others it was a sheer barriers between academic departments. administrative process. Accordingly, the amount and quality of information that was conveyed varied considerably. 3.6 OPPORTUNITIES Funding for meetings and other consortia Research Consortia activities appeared to the Panel as valuable instruments for the exploitation of synergies The Panel feel that the consortium structure is an within the consortium. However, the Panel excellent mechanism for granting resources to the observed some reluctance to request an allocation research community. for consortium management support, as there was a fear of cannibalizing the allocation for the From an operational point of view, this mechanism scientific agenda. helps to smoothen the volatility in the demand for compute cycles commonly observed for research In certain instances, there is a strong overlap users. Consortia are able to nimbly re-apportion between the consortia and the Collaborative resources in response to sudden changes in Computational Projects (CCPs). Some consortia demand caused, for example, by the progression even appear to be offspring of the CCPs. The of the research or the addition or loss of CCPs represent an organizational structure which personnel, thus making resource management has grown organically from the very beginning of easier for all parties involved (servers and clients). HPC in the UK, and which is still in a process of For the computing centres (servers), the demand is growth. However, due to the focus on scientific more balanced, steady, and predictable, resulting disciplines and the lack of well established general in fewer compute cycles going to waste. Also, the HPC programs, there appears to be no strong communication between servers and clients is national computational science community. more coherent, as each consortium is represented (Section 3.3 above discusses some of the by one single voice. The more coherent drawbacks attached to this situation.) communication between users and compute infrastructure management makes the response to On the basis of these observations, the Panel user needs as well as the improvement of services makes the following recommendations: much easier. 1. The consortium model is an organizational From a scientific point of view, the formation of asset of HPC in the UK and should be communities with a similar but not necessarily maintained, if not expanded. In particular identical agenda facilitates the flow of scientific for large, distributed multi-group consortia, and technical know-how among consortia “coordination funds” should be automatically members. allocated, i.e., the proposers should be asked

10 to present an agenda for the management In our review of UK research using HPC, the and coordination of their consortium. Panel found several examples of outstanding and productive collaborations between applications 2. Mechanisms for coordination of efforts scientists and computer scientists or between consortia need to be defined. These mathematicians. The Reality Grid project at can be in the form of student exchanges, University College London demonstrated some meetings, workshops, etc. An enhancement first-rate examples of computational steering that of the transfer of technical know-how required Grid middleware, understanding of (algorithms, software engineering, data networking technology, and advanced processing) across disciplinary boundaries of visualization. Also commendable was the the UK community is needed. inclusion of a visualization researcher from the computer science department into the Institute for 3. However, before introducing new formal Computational Cosmology at the University of structures, the role of the consortia as well Durham. However, in general, these types of as the relationship between the consortia collaborations were the exception rather than the and the CCPs needs to be revisited in order norm. to avoid duplicate efforts or even competition. The Review Panel were surprised not to find more 4. The Research Councils need to keep an eye collaborations, given the high level of competence on the development of a national and standing of both computer science research computational science community to make and applied mathematics in the UK. For example, UK modelling and simulation more there was no mention of collaborations with competitive. numerical mathematicians, even though the UK is home to some of the world’s leading numerical analysts. The reasons for the absence of more Collaboration with Computer Science and collaborations are not obvious to the Review Applied Mathematics Panel. However, the Panel believe that the UK may be missing potentially important scientific Collaborations with computer scientists have opportunities, and therefore recommend that become a new element in computational science EPSRC devise strategies to increase the and engineering that often leads to successful new collaborations between the computer scientific accomplishments that would not have science/applied mathematics community and the been possible otherwise. The SciDAC (Scientific researchers using HPC technology. There should Discovery through Advanced Computing) program be plenty of opportunities in areas such as in the US has several examples of such successful numerical algorithms, visualization, scientific data collaborations. management, and distributed computing.

11 4. HPC Resources scheduling policies to allow significant code development. This requires rapid turnaround on smaller jobs, and may require setting aside special As explained in Section 2.1 above, high partitions of the machine during parts of the day, performance computing is essential to the future targeted exclusively for interactive code of science in the UK and around the world. To development. best realize scientific payoff for the funds invested, a diversity of resources and of controls are The Panel found an unnatural divide between the required. The Panel found a vibrant situation in ability of PPARC-funded researchers and EPSRC, the UK, spanning the spectrum from leading-edge NERC, and BBSRC researchers to access the facilities to university-based departmental and current high-end facilities (HPCx and CSAR). The even research group facilities. This feature must situation does not seem to be improving under be maintained as the UK goes forward. HECToR. While we understand the historical reasons for this, our sense is that this hurts British science, and we urge that this divide be 4.1 NATIONAL LEADING-EDGE reexamined. We are also concerned that it is COMPUTATIONAL SYSTEMS currently too cumbersome for users to casually try out the leading-edge systems. We have been told The UK has been very well served by the HPCx that there may be up to a six-month delay and CSAR systems, as their leading-edge systems, between submitting a request for such access and and by QCDOC as a special-purpose high-end its granting. Since the overall machine time in machine. The HECToR program promises to build such requests can be kept quite small, the impact on this excellent tradition. on other users can also be quite small, and the Panel urge that an expedited mechanism be found At the national level, leading-edge facilities are to allow such small-scale experimental use. (Large essential to support both capability computing facilities in the US can accommodate such (targeting substantial fractions of a tightly coupled requests within a day or two; and on the Earth resource at a single problem) and capacity Simulator in Japan, even a full-node job can be computing (running many smaller, largely run within two days. The UK might examine how independent jobs simultaneously). A balance the US and Japanese systems achieve this goal between these two is essential because both are while minimally impacting other users). fruitful scientifically and are often needed on the same scientific project—for example, generation of simulations may require capability computing, 4.2 UNIVERSITY-BASED while the analysis of these simulations may best COMPUTATIONAL SYSTEMS be done on capacity machines. Below the leading-edge, there has been excellent For both capability and capacity systems, there progress in distributing less powerful computing needs to be a diversity of architectures. One size facilities throughout universities. Much excellent does not fit all. One needs at least tightly coupled science is carried out on these machines by, for distributed-memory machines and large shared- example, the UKTC and UKCP groups at memory systems. Arguments can sometimes also Cambridge, the ICC at Durham, the Mineral be made for special-purpose machines (e.g., the Physics group at UCL, UKAFF at Leicester, and QCDOC machine in the UK). While all of these many others. These systems also serve as machines operate primarily in standard batch development platforms for work that migrates to mode, they can be scheduled to allow interactive the national high-end facilities. access and even co-scheduling with other facilities (e.g., for visualization). One must also adopt Local facilities have many advantages over national ones:

12 • They can be more nimble. For example, changes in research direction or approaches Leadership Capability can be immediately attempted without ) Computer(s Computing formal justification. High-end • New casual users can be immediately National Capability-Capacity uters accommodated. Supercomp Computing

Local • Attracting and training students is easier. University- Capacity partment- De rs pute Cluste Computing ° Local systems can be easily integrated Group Com into formal university-based courses Personal involving other departments, mputers Computing Personal Co ations strengthening the interdisciplinarity and Workst which the Panel feel is so desirable.

° Students can learn systems Figure 1. The spectrum of computing resources. administration.

° Local facilities can be showcased in 4.3 DATA RESOURCES programs aimed at pre-college students, as is done so well at the ICC in Durham. A majority of the computational science disciplines the Panel reviewed rely on numerical • Links to local storage and visualization solutions of physics equations. Data is principally resources are tighter and easier to the output of these simulations. With the implement. extraordinary increase in computing power, the • Pride of ownership increases political amount of data output and the ability to store support throughout the nation for the larger fine-grained data from trajectories or time series HPC program. analysis of systems is beginning to produce enormous volume of data. This, by itself, would Local facilities are the major route by which new warrant a change in the way the computational users are attracted to HPC, which is so vital, as science community handles data. has already been pointed out. However, another revolution is changing the role The Panel found significant apprehension that of data in computation and has in some cases led with the termination of Science Research to a shift in the data paradigm towards data-driven Investment Fund (SRIF) and Joint Infrastructure computing. This revolution was triggered primary Fund (JIF) support, and under Full Economic by having data be the output of automated Costing, it will be increasingly difficult to procure measurement, primarily in life sciences, these local facilities in the future. It is critical that astronomy, environmental science, climate studies, they be sustained and upgraded. and computational particle physics. In this revolution, terabytes of data are already being The hierarchy of computing resources is often produced, and soon it will be petabytes. These captured in the notion of pyramid with a broad data, often spanning multiple dimensions and base (many more smaller systems) and a narrow containing valuable information about functional top (a few leading-edge systems), as shown in systems, are not amenable to easy abstraction or Figure 1.5 Scientific progress depends on ready modelling with equations of motion. Sometimes access to the full spectrum of computing they may provide parameters and boundary resources. conditions for physics-driven modelling. It is mandatory for computational science to

5 The need for a hardware hierarchy to support computational research was first articulated in a 1993 report to the US National Science Board entitled “From Desktop to Teraflop: Exploiting the U.S. Lead in High Performance Computing,” frequently referred to as the “Branscomb report.” The famous “Branscomb pyramid” was updated in January 2005 by the Subcommittee on Theory and Computation of the US Basic Energy Sciences Advisory Committee in “Opportunities for Discovery: Theory and Computation in Basic Energy Sciences.”

13 recognize the importance of data management and metadata catalogues. This is a major area of analysis. The international community have research in computer science, and brokering already made inroads in this emerging discipline. of data and ontologies between databases During site visits to UK institutions, the Panel requires planning and design. The outcome found that most of the research groups are aware of such queries can sometimes be entirely of the need to focus on data, but do not have new knowledge. specific plans for, or a comprehensive understanding of, how to handle data flexibly and 3. Often the size of the data is beyond simple efficiently. The UKQCD consortium are the most visual analysis by humans. The data needs advanced and articulated their data needs most to be reduced to smaller dimensions and clearly. They have initiated the International details, or the data needs to be mined using Lattice Data Grid (ILDG) and have developed supervised and unsupervised machine Grid tools for semantic-based data access using learning tools. Often these tools warrant metadata catalogues and XML schemata. UKQCD extensive statistical algorithms and already have a middleware layer running, have significant computational time. successfully tested a prototype of their setup, and 4. The most important task involves the are now ready to go into production mode. UKTC integration of diverse data and legacy have put the data generated from direct numerical knowledge to produce new knowledge, simulations of turbulence and transition on a either by combining data on an international Web-based database, currently accessible level or by purely querying the data. Each worldwide, with ten registrations for use received discipline will have to develop specialized between May and September 2005. The UKTC mechanisms to achieve this integration. databases, under continuous development, make available statistical summary data (Moser format), Based upon this analysis and our review of the complete flow field data, and associated Fortran research groups, the Panel recommend the code for I/O. The Integrative Biology Grid is following: arguably the most heterogeneous data collection, and its organizers are also aware of the data 1. Each consortium begin to generate design management problem, but are only in the early plans for handling and analyzing data in stages of planning. The CSED group at Daresbury, their discipline. Large consortia should who provide a large computing resource, organise meetings to document data needs expressed the importance of planning for data of different communities with a view to needs, but believed that the e-Science programme identifying new strategies that need to be is addressing this issue. The ICC in Durham plan adopted. to make their cosmology data publicly accessible but do not yet have a fixed concept on how to do 2. Specific funding and a proposal mechanism it. should be developed to create a workforce that develops common tools and There are four general issues connected to data: infrastructures for data handling, organization, manipulation, and creation of 1. Data, whether from experimental high- user interfaces. The workforce should throughput measurements or from large- evaluate how much of the technology scale simulations, need to be organised, developed by the UKQCD consortium can stored, and disseminated. This requires be taken over for other consortia. The establishing structured mechanisms of workforce should create a library of organising, databasing, and providing the software for scientists dealing with large tools for dissemination of the data. This task amounts of data. is highly discipline-specific, and context- specific infrastructure needs to be built on 3. Funding agencies announce targeted some common foundations. requests for applications to encourage interdisciplinary projects that involve data, 2. Diverse sources of data, such as collections computation, and visualization. of distinct databases, need to be queryable in an interoperable manner, which requires

14 4.4 VISUALIZATION believe that effective user support is critical to RESOURCES productivity in HPC. Most computational science groups in a university have one or a few “go-to” While the reviewed research groups well people to whom the students or postdocs turn for understand that visualization is indispensable in answers to computing questions. Such support computational science, the Panel observed that needs to be nurtured. The Panel have already visualization in the UK lags behind international argued for more support within individual standards. A few groups showed us visualization university groups for people who supplement labs, but they appeared to use conservative their applications expertise with software visualization devices and tools. An exception is the engineering or algorithm expertise. ICC at Durham, who showed us an impressive 3D movie of cosmology. The Panel are concerned that But this is not enough. The research community without an improvement in visualization also needs national, institutionalized user support, sophistication (both hardware and software), where the practitioners have a long-term career hidden scientific treasures will increasingly lurk path, which is not the case in a typical university undiscovered in the massive data to be produced environment. This support must be provided by by the enhancement of HPC capability and well-rounded professionals who can speak to the capacity. users as scientific peers, but who also have excellent understanding of algorithms, code As an example of the state of the art, the Theory optimization, and software engineering, as well as and Computer Simulation Centre and the Earth a mindset geared to helping others rather than a Simulator Centre in Japan use a two-stage focus on independent scientific research. Such visualization system: the first stage is a Cave people can be a source of cross-fertilization virtual reality system, whereby overall global between different research groups, recognizing evolutionary behaviour is surveyed to identify that the expertise of one group can help another. interesting events; then, in the second stage, The Panel were told that HECToR foresees detailed analysis is made to explore these events supporting such people for a targeted portion of more fully using advanced desktop visualization the total budget, which we applaud. CSED at systems. Daresbury appears to play this role already, and the Panel often heard testimony of users’ In the forthcoming leading-edge facilities in the satisfaction with the support they received from UK such as HECToR, much more massive and Daresbury. valuable data will be produced. Therefore, the Panel recommends that the UK HPC community It is not essential that such a user support team be prepare immediately to establish a balance among co-located with the leading-edge machines, but it leading-edge computing facilities, visualization helps. Many user issues require system responses, technologies, and well-educated computational such as allocating large disk quotas for a restricted scientists. period, special scheduling for individual jobs, etc. It improves communication and reduces institutional suspicions if the requests for such 4.5 USER SUPPORT system accommodations come from someone within the institution who represents the external Examining user support was not explicitly in the user. mandate for this review. However, the Panel

15 5. Conclusions and leveraged across multiple users. The productivity gains are obtained by more rapid insertion of new Recommendations technology into applications codes and by realization of technology transfer between research groups. During the course of this review, the Panel observed a range in the quality of science 2. Strengthen the computational delivered through the use of HPC in the UK but infrastructure by: overall were very impressed. The Review Panel a. systematically deploying leading-edge judged that in many of the consortia and research capability systems, large-scale capacity groups visited, the scientific results compared computing, and resources deployed well with the highest international standards. widely at universities With respect to computational resources, the Panel found a vibrant situation in the UK, The Panel believe that all elements of the spanning the spectrum from leading-edge facilities computational infrastructure (the “pyramid” of to university-based departmental and even Figure 1) are important for creating a healthy and research group facilities. The EPSRC’s recognition productive HPC environment. In our review we of the importance of HPC and the research saw examples of good and appropriate use of the community’s already strong demand for HECToR different elements of the pyramid. This balanced indicate that the UK research environment is well approach should be continued in the future, with positioned for fostering further scientific progress investments being made at all levels of the through HPC. Thus the state of research using pyramid. HPC in the UK is excellent, with researchers being engaged in scientific projects at the b. supporting and developing a forefront of the challenges in their field, making state-of-the-art applications software internationally recognized, significant infrastructure encompassing contributions, and having access to state-of-the-art algorithms, data management and platforms. analysis, visualization, and best- practices software engineering. Research using high performance computing is a rapidly growing and dynamically changing field. In spite of some notable examples to the contrary, The Panel therefore make several the Panel found that UK researchers using HPC recommendations that will contribute to the are often not aware of or lack the resources to continued improvement of the research apply the latest results in numerical algorithms, environment for HPC in the UK: data management and analysis technologies, and visualization. They are also often not aware of 1. Create a more balanced HPC software engineering practices. The Panel infrastructure between computational recommend devising strategies to link computer technologies and intellectual resources. science research in these areas more closely to the applications scientists, and facilitating innovative The UK has a well thought-out investment means for computer scientists and mathematicians strategy with HPCx and CSAR platforms in the to collaborate with researchers using HPC. past and HECToR in the near future. However, as high performance computing platforms become 3. Develop human resources in HPC. increasingly more difficult to use efficiently, the gap between users and non-users becomes larger. The future of CSE depends critically on the HPC platforms generally can be utilized more availability of highly specialized experts. Because effectively if users have access to intellectual of its interdisciplinary nature, research using HPC resources, for example experts in code requires specialists who have received additional optimization, parallel algorithms, etc., who can education and training beyond traditional work with the researchers as consultants or academic disciplinary programs. While EPSRC has collaborators. A relatively small group of such already taken promising initial steps such as the experts, for example associated with HECToR, can HEC studentships, the Panel encourages further achieve high productivity gains if they are initiatives in the UK that will lead to increasing

16 support for the trained professionals and experts disciplines, an all-embracing computational in CSE, as well as the development of formal science community (in the sense of an “academic programs for the education and training of community”) does not yet exist in the UK. The computational scientists. Panel recommends taking proactive steps towards the creation of such a computational science 4. Bridge disciplines and build a community by such means as encouraging and computational science community by developing local workshops and national meetings, increasing interactions and fostering fostering support for a professional society in the collaborations between disciplinary groups field, and participating in leading international nationally and internationally. events such as the annual International Supercomputer Conference and the SCxy The Panel notes that while there are several conference series. examples of excellent collaborations in individual

17 6. Acknowledgements

The Panel are grateful to the Steering Group, in particular Harald Knobloch, for their excellent preparatory work and for their assistance throughout the review. The Panel are also very appreciative of all the logistical support provided by Linda Sayers. Hugh Pilcher-Clayton is acknowledged for briefing the Panel and answering questions about EPSRC. John Hules is acknowledged for preparing the background data and editing this report. Finally, the Panel would like to express their sincere appreciation to the universities visited, where both staff and students invested considerable efforts in ensuring that the visits were informative and enjoyable.

18 7. Appendices 7.3 TERMS OF REFERENCE The International Review Panel is requested to:

• report on the calibre, standing and research potential in research using High 7.1 STEERING GROUP Performance Computing in the UK

Professor Ron Perrott, Chair, • discuss the potential impact of university- Queen’s University Belfast based research using High Performance Professor John O’Reilly, EPSRC Computing on the UK’s knowledge Professor Ernst-Ludwig Winnacker, DFG economy Dr. Horst Simon, Lawrence Berkeley National Laboratory • provide comparisons with international Dr. Randal Richards, EPSRC research using High Performance Dr. Christoph Schneider, DFG Computing Ms. Linda Sayers, EPSRC Dr. Harald Knobloch, DFG • make recommendations on future actions and/or priorities

7.2 PANEL MEMBERSHIP 7.4 RESEARCH GROUPS Dr. Horst Simon, Chair, REVIEWED Lawrence Berkeley National Laboratory Professor Hans-Peter Bunge, Centre for Computational Science Ludwig Maximilians University (CCS)/RealityGrid Professor Roberto Car, Princeton University ChemReact Computing Consortium Professor Charles Hirsch, CLRC Computational Science and Engineering Vrije Universiteit Brussel Department (CSED) at Daresbury Laboratory Dr. Karl Jansen, HPC Materials Chemistry Consortium J.v.Neumann-Institute for Computing, Institute for Computational Cosmology (ICC), DESY, Zeuthen University of Durham Professor Hans Peter Lüthi, Integrative Biology at the University of Oxford Swiss Federal Institute of Technology, Zurich Mineral Physics at University College Professor Ralph Z. Roskies, London (UCL) Pittsburgh Supercomputing Center Multiphoton, Electron Collision, and Bose Professor Tetsuya Sato, Einstein Condensates (MECBEC) HPC The Earth Simulator Center Consortium Professor Shankar Subramaniam, NERC Centres for Atmospheric Science (NCAS) University of California, San Diego UK Applied Aerodynamics Consortium (UKAA) Professor William M. Tang, UK Astrophysical Fluids Facility (UKAFF) Princeton University UK Car-Parinello Consortium (UKCP) UK Turbulence Consortium (UKTC) UKAEA Fusion Research at Culham Science Centre UKQCD Collaboration

19 7.5 REVIEW SHEET II. Resources and instrumentation

Research Institution: Please comment on the following aspects:

Reviewer: 1. Computer resources (e.g., access to computing resources, availability of I. International standing and research architectures, allocation of resources, etc.) potential 2. Human resources

Please comment on the following aspects: 3. Level of funding/success in fundraising 1. Evaluation of Research in comparison to the III. Open discussion on strategic issues for international level promoting science using high performance 2. Impact of research activities to international computing in the UK progress in the scientific field IV. Recommendations on future actions 3. Visibility on an international level and/or priorities 4. Publication activities and participation in V. Other comments conferences

5. Research progress during the last 3–5 years VI. Conclusion

6. Impact on the UK’s knowledge economy

20