Quick viewing(Text Mode)

Introduction of Supercomputer System for DNA Data Bank of Japan

Introduction of Supercomputer System for DNA Data Bank of Japan

Introduction of for DNA Bank of Japan

V Hideaki Sugawara V Hiroaki Yamada V Hisahito Yamada (Manuscript received March 19, 2008)

The Center for Biology and DNA Data Bank of Japan (CIB-DDBJ) at the National Institute of Genetics (NIG) has joined forces with GenBank (USA) and EMBL (Europe) to the International Nucleotide Sequence Collaboration (INSDC) and make their accessible to researchers over the . In parallel with advances made in the life sciences, the INSDC databases are increas- ing rapidly in size by 30–50% annually. Moreover, a large-scale system for constructing and releasing them and providing search and analysis services has become necessary. Important requirements of such a system will include ongoing enhancements to system performance and capacity and support for evolu- tion. This paper describes a new supercomputer system that went into operation in March 2007. It meets a variety of difficult requirements including support for a rap- idly growing database, provision of high-speed and stable search/analysis services, smooth migration from the old system, compatibility with existing assets, high reliability, and support for security. This paper also introduces the contributions that this new system is making to life sciences both inside and outside Japan.

1. Introduction issues and how it is contributing to Japanese and In February 2007, the National Institute of international research in the life sciences. Genetics (NIG) of the Research Organization of Information and introduced a super- 2. DDBJ overview computer system consisting of large-scale CIB-DDBJ is working with EBI/EMBL4) servers, clusters of personal (PCs), in Europe and NCBI/GenBank5) in the USA as large-capacity storage, keyword search systems, part of the DDBJ/EMBL/GenBank International and other components. It has been in opera- Nucleotide Collaboration tion since March 2007. This supercomputer (INSDC).6) The INSDC databases consist of system aims to provide data collection, database deoxyribonucleic (DNA) (or ribonucleic acid construction, data release, and search and [RNA]) nucleotide sequences experimentally analysis services for the DNA Data Bank of determined by researchers around the world Japan (DDBJ)1)-3) operated by the Center for and collected and edited by three inter- Information Biology and DNA Data Bank of national databanks—DDBJ, EMBL, and Japan (CIB-DDBJ) at NIG. It is provided for GenBank—according to a data construction both NIG and research institutions inside and scheme established by these three parties. To the outside Japan for research purposes. user, this data is provided in the form of a unified In this paper, we describe how this super- computer file. The databases are prepared by computer system has solved a number of difficult editing DNA nucleotide sequence data received

442 FUJITSU Sci. Tech. J., Vol.44, No.4, pp.442-448 (October 2008) H. Sugawara et al.: Introduction of Supercomputer System for DNA Data Bank of Japan

directly from researchers, but they also include in data from here on. DNA data processed by the Japan Patent Office, Each of the INSDC databanks (DDBJ, European Patent Office, and United States Patent EMBL, and GenBank) makes its data accessible and Trademark Office. over the Internet and provides keyword search The nucleotide sequence database is a set and homology search services (discussed in more of data units called “entries”. In addition to the detail later). These databanks have, as a result, nucleotide sequence itself, each entry includes come to be used by researchers throughout the information about the researcher who determined world. The database provided by each databank the sequence plus related references, organism, may be copied and used by anyone in the world. and gene function and features. Data collect- However, the ongoing increase in the amount of ed by each databank is exchanged daily among data makes it difficult for a user to keep a copy the three databanks so that the same informa- up to date. For this reason, these search services, tion will be provided by them. DDBJ collects and which enable a user to use a personal computer edits about 20% of the data released by these to search for data in a database that is always three international databases. maintained with current data from around the When DDBJ first released its nucleotide world, are increasing in value, and they have sequence database in July 1987, it consisted of been in great demand by researchers. only 66 entries and 108 970 base pairs. In recent years, however, the INSDC databases have been 3. New computer system increasing at an annual rate of 130–150%. In supporting DDBJ operations September 2007, about 20 years after the initial To provide a dependable database that release, they had reached 76 273 345 entries and continues to grow at an explosive rate, as 79 706 204 461 base pairs. The growth in data described in the previous section, it is impor- registered in INSDC is shown in Figure 1. tant that the performance and capacity of the In addition, sequencers that can computer system be continuously upgraded and operate several tens to a hundred times as fast that software be allowed to evolve. Six years as conventional sequencers are now being intro- after the launch of the old system, it became duced, and this should only accelerate the growth difficult to support the rapid increase in data and the demand for new services. Plans for upgrading to a new system were drawn up in   2005–2006, culminating in the delivery of a .UCLEOTIDES new supercomputer system from Fujitsu. This  %NTRIES  

 section summarizes the requirements of this new   supercomputer system and explains how those

  requirements were met. %NTRIES§

.UCLEOTIDES§   3.1 Requirements of new supercomputer   system 1) Support explosive growth in data • Support a massive amount of nucleotide               sequence data that increases by 1.5 times 3OURCE$$"*$$"*%-",'EN"ANKDATABASEGROWTH annually, making for about 400 million

Figure 1 entries in five years time INSDC database growth. • Shorten the update interval for all data

FUJITSU Sci. Tech. J., Vol.44, No.4, (October 2008) 443 H. Sugawara et al.: Introduction of Supercomputer System for DNA Data Bank of Japan

(from 3 to 2 months) database , achieving high-speed and 2) Provide high-speed and stable stable response. search/analysis services 2) Provision of high-speed and stable • Achieve high-speed response and high search/analysis services throughput in homology search services • For keyword searches, it achieves a stable • Achieve high-speed and stable response in response within ten seconds for 30 million keyword search services regardless of search entries of complex data, even for complicat- conditions for data that is continuously ed search conditions. growing • For homology searches, it achieves improved 3) Achieve smooth migration from the old standalone performance five to eight times system and provide user-asset inheritance the old level and high-throughput perfor- and compatibility mance by parallel processing. • Achieve smooth and rapid migration of 3) Smooth migration from the old system and the large number of existing application user-asset inheritance and compatibility programs used for diverse services • For core , it uses the Solaris 4) Achieve high reliability and provide security (OS), the same as the old measures system. • Achieve a safe and secure system that • For homology search processing, it uses supports data exchange among the three Linux as the de facto standard OS. international databases and the provision • For analysis processing, it uses both Solaris of services to researchers inside and outside and Linux to provide optimal processing Japan according to the characteristics of the analy- sis program. 3.2 Features of the new supercomputer 4) High reliability and security measures system • It supports active maintenance and a redun- The new supercomputer system was devel- dant configuration and detects faults in oped to satisfy the above requirements. Its advance through a diagnostics monitoring features are summarized below. function. 1) Resource allocation and system design • It uses Symfoware and HiRDB, which are supporting explosive growth of data domestically developed RDBMS software. • For core data processing, the system combines large-scale symmetric multi- 3.3 Configuration and performance of new processing (SMP) servers, large- supercomputer system capacity storage, equipment, The main items of equipment making up and the latest the newly introduced supercomputer system are management system (RDBMS) software listed below and shown in Figure 2. to achieve high-speed processing of large 1) Database construction/release servers volumes of data. Large-scale SMP server: PRIMEPOWER2500 • For homology searches and analysis process- (28 CPUs) ing, it uses a new PC-cluster system, Large-capacity storage: ETERNUS8000 achieving high-speed response and high (125 TB) throughput through distributed processing. Blade server: BladeSymphony • For keyword searches, it uses a large-scale 2) Homology search server extensible (XML) PC cluster: PRIMERGY RX200 S3 (66 nodes)

444 FUJITSU Sci. Tech. J., Vol.44, No.4, (October 2008) H. Sugawara et al.: Introduction of Supercomputer System for DNA Data Bank of Japan

3) Keyword search server es. This system consists of ETERNUS8000 Large-scale XML : 125 TB high-capacity storage equipment and ShunsakuEngine (2520 CPUs) a BladeSymphony blade server centered on a 4) Analysis servers PRIMEPOWER2500 (featuring 28 CPUs and Large-scale SMP server: PRIMEPOWER2500 88 GB of memory) large-scale SMP server running (32 CPUs) the Solaris OS, which has a proven record of PC cluster: PRIMERGY RX200 S3 (66 nodes) stable operation of large-scale database systems. 5) Backup equipment Two types of database management software are Tape : VD800 (750 TB) used for different applications: HiRDB is used for 6) Other servers/network equipment the database construction system and Symfoware Network server: PRIMERGY is used for the database release system. Both RX200 S3/RX300 S3 the database and release data are stored in Switches/routers: the ETERNUS8000 high-capacity storage equip- Catalyst6506/IPCOM S2400, etc. ment, and in addition to performing periodic The performances of the new supercomputer to VD800 backup equipment, the system system and the old system are compared in terms can perform remote backups of especially impor- of major components in 1. tant data to disk drives connected to servers belonging to the National Institute of . 4. Details of new genome The database construction system stores sequence database system nucleotide sequence analysis data submitted 4.1 Database construction/release servers mainly by researchers in Japan using either The database construction/access servers the SAKURA8) nucleotide sequence submis- constitute a system for uniformly managing sion system or a mass submission system. This DDBJ data and creating data for release purpos- data is checked and edited by specialists called

.ETWORKSERVERS 02)-%2'9 #ATALYST)0#/- )NTERNET 283283 3

+EYWORDSEARCHSERVER (OMOLOGYSEARCHSERVER

3HUNSAKU%NGINE 02)-%2'983 #05S NODES

$ATABASECONSTRUCTIONRELEASESERVERS !NALYSISSERVERS

02)-%0/7%2 #05S#05S

02)-%2'983 NODES "LADE3YMPHONY "ACKUPEQUIPMENT

%4%2.53 6$ 4" 4"

Figure 2 New supercomputer system.

FUJITSU Sci. Tech. J., Vol.44, No.4, (October 2008) 445 H. Sugawara et al.: Introduction of Supercomputer System for DNA Data Bank of Japan

“annotators” at NIG. Checked data is edited into making extensive use of multiple-alignment units of entries and passed on to the database analysis based on biological similarities among release system on a daily basis. nucleotide sequences and amino acid sequences. In addition to data received from the The DDBJ provides Web-based services11) database construction system, the database for conducting homology searches by the above release system edits updated data received software or multiple-alignment analysis (by from EMBL/GenBank also on a daily basis and CLUSTALW) of the INSDC database and converts it to DDBJ format. The system releases other databases having more than 70 million all daily edited data of the three internation- entries (as of November 2007), including a al databases. The entire database is released key-protein database. Search requests can be in entry (flat-file) format, DDBJ-XML format, made by E- or from programs using a Web INSDC-XML format, or FASTA format. Users application programming interface. Search may download data from a DDBJ anonymous results may be viewed on a or FTP site9) and may data using the received by E-mail. getentry10) search system (in units of entries) The homology search server uses a PC or by the homology/keyword search systems cluster consisting of 66 nodes (2 of which are described below. management nodes) of PRIMERGY RX200 S3 servers. Each node has two 3.0-GHz dual-core 4.2 Homology search server CPUs, 8 GB of memory, and 140 GB of hard disk A homology search searches for biological storage (RAID1) and runs 64-bit-mode Linux for similarity among DNA, RNA, and amino-acid the OS. sequences. In life sciences research, it is widely In response to a search request from a user, used for estimating genetic function, evolutionary the system automatically distributes the request analysis, etc. There are various types of software to all the nodes using a network queuing system. for conducting homology searches depending on This minimizes the waiting time and achieves the type of data targeted for retrieval and the a high-speed response and high throughput for used to conduct the search. At present, the user. The HMMPFAM , in particu- researchers are making much use of BLAST, lar, achieves ultrahigh-speed processing by using PSI-BLAST, FASTA, SSEARCH, and HMMPFAM, parallelization software developed by Fujitsu for example. Life sciences researchers are also Laboratories. As a result of these performance improve- ments, the usage rate of the popular Table 1 BLAST/FASTA/CLUSTALW services increased Performance comparison with old system. by 10–30% in October 2007 over the same month 0ERFORMANCE 3ERVICEAPPLICATION %QUIPMENT COMPARISON a year earlier. 5.)8SERVER $ATABASE "LADESERVER !BOUTTIMES 4.3 Keyword search server CONSTRUCTIONRELEASE 3TORAGE The DDBJ provides the ARSA12),13) integrat- +EYWORDSEARCH 3HUNSAKU%NGINE !BOUTTIMES ed keyword search service targeting INSDC (OMOLOGYSEARCH 0#CLUSTER !BOUTTIMES and 24 other important databases in the life 5.)8SERVER sciences. In past keyword search systems such !NALYSIS !BOUTTIMES 0#CLUSTER as SRS,14) the deterioration in response time 3TORAGE 3!.STORAGE !BOUTTIMES accelerated as search conditions became more "ACKUPEQUIPMENT 4APE complex. Moreover, as the volume of data

446 FUJITSU Sci. Tech. J., Vol.44, No.4, (October 2008) H. Sugawara et al.: Introduction of Supercomputer System for DNA Data Bank of Japan

increased, a relatively long time was needed to 4.5 Backup equipment create a keyword index. In the light of these For backup purposes, the supercomputer problems, the ARSA keyword search service system uses a VD800 tape library with a capacity uses the ShunsakuEngine, a server dedicated of 750 TB. This equipment comes with 6.25 TB to large-scale XML databases equipped with of hard disk storage for automatically backing up the Fujitsu-developed Shunsaku XML database and restoring data to and from tape. By access- engine loaded in dedicated hardware. This ing these disk drives, a user can use the backup server provides a response time of less than 10 s equipment as if it were a 750-TB disk system (3 to 4 s on average) even for complex, compound without been aware of the actual tape library. search conditions for 24 databases with This backup equipment is normally used for 30 million entries (as of November 2007). The backing up the DDBJ database, but it can also be ShunsakuEngine has 168 core CPUs and a used by NIG researchers on application. total of 42 GB of memory in a 2 U chassis and includes a dedicated for retriev- 4.6 Other network servers/equipment ing data in memory, resulting in short search The servers described above are separated response times in addition to a space-saving and from the Internet by a firewall at NIG. To provide power-saving configuration. The entire keyword robust security, Web servers, FTP servers, and search system uses 15 ShunsakuEngine units mail servers for releasing data to the Internet are and 9 Linux servers mounted in two racks. As a installed in a “demilitarized zone”, and systems result of the above deployment, the usage rate of that store crucial data like the INSDC databases the keyword search service increased by 10–20% are separated in sub-networks. in 2007 over the previous year. 5. Future outlook 4.4 Analysis servers This supercomputer system has been NIG operates—for joint research and analy- operating since March 2007 in a stable fashion sis use—the PRIMEPOWER2500 SMP-type without any major problems. The plan is to server featuring 32 CPUs, 128 GB of memory, enhance system performance and capacity by the Solaris OS, which is applicable to large-scale about two times at the two-year mark after memory jobs, and a Linux PC cluster system system introduction. The idea here is to upgrade with 66 nodes (2 of which are management the system after analyzing how it is being used nodes), which is applicable to distributed memory while also giving sufficient consideration to usage jobs. Each node of the PC cluster system has trends in information systems in the field of life two 3.0-GHz dual-core CPUs, 8 GB of memory, sciences research. and 140 GB of hard disk storage (RAID1) using 64-bit-mode Linux for the OS. In addition 6. Conclusion to ordinary Gigabit Ethernet, the PC cluster is This paper described how a newly intro- equipped with a high-speed InfiniBand intercon- duced supercomputer system has solved a nect, which achieves high-speed processing of number of key issues surrounding the operation distributed programs. of a very large nucleotide sequence database in Researchers at NIG can make use of life sciences research. We can expect the amount analysis servers by application, and outside of life sciences data available to researchers researchers can do the same if they are involved to continue its dramatic growth in the years to in joint research with NIG. come, but we also foresee a rapid move away from the separate databases and separate research

FUJITSU Sci. Tech. J., Vol.44, No.4, (October 2008) 447 H. Sugawara et al.: Introduction of Supercomputer System for DNA Data Bank of Japan

themes of the past to an interlinking of data 4) EMBL-EBI: European Institute. http://www.ebi.ac.uk/ and the pursuit of the “big picture” in biologi- 5) NCBI: GenBank Overview. cal phenomena. There will be a need to manage http://www.ncbi.nlm.nih.gov/Genbank/index.html 6) INSDC (International Nucleotide Sequence more complicated data in a more systematic way Database Collaboration). and provide user-friendly application services. http://www.insdc.org/page.php?page=home 7) DDBJ: DDBJ/EMBL/GenBank database growth. Our goal is to operate this new supercomputer http://www.ddbj.nig.ac.jp/images/ system as a cutting-edge information platform breakdown_stats/DBGrowth-e.gif 8) DDBJ: SAKURA. for life sciences research in Japan and through- http://sakura.ddbj.nig.ac.jp/ out the world. 9) DDBJ: FTP Listing of Root at ftp.ddbj.nig.ac.jp. ftp://ftp.ddbj.nig.ac.jp/ 10) DDBJ: getentry. References http://getentry.ddbj.nig.ac.jp 11) DDBJ: Search and Analysis. 1) DDBJ home page. http://www.ddbj.nig.ac.jp/search/ http://www.ddbj.nig.ac.jp 12) DDBJ: ARSA (All-round Retrieval of Sequence http://www.ddbj.nig.ac.jp/index-e.html and Annotation). 2) H. Sugawara et al.: DDBJ with new system http://arsa.ddbj.nig.ac.jp/ and face. Nucleic Acid Research, 2007, 1-3 13) N. Nishinomiya: Introduction of Interstage doi:10.1093/nar/gkm889. Shunsaku to of International http://nar.oxfordjournals.org/cgi/content/full/ Nucleotide Sequence Database. (in Japanese), gkm889v1 FUJITSU, Vol.56, No.1, pp.54-58 (2005). 3) M. Yamaguchi: Introduction of the DNA Data http://img.jp.fujitsu.com/downloads/jp/jmag/ Bank of Japan (DDBJ). (in Japanese), FUJITSU, vol56-1/paper05.pdf Vol.53, No.5, pp.353-358 (2002). 14) DDBJ: SRS (Sequence Retrieval System). http://img.jp.fujitsu.com/downloads/jp/jmag/ http://srs.ddbj.nig.ac.jp/ vol53-5/paper04.pdf

Hideaki Sugawara Hisahito Yamada National Institute of Genetics Fujitsu America Inc. Professor Sugawara received the Ph.D. Mr. Yamada received the B.S. degree in Applied Physics from the University in Physics from Kochi University, Kochi, of Tokyo in 1973. He joined the Life Japan in 1989. He joined Fujitsu Ltd., Science Information Section of RIKEN in 1989, where he introduced and sup- in 1977 and moved to CIB-DDBJ, NIG ported supercomputer systems and net- in 1996. He has developed a number work systems. From October 2004, he of biological databases and Web ser- introduced and supported large-scale vices. He has also contributed to inter- server systems and networks. In July national initiatives such as the Global 2008, he joined Fujitsu America Inc. Biodiversity Information Facility, Global Biological Resource Centers Network, World Federation for Culture Collections and Committee on Data for Science and , and International Council for Science.

Hiroaki Yamada Fujitsu Ltd. Mr. Yamada received the B.S. degree in from Tokyo Institute of Technology, Tokyo, Japan in 1978. He joined Fujitsu Ltd., in 1978, where he supported laboratory automa- tion systems for nuclear engineering. He is currently supporting bioinformat- ics systems.

448 FUJITSU Sci. Tech. J., Vol.44, No.4, (October 2008)

Top View