RIKEN Super Combined Cluster System Application Form

Advanced Center for Computing and Communication, RIKEN http://accc.riken.jp/index_e.html Revised on June 3, 2010 Making proposals for RIKEN Super Computer System Advanced Center for Computing and Communication

1. Introduction

The Advanced Center for Computing and Communication (the Center) has started a routine operation of RICC on October 1, 2009. RICC supports the development of large-scale parallel applications, which demand is expected to increase, and GPGPU accelerators which is becoming a growing trend in scientific computation.

We do not charge fees for using the RIKEN Supercomputer System. This is because we want to provide computation environment as free and comfortable as possible to RIKEN researchers and help them obtain better research results. However, more than 800million yen are being spent to keep this environment in a year. To share computers between users means research fee necessary to conduct researches is shared between computer users. So, our policy is to review research proposals beforehand for fair allocation of computational resources.

For the review, proposers estimate the computation time they will need, and their proposals are reviewed whether they are valuable worth the computation time (cost). Small proposals that use less than 1% of the total resources are reviewed by the Center at any time upon submission. Proposals that use 1% or more of the total resources are reviewed by the Committee for the Review of Research Project Proposals Using the Supercomputer System (the Review Committee) that meets four times a year. Committee members are selected from extensive research fields that RIKEN is actively involved in. Since the review is performed by all committee members whose research area may be different from yours, it is requested that you write the description of your research theses comprehensible.

We appreciate if you could understand this policy, estimate necessary computation time and cooperate with us for the fair review of proposals. You are strongly requested to read the Supercomputer System Usage Policy before you make applications.

Usage Report and Research Result

Usage report and the handling of research results are stipulated in the Supercomputer System Usage Policy. We abstract the concerned chapters as follows. Using the system violating the Policy may result in the cancellation of your account or will affect future proposals from you.

6. Usage Report

Users submit the Usage Report for each proposal. Submission period is a fiscal year end or the end of usage. If no Usage Report is submitted, we will not accept the user's making application from next fiscal year and the continuous use of the System by the user. The report must include the description of computation/research users have performed and research achievement they have made with the attachments of "List of research achievement" 1 and "Copies of Published Papers, etc".

7. Notes on Making Publications

When users make publications on research achievements obtained by the use of the System, they must acknowledge the use of the System. If for some reason a user cannot include acknowledgement, the user must provide the fact in the Usage Report. When users make publications on research achievement obtained by the use of the System, they must state that their affiliations are RIKEN.

3. Usage Categories and Review

In this call, we receive proposals for user categories of special use, general use and quick use, each with different available computational resources (core time). Please make an application Advanced Center for Computing and Communication, RIKEN http://accc.riken.jp/index_e.html Revised on June 3, 2010 appropriate for your research thesis. Proposals are reviewed by the Review Committee.

(1) Special User account:

 Priority use of the CPU resources on weekends (60 hours elapsed time) will be given to researchers of approved projects. Time period is until fiscal year end (March 30).  Research proposals need to have sufficient necessity for the special use of the system.  The amount of CPU resources that can be reserved are given below. Users will select resources when they make applications.

Massively Parallel Cluster Between 8000 and 8120 cores for maximum of 4 weekends or 4000 cores for maximum of 8 weekends.

Multi-purpose PC Cluster 800 cores for maximum of 4 weekends or 400 cores for maximum of 8 weekends * Users are asked to use the GPU accelerators.

MDGRAPE-3 Cluster 256 cores for maximum of 4 weekends or 128 cores for maximum of 8 weekends * Users are asked to use MDGRAPE-3

 It is a users' duty to maintain the CPU utilization rate high. If the utilization rate is extremely low, we will request the reason for this.  Research proposals will be reviewed by the Review Committee that meets four times a year (March, June, September and December).

(2) General User account:

 Approved users will be permitted to use computation time of 1% or more of all computational resources.  Proposers must write usage period (3 months, 6 months or 12 months) and computation time in application forms.  The "all computational resources" is calculated by [the number of all cores × time that the system can provide in a fiscal year]. The GPU accelerators and the MDGRAPE-3 are not included into the "all computational resources"  "Computation time" is calculated by the total of [the number of used cores x elapsed time] of all jobs.  Research proposals will be reviewed by the Review Committee that meets four times a year (March, June, September and December).  If CPU time approved and allocated to you as a General User account runs short, you must make an application again for a General User and undergo the review by the Review Committee.  If you have consumed CPU time approved and allocated to you as a General User account, you cannot make an application for Quick User account with the same proposal.

(3) Exclusive User account:

 Approved users will be permitted to use a portion of the entire System exclusively.  Usage period is one fiscal year only (from April until next March)

(4) Quick User account:

 Approved users will be permitted to use computation time of less than 1% of the computational resources of the entire System. Advanced Center for Computing and Communication, RIKEN http://accc.riken.jp/index_e.html Revised on June 3, 2010

 Approved users will be permitted to use the system until a fiscal year end, for maximum one year.  When consumed computational resources have accumulated to 1%, the system will not accept jobs submitted by the user.  Research proposals will be reviewed by the Director of the Center at any time.  If this is your first time to use the system, we request you to make an application for a Quick User account and estimate the amount of CPU resources necessary to conduct your research.  After you find that you will need 1% or more of the total CPU resources, please make an application for a General User account and undergo the evaluation by the Review Committee.

4. Advance Review for Temporary Usage

If you cannot wait for the next Review Committee meeting, you can make a proposal with the specific reason for the request. The Director of the Center and the chairperson of the Review Committee will undergo a review and will decide whether or not to approve using the system until the next Review Committee meeting temporarily.

5. Application Method

Proposers must submit Account Request Forms and the data files (or copies) of publications. Please read the following explanation to decide what types of Account Request Form must be used.

 Form A must be filled in and submitted by all proposers (if a proposal is made by a group, all members of the group)  Form B must be submitted by proposers for Exclusive User account, Special User account and General User account.  Form C must be submitted by a project leader if a proposal is made by a group)

Application forms must be saved in Microsoft Word and sent to an e-mail address written below. Copies of publications must be saved in PDF and uploaded to a server which instructions will be informed after the receipt of application forms.

E-mail address: [email protected]

If you cannot submit publications electronically, send application forms by e-mail, and after that, send the copies of publications by internal pouch or postal mail.

RIKEN Wako main campus Information Science Bldg. 2/F The Advanced Center for Computing and Communication Attn: Reception desk

Researchers in the Yokohama Institute – please use the following address for internal pouch: RIKEN Yokohama Institute The Advanced Center for Computing and Communication Yokohama team Attn: HPC group

Our postal address: The Advanced Center for Computing and Communication RIKEN 2-1, Hirosawa, Wako Saitama 351-0198 Attn: HPC group Advanced Center for Computing and Communication, RIKEN http://accc.riken.jp/index_e.html Revised on June 3, 2010

(Please write on the envelope “RICC application form enclosed” in red)

6. Making Proposals by a Group

You can make proposals by a group for one research topic.

 Form A must be filled in all members of a group and be submitted by a project leader.  Form B must be filled in and submitted by a project leader.  Please describe the actual role each group member plays in the Form B. If you are applying for a Quick Use user account, it is not necessary to submit the Form B.  Form C (group member list) must be submitted together with the Form A and the Form B.  Accounts are issued individually, but consumed computation time is accumulated as a group.  Project leaders must inform us of any changes or additions in the member of their groups. Advanced Center for Computing and Communication, RIKEN http://accc.riken.jp/index_e.html Revised on June 3, 2010 RIKEN Supercomputer System Grant Request Form A [New account / Make changes]

Date: 27 / Sep / 2011

There are three request forms for the use of the RIKEN Supercomputer System. Please submit forms appropriate for a user category you are applying for.  Form A must be submitted by all proposers (all group members if a proposal is made by a group).  Form B must be submitted by proposers for Exclusive Use, Special Use and General Use user accounts.  Form C must be submitted by a project leader if the proposal will be carried out by two or more project members.

To request a new account: please circle "New account". To make changes in registration information, please circle "Make changes", write the changes in red ink and submit the form as soon as possible.

1. Proposer information (Please fill in all items)

Name: Fumie Costen E-mail address: [email protected] This e-mail address will be used for communication from the Center. Research topic: Computation of electromagnetic wave propagation in the human body Description of your research project (please write significance of your research project as well as necessity to use the system) and 5 keywords: Keywords: the finite difference time domain method, implicit method, large scale computation, biomedical engineering, healthcare

The neuro-stimulation is performed in various scenario and cases for a wide range of the illness treatment. One of the examples include the deep brain stimulation. Illness related to the brain such as Parkinson's disease, depression, Altzheimers' illness can be treated by the deep brain stimulation. However, the treatment is severely invasive. In our research group, we are trying to develop a system which can stimulate a part of the brain non-invasively. This may be realized by sending the electomagnetic signals from multiple locations on the skull and focusing the energy inside the brain. However, in order to develop a system, we need to understand the suitable frequency range, location of the excitations and so on. Since we can not try out such an experiment on humans, we have to perform the numerical simulations with digital human phantom. The human tissues are very small and in order to obtain an appropriate accuracy, we need to have a very fine mesh for modelling human. One of the most appropriate numerical methods for this type of problems is the finite difference time domain method. It is the time domain solution of the Maxwell's equations. However, given the small spatial sampling, the stability condition forces the time step to be unreasonably small. In the end, the explicit FDTD method can take quite long time. On the other hand, implicit schemes can take a larger temporal step. We are working on the improvement of the speed and accuracy of the FDTD method from many aspects such as boudary condition, usage of SSE instruction , MPI and GPU accelaration. Since we have to handle a human in a very small spatial resolution, we need a big computational facility for activities such as algorithm development , accuracy assessment and real numerical experiments for clinicians.

* To add a member to an already approved project, please write the number and the title of the project, and submit the Form C with a revised project member list.

Personal information you have registered is subject to RIKEN Privacy Policy and will not be used for any other purposes than confirmation of the eligibility and management of the system. Advanced Center for Computing and Communication, RIKEN http://accc.riken.jp/index_e.html Revised on June 3, 2010

2. Proposer information (Please fill in only the applicable item)

A. My primary employer is RIKEN Contact telephone number (ext.): RIKEN ID card number: Laboratory/ Position (please write official laboratory name) Laboratory: Position:

B. My primary employer is not RIKEN but I am affiliated with RIKEN Main affiliation name: Country: Address: Laboratory: Position: Contact telephone number: RIKEN ID card number: Laboratory/ Position in RIKEN Laboratory: Position: Supervisor in RIKEN *1 Name: Position: E-mail address:

If your affiliation applies neither A. nor B. (i.e. if you are not affiliated with RIKEN and if you wish to use the system for research activities based on collaborative research between RIKEN and your affiliation), the application must be made by your collaborative researcher in RIKEN in a position of team leader, chief scientist or equivalent. Please forward this document to the collaborative researcher and ask for submission.

Proposer's information Affiliation name: The University of Manchester Address: School of Electrical and Electronic Engineering Country: United Kingdom Position: Lecturer Contact telephone number: +44 161 304 4717 Name of the contract for cooperative research: Fumie Costen

Information on proposer's collaborative researcher in RIKEN (in a position of team leader, chief scientist or equivalent) *1 Name: Hideo Yokota Position: Team Leader E-mail address: [email protected]

*1 After receipt of account request, we will send a confirmation letter to a collaborative researcher in RIKEN and ask him to confirm the correctness of information in the application form, your compliance with the Usage Policy, etc. We do not issue an account until the receipt of the letter.

3. Usage category (Select one) [ ] Exclusive Use [ ] Create a new account [ ] Add a project member to an approved research topic Research topic:

[ ] Special Use [ ] Create a new account [ ] Add a project member to an approved research topic Research topic: [v ] General Use [ v ] Create a new account [ ] Add a project member to an approved research topic Research topic: [ ] Quick use [ ] Create a new account [ ] Add a project member to an approved research topic Research topic:

4. Account name Desired account name First choice: fumie Second choice: costen Third choice: fumiecosten

・ The number of characters must be 4-8. ・ You can use alphanumeric characters (A-Z, a-z and 0-9), hyphen (-), underline (_) and period (.) ・ The first character must be a letter (A-Z, a-z). ・ You must use at least one lower-case letter (a-z).

If you already have an account wish to participate in a different project, please write "your account name (already obtained)"

5. How and where do you pick up your password and a user guide? Being confidential information, we hand deliver passwords to proposers. Please select a place you wish to pick up your password and a user guide. If your working location is other than the Wako and the Yokohama Institutes, and if it is difficult for you to come to our offices, we will mail a password by registered mail. Please check the [Receive by mail] checkbox and provide your postal mail address.

[ ] Reception desk of the Center, Wako Institute [ ] Reception desk of the Center, Yokohama Institute [ v] Receive by mail Address:

Fumie Costen, School of Electrical and Electronic Engineering, Sackville Street Building, Sackville Street, University of Manchester, Manchester, M13 9PL, UK

------Contact: Advanced Center for Computing and Communication, RIKEN Tel: 048-467-9323 (ext. 3812), e-mail address: [email protected]

Personal information you have registered is subject to RIKEN Privacy Policy and will not be used for any other purposes than confirmation of the eligibility and management of the system. Advanced Center for Computing and Communication, RIKEN http://accc.riken.jp/index_e.html Revised on June 3, 2010 RIKEN Supercomputer System Grant Request Form B (Applicants for a Quick user category do not need to submit this form)

There are three request forms for the use of the RIKEN Supercomputer System. Please submit forms appropriate for a user category you are applying for.  Form A must be submitted by all proposers (all group members if a proposal is made by a group).  Form B must be submitted by proposers for Exclusive Use, Special Use and General Use user accounts.  Form C must be submitted by a project leader if the proposal will be carried out by two or more project members.

1 [Research description] (background, purpose, goal, computation method or justification to use the system)

Please describe in a comprehensible way. You can attach reference figures such as graphs or illustrations.

1.1. Research background

UltraWideBand(UWB) systems are a recent development and promise to revolutionise many areas such as communications, and sensor measurement systems. They benefit from low power, low cost wideband techniques that provide high resolution. The method generally involves generating a wideband pulse and transmitting this via an antenna. As the method is relatively new very little is known or understood about the propagation characteristics and issues such as multi-path and fading effects. In order to optimise and develop these systems for practical applications it is necessary to model and understand the propagation features. The potential for UWB systems is wide and includes wireless networks, Body-Area Networks and Smart ID Tags. One typical example is 3D biomedical imaging to view non-invasively inside the body for example to detect breast cancer without the high cost of magnetic resonance imaging. It is envisaged that UWB biomedical imaging will overcome limitations associated with current techniques such as ultra-sound and X-ray imaging. This project is going to focus on the optimisation of the electrodes' location and the waveforms for the non -invasive deep brain stimulation for the brain-related illness. The radio environment for such systems involves Frequency Dependent (FD) lossy media. One of the most promising full-wave Maxwell equation solvers for this scenario is the 3D implicit Finite Difference Time Domain(FDTD) method. This is able to simulate UWB propagation in FD lossy media with a shorter overall elapsed time than traditional FDTD methods. All commercial FDTD software are explicit with the theoretical upper limit for the temporal step governed by the stability condition. Since the biomedical engineering applications require more than 1000000000 FDTD iterations for typical problems, the usage of the explicit FDTD method is not appropriate for this project. The currently available major implicit scheme called the alternating direction implicit(ADI) -FDTD method is still slow because the upper limit of the temporal step is set by the practically acceptable numerical noise. Thus we investigate the more accurate implicit FDTD method.

1.2 Research purpose and achievement you aim for with requested computational resources (Refer to the notes 1, 2, 3 on the next page)

Research aim:

When medication for the brain-related illness such as Amnesia, Alzheimer,Epilepsy, Parkinson's disease, Depression, Schizophrenia is not effective, an alternative treatment is taken; the deep brain stimulation which involves the insertion of the electrode into the brain. This invasive treatment could be replaced with the non-invasive approach by applying the electrode around the head and focusing the energy inside the brain. This requires the precise location of electrodes, the excitation waveforms. Experimental trial and error to perform the optimisation costs and is time-consuming. This project develops an accurate and cost-effective software which accommodates the real MRI data of human head as the medium of the electromagnetic wave propagation and performs the optimisation.

1.3 Specific calculation method to conduct research (Refer to the notes 1, 2, 3 on the next page)

The explicit and implicit finite difference time domain method

1.4. Academic merit or impact that this research is expected to bring.

a) Development of a novel algorithm which is going to be used for the efficient calculation of the Maxwell equations in time domain b) Development of the parallelisation methodology of the locally one dimensional FDTD algorithm c) Understanding the wave propagation inside human brain from an invasive electrode.

1.5. If you have been a RICC user and wish to continue using the system, provide research situation and future schedule.

1.6. Justification to use RICC in an Exclusive Use or Special Use user category.

Note: 1. If this research thesis is related to research conducted at a laboratory or by a research group or is a part of them, describe their relation specifically. 2. If this research thesis is conducted by two or more researchers, describe work assignment for each member. 3. If your primary employer is not RIKEN, describe the actual role you will play in the collaborative research.

Personal information you have registered is subject to RIKEN Privacy Policy and will not be used for any other purposes than confirmation of the eligibility and management of the system. Advanced Center for Computing and Communication, RIKEN http://accc.riken.jp/index_e.html Revised on June 3, 2010 2. [Necessary CPU resources]

2.1. Period of time you will need to estimate 6 months (estimation must finish by March, the end of fiscal year)

2.2. Necessary computation time

Total Computation Time 17280 core x hours

Calculation method: Computation time is a total of [number of cores x elapsed time] for all jobs. For example, if you wish to run 200 jobs on 32 cores for 12 hours, your computation time will be 32 x 12 x 200 = 76,800 core x hours.

If you run multiple jobs using the different numbers of cores, write a total number.

When you have consumed 80% or more of your computation time, you will be notified so by e-mail.

Total computation time of RICC is 80,400,000 core x hours in a year. 1,000 core x hours is equivalent to the cost of US$ 110.

Please note that the shorter period of time you request the higher core utilization ratio becomes, and your request will be deemed as a large scale request by the Review Committee.

2.3. Justification for the computation time you requested in the item 2.2.

We are currently running 4 jobs per day(24 hours) in average and some jobs require 7 days. Since we shall be contnuously running jobs for 30*6 days, the total hour will be 4*24*30*6 = 17280. In the future when we develop our MPI jobs, we request the CPU time again.

Personal information you have registered is subject to RIKEN Privacy Policy and will not be used for any other purposes than confirmation of the eligibility and management of the system. Advanced Center for Computing and Communication, RIKEN http://accc.riken.jp/index_e.html Revised on June 3, 2010 3. [Reference Publications]

• Unconditionally stable finite difference time domain methods for frequency dependent media by Hasan Rouf, a PhD thesis at University of Manchester, Summer, 2010

Personal information you have registered is subject to RIKEN Privacy Policy and will not be used for any other purposes than confirmation of the eligibility and management of the system. Advanced Center for Computing and Communication, RIKEN http://accc.riken.jp/index_e.html Revised on June 3, 2010 RIKEN Super Computer System Application Form C

There are three request forms for the use of the RIKEN Supercomputer System. Please submit forms appropriate for a user category you are applying for.  Form A must be submitted by all proposers (all group members if a proposal is made by a group).  Form B must be submitted by proposers for Exclusive Use, Special Use and General Use user accounts.  Form C must be submitted by a project leader if the proposal will be carried out by two or more project members.  Add rows to the table if necessary.

Application date / Revision date (please circle one) ______

Project Name Submissions Manager ○ Form A & Form B Form A Form A Form A Form A Form A Form A Form A Form A Form A

Personal information you have registered is subject to RIKEN Privacy Policy and will not be used for any other purposes than confirmation of the eligibility and management of the system.