Requesting Resources on an HPC Facility
(Using the Sun Grid Engine Job Scheduler)
Deniz Savas
dsavas.staff.shef.ac.uk/teaching
June 2017 Outline
1. Using the Job Scheduler 2. Interactive Jobs 3. Batch Jobs 4. Task arrays 5. Running Parallel Jobs 6. GPUs and remote Visualisation 7. Beyond Iceberg Accessing the N8 tier 2 facility Running Jobs A note on interactive jobs
• Software that requires intensive computing should be run on the worker nodes and not the head node. • You should run compute intensive interactive jobs on the worker nodes by using the qsh or qrsh command. • Maximum ( and also default) time limit for interactive jobs is 8 hours. Sun Grid Engine
• Two iceberg or sharc headnodes are gateways to the cluster of worker nodes. • Headnodes’ main purpose is to allow access to the worker nodes for the logged in users. • All CPU intensive computations must be performed on the worker nodes. This is achieved by using one of the following two commands on the headnode.
– qsh or qrsh : To start an interactive session on a worker node. – qsub : To submit a batch job to the cluster
• Once you log into iceberg, you are recommended to start working interactively on a worker-node by simply typing qsh and working in the new shell window that is opened. The next set of slides assume that you are already working on one of the worker nodes (qsh session). Practice Session 1 Running Applications on Iceberg (Problem 1)
• Case Studies – Analysis of Patient Inflammation Data • Running an R application how to submit jobs and run R interactively • List available and loaded modules load the module for the R package • Start the R Application and plot the inflammation data Other Methods of submitting Batch Jobs on the Sheffield HPC clusters
Iceberg has a number of home grown commands for submitting jobs for some of the most popular applications and packages to the batch system. These commands create suitable scripts to submit the users’ job to the cluster automatically
These are; runfluent , runansys , runmatlab , runabaqus
To get information on how to use these command, simply issue the command name on a worker node without any parameters. Exercise 1: Submit a job via qsub
• Create a script file (named example.sh) by using a text editor such as gedit ,vi or emacs and inputing the following lines: #!/bin/sh # echo “This code is running on” /bin/hostname /bin/date
• Now Submit this script to SGE using the qsub command: qsub example.sh Tutorials
On iceberg copy the contents of the tutorial directory to your user area into a directory named sge: cp –r /usr/local/courses/sge sge cd sge In this directory the file readme.txt contains all the instructions necessary to perform the exercises. Managing Your Jobs Sun Grid Engine Overview
SGE is the resource management system, job scheduling and batch control system. (Others available such as PBS, Torque/Maui, Platform LSF ) • Starts up interactive jobs on available workers • Schedules all batch orientated ‘i.e. non-interactive’ jobs • Attempts to create a fair-share environment • Optimizes resource utilization SGE SGE SGE SGE SGE worker worker worker worker worker node node node node node B Slot C Slot 2 C Slot 1 ASlot 1 B Slot 1 C Slot A 2 ASlot 1 B Slot 1 C Slot 2 C Slot 3 C Slot 1 B Slot 2 B Slot 3 B Slot Slot 1 1 1
Queue-A Queue-B Queue-C
SGE MASTER node
.Queues .Policies .Priorities
JOB Y JOB Z .Share/Tickets JOB X
JOB O .Resources JOB N JOB U .Users/Projects
Scheduling ‘qsub’ batch jobs on the cluster Working with SGE as a user
Although the SGE system contains many commands and utilities most of them are for the administration of the scheduling system only. The following list of SGE commands will be sufficient for most users. – qsub : Submits a batch job – qsh or qrsh : Starts an interactive session – qstat : Queries the progress of the jobs – qdel : Removes unwanted jobs. Running interactive jobs on the cluster
1. User asks to run an interactive job (qsh, qrsh ) 2. SGE checks to see if there are resources available to start the job immediately (i.e a free worker ) – If so, the interactive session is started under the control/monitoring of SGE on that worker. – If resources are not available the request is simply rejected and the user notified. This is because by its very nature users can not wait for an interactive session to start. 3. User terminates the job by typing exit or logout or the job is terminated when the queue limits are reached (i.e. currently after 8 hours of wall-clock time usage). Demonstration 1
Running Jobs batch job example Using the R package to analyse patient data
qsub example:
qsub –l h_rt=10:00:00 –o myoutputfile –j y myjob
OR alternatively … the first few lines of the submit script myjob contains -
$!/bin/bash $# -l h_rt=10:00:00 $# -o myoutputfile $# -j y
and you simply type; qsub myjob Summary table of useful SGE commands
Command(s) Description User/System qsub, qresub,qmon Submit batch jobs USER qsh,qrsh Submit Interactive USER Jobs qstat , qhost, qdel, Status of queues and USER qmon jobs in queues , list of execute nodes, remove jobs from queues qacct, qmon, qalter, Monitor/manage SYSTEM ADMIN qdel, qmod accounts, queues, jobs etc Using the qsub command to submit batch jobs
In its simplest form any script file can be submitted to the SGE batch queue by simply typing qsub scriptfile . In this way the scriptfile is queued to be executed by the SGE under default conditions and using default amount of resources. Such use is not always desirable as the default conditions provided may not be appropriate for that job . Also, providing a good estimate of the amount of resources needed helps SGE to schedule the tasks more efficiently. There are two alternative mechanisms for specifying the environment & resources;
1) Via parameters to the qsub command 2) Via special SGE comments (#$ ) in the script file that is submitted.
The meaning of the parameters are the same for both methods and they control such things as;
- cpu time required - number of processors needed ( for multi-processor jobs), - output file names, - notification of job activity. Method 1 Using qsub command-line parameters
Format: qsub [qsub_params] script_file [-- script_arguments]
Examples: qsub myjob qsub –cwd $HOME/myfolder1 qsub –l h_rt=00:05:00 myjob -- test1 -large
Note that the last example passes parameters to the script file following the -- token. Method 2 special comments in script files
A script file is a file containing a set of Unix commands written in a scripting language ‘usually Bourne/Bash or C-Shell’. When the job runs these script files are executed as if their contents were typed at the keyboard. In a script file any line beginning with # will normally be treated as a comment line and ignored. However the SGE treats the comment lines in the submitted script, which start with the special sequence #$ ,in a special way. SGE expects to find declarations of the qsub options in these comment lines. At the time of job submission SGE determines the job resources from these comment lines. If there are any conflicts between the actual qsub command-line parameters and the special comment (#$) sge options the command line parameters always override the #$ sge options specified in the script. An example script containing SGE options
#!/bin/sh #A simple job script for sun grid engine. # #$ -l h_rt=01:00:00 #$ -m be #$ -M [email protected] benchtest < inputfile > myresults More examples of #$ options in a scriptfile
#!/bin/csh # Force the shell to be the C-shell # On iceberg the default shell is the bash-shell #$ -S /bin/csh
# Request 8 GBytes of virtual memory #$ -l mem=8G
# Specify myresults as the output file #$ -o myresults
# Compile the program pgf90 test.for –o mytestprog
# Run the program and read the data that program # would have read from the keyboard from file mydata mytestprog < mydata Running Jobs qsub and qsh options
-l h_rt=hh:mm:ss The wall clock time. This parameter must be specified, failure to include this parameter will result in the error message: “Error: no suitable queues”. Current default is 8 hours . -l arch=intel* Force SGE to select either Intel or AMD architecture nodes. No -l arch=amd* need to use this parameter unless the code has processor dependency. -l mem=memory sets the virtual-memory limit e.g. –l mem=10G (for parallel jobs this is per processor and not total). Current default if not specified is 6 GB . -l rmem=memory Sets the limit of real-memory required Current default is 2 GB. Note: rmem parameter must always be less than mem. -help Prints a list of options
-pe ompigige np Specifies the parallel environment to be used. np is the -pe openmpi-ib np number of processors required for the parallel job. -pe openmp np Running Jobs qsub and qsh options ( continued)
-N jobname By default a job’s name is constructed from the job-script-file- name and the job-id that is allocated to the job by SGE. This options defines the jobname. Make sure it is unique because the job output files are constructed from the jobname. -o output_file Output is directed to a named file. Make sure not to overwrite important files by accident. -j y Join the standard output and standard error output streams recommended -m [bea] Sends emails about the progress of the job to the specified email address. If used, both –m and –M must be specified. Select any or -M email-address all of the b,e and a to imply emailing when the job begins, ends or aborts. -P project_name Runs a job using the specified projects allocation of resources. -S shell Use the specified shell to interpret the script rather than the default bash shell. Use with care. A better option is to specify the shell in the first line of the job script. E.g. #!/bin/bash -V Export all environment variables currently in effect to the job. Qsub options ( notifications and testing related )
-M email_address Email address for job notifications. Example: -M [email protected]
-m b e a s Send email(s) when the job begins , ends, aborted or suspended Example: –m be
-now Start running the job now or if can’t be run exit with an error code.
-verify do not submit the job but check and report on submission. Qsub options (output files and job names related )
When a job is submitted it is given a unique job_number . Also by default the name of the script file is used as the jobname . When the job starts running the standard output and error outputs are sent to files named jobname.ojob_number and jobname.ejob_number respectively. For example: myscript.o45612 myscript.e45612 The –o , -e , -j y and –N name parameters modify this behaviour. Example: relating to job output files
Passed to qsub as arguments during submission:
qsub –N walldesign –j y walljob
OR insert in the submit script file walljob:
#$!/bin/bash #$ -N walldesign #$ -j y /home/data/my_app and submit the job qsub walljob Using either of these methods, when the job runs the both normal and error output will be contained in a file named walldesign.onnnnn where nnnnn is the unique job number SGE designated to your job at the time of submission. More on starting interactive iobs qsh and qrsh commands
• qsh : starts an Xterm session on a worker node. Use this command if you have XWindows capabilities.
• qrsh : starts a remote command shell and optionally executes a shell- scripts on a worker node. If you do not have Xwindows capability, i.e. you are not using “Exceed, Xming Cygwin ” so on, this is the way for starting interactive jobs on iceberg. It is suitable when you log in using putty or ssh in line-mode.
In Sheffield all interactive jobs are put into the short queues that limit the clock time to 8 hours of wall clock time. BEWARE: As soon as the time limit is exceeded the job will terminate without any warning. More on qrsh command
qrsh [parameters]
• If no parameters are given it behaves exactly like qsh. This is the normal method of using qrsh.
• If there are parameters a remote shell is started up on one of the workers and the parameters are passed to shell for execution. For example, if a script file name is presented as a parameter, commands in the script file are executed and the job terminates when the end of the script file is reached. Example : qrsh myscript More on qsh command
• qsh –display display_specifier
qsh starts up an X-terminal within which the interactive job is started. It is possible to pass any Xterm parameters via the -- construct. Example : qsh -- –title myjob1 Type man xterm for a list of parameters.
• Qsh : this is a home produced variation of the qsh command. It passes suitable –display parameters to qsh to produce a more pleasant looking command window. Monitoring your jobs
A submitted job will either be; 1. still waiting in the queue, 2. be executing, 3. finished execution and left the SGE scheduling system. In order to monitor the progress of your job while in states (1) and (2) use the qstat or Qstat commands that will inform you if the job is still waiting or started executing. The command qstat gives info about all the jobs but Qstat gives info about your jobs alone. While executing (state 2) ; use qstat –j job_number to monitor the jobs status including time and memory consumptions. Contains too much information ! Better still use qstat –j job_number | grep mem that will give time and memory consumed information. Also use tail –f job_output_filename to see the latest output from the job Finished executing ( state 3) ; qacct is the only command that may be able to tell you about the past jobs by referring to a data-base of past usage. Output file names will contain the job number so; qacct -j job_number should give some information. Monitoring your job
• If you are interested in only your job use Qstat
job-ID prior name user state submit/start at queue slots ja-task-ID ------3067843 0.50859 INTERACTIV cs1ds r 10/21/2010 09:03:05 [email protected] 1 3076264 0.50500 INTERACTIV cs1ds r 10/21/2010 16:37:37 [email protected] 1
• If you want to see all the jobs use qstat job-ID prior name user state submit/start at queue slots ja-task-ID ------3064625 0.57695 grd80x120x cpp06kw r 10/15/2010 02:35:43 [email protected]. 1 3064818 0.56896 M11_NiCl.3 chp09bwh r 10/15/2010 15:44:44 [email protected] 4 3065270 0.56657 pythonjob co1afh r 10/16/2010 13:34:33 [email protected]. 1 3065336 0.56025 parallelNe elp05ws r 10/16/2010 13:34:33 [email protected]. 1 3065340 0.56025 parallelNe elp05ws r 10/16/2010 15:31:51 [email protected]. 1 3065558 0.55060 coaxialjet zzp09hw r 10/17/2010 14:03:53 [email protected] 17 3065934 0.54558 periodichi mep09ww r 10/18/2010 08:39:06 [email protected] 8 3066207 0.53523 gav1 me1ccl r 10/18/2010 20:00:16 [email protected]. 1 3066213 0.53510 ga2 me1ccl r 10/18/2010 20:00:26 [email protected]. 1 3066224 0.53645 DDNaca0 mep09ww r 10/18/2010 23:41:56 [email protected] 12 3066226 0.53493 ga3 me1ccl r 10/18/2010 20:00:46 [email protected]. 1 3066231 0.53491 ga4 me1ccl r 10/18/2010 20:00:46 [email protected]. 1 3066415 0.53078 job elp07dc r 10/19/2010 09:25:24 [email protected]. 32 3066896 0.52323 fluent12jo fc1jz qw 10/19/2010 05:32:01 3067083 0.52222 Oct_ATLAS php09ajf qw 10/19/2010 17:41:01 qstat command
• qstat command will list all the jobs in the system that are either waiting to be run or running. This can be a very long list ! • qstat –f full listing ( even longer) • qstat –u username or Qstat ( recommend this ! ) • qstat –f –u username ( detailed information ) Status of the job is indicated by letters in qstat listings as: qw waiting t transfering r running s,S suspended R restarted T treshold Deleting Jobs with the qdel command
qdel command will remove from the queue the specified jobs that are waiting to be run or abort jobs that are already running. • Individual Job qdel 15112
• List of Jobs qdel 15154 15923 15012
• All Jobs running or queueing under a given username qdel –u
– SGE cannot find the binary file specified in the job script – One of the Linux environment resource limits is exceeded (see command ulimit –a ) – Required input files are missing from the startup directory – You have exceeded your quota and job fails when trying to write to a file ( use quota command to check usage ) – Environment variable is not set (LM_LICENSE_FILE etc) – Hardware failure Job Arrays
By using a single qsub command, it is possible to submit a series of jobs that use the same job template. These jobs are described as array jobs. For example: qsub myanalysis –t 1-10 will submit the script named myanalysis as 10 separate jobs. Of course it is pointless to run the same job 10 times. The only justification for doing so is that all these jobs are doing different tasks. This is where a special environment variable named SGE_TASK_ID becomes essential. In the above example in each job the variable SGE_TASK_ID will contain a unique number between 1 and 10 to differentiate these jobs from each other. Thus we can use this variable’s value to control each job in a different manner. Please note that there is no guarantee about the order of execution of these tasks. I.e. there is no guarantee that task number m will start before task number n , where m #$ -S /bin/tcsh #$ -l h_cpu=01:00:00 #$ -t 2-16:2 #$ -cwd myprog > results.$SGE_TASK_ID < mydata.$SGE_TASK_ID This will run 8 jobs. The jobs are considered to be independent of each other and hence may run in parallel depending on the availability of resources. Note that tasks will be numbered 2, 4 ,6 ,8 … (steps of 2) For example job 8 will read its data from file mydata.8 and write its output to file results.8 It is possible to make these jobs dependent on each other so as to impose an order of execution by means of the –hold_jid parameter. Practice Session: Submitting A Task Array To Iceberg (Problem 4) • Case Study – Fish population simulation • Submitting jobs to Sun Grid Engine • Instructions are in the readme file in the sge folder of the course examples – From an interactive session • Run the SGE task array example – Run test4, test5 An example OpenMP job script OpenMP programming takes advantage of the multiple CPU’s that reside in a single computer to distribute work amongst CPUs that share the same memory. Currently we have maximum of 8 CPU’s per computer and therefore only upto 8 processors can be requested for an iceberg openmp job. After the next upgrade this figure will increase to minimum 24. #$ -pe openmp 4 #$ -l h_rt=01:30:00 OMP_NUM_THREADS=4 ./myprog An example MPI job script MPI programs are harder to code but can take advantage of interconnected multiple computers by passing messages between them ( MPI= Message Passing Interface ) 23 workers on the iceberg pool are connected together with fast Infiniband communications cabling to provide upto 10 Gbits/sec data transfer rate between them. The rest of the workers can communicate with each other via the normal 1 Gbits/sec ethernet cables. #$ -pe mvapich2-ib 4 # limit run to 1 hours actual clock time #$ -l h_rt=1:00:00 mpirun_rsh -rsh -np $NSLOTS -hostfile $TMPDIR/machines ./executable Managing Jobs : Running cpu-parallel jobs • More many processor tasks – Sharing memory – Distributed Memory • Parallel environment needed for a job can be specified by the: -pe – openmpi-ib : OpenMPI library-Infiniband. These are MPI jobs running on multiple hosts using the Infiniband Connection ( 32GBits/sec ) – mvapich2-ib : Mvapich-library-Infiniband. As above but using the MVAPICH MPI library. • Compilers that support MPI. – PGI , Intel, GNU Running GPU parallel jobs GPU parallel processing is supported on 8 Nvidia Tesla Fermi M2070s GPU units attached to iceberg. • In order to use the GPU hardware you will need to join the GPU project by emailing [email protected] • You can then submit jobs that use the GPU facilities by using the following three parameters to the qsub command; -P gpu -l arch=intel* -l gpu=nn where 1<= nn <= 8 is the number of gpu-modules to be used by the job. P stands for project that you belong to. See next slide. Demonstration 3 Running a parallel job • Test 6 provides an opportunity to practice submitting parallel jobs to the scheduler. • To run testmpi6, compile the mpi example – Load the openmpi compiler module – module load mpi/intel/openmpi/1.8.3 • compile the diffuse program – mpicc diffuse.c -o diffuse – qsub testmpi6 – Use qstat to monitor the job examine the output The progress of your batch job • The user submits a batch job as described above eg. qsub myscript_file • The job is placed in the queue and given a unique job number • Once you prepared your job script you can test it by simply running it on its own for a very small or trivial problem. For example- if your script is called analyse.sh you simply type ./analyse.sh This will immediately expose any errors in your script. Note: that the qsub parameters which are defined using the #$ sequence will be treated as comments during this run. • Q: Should I define the qsub parameters in the script file or as parameters at the time of issuing qsub ? A: The choice is yours, I prefer to define any parameter, which is not likely to alter between runs, within the script file to save myself having to remember it at each submission. SGE related Environment Variables Apart from the specific environment variables passed via the –v or –V options, during the execution of a batch job the following environment variables are also available to help build unique or customized filenames messages etc. • $HOME : Your own login directory • $USER : your iceberg username • $JOB_NAME : Name of the job • $HOSTNAME : Name of the cluster node that is being used • $SGE_TASK_ID : Task number (important for task arrays) • $NSLOTS : Number of processors used ( important for parallel ‘openmp or mpi’ jobs ) Submitting Batch Jobs via the qmon command If you are using an X terminal ( such as provided by Exceed ) then a GUI interface named qmon can also be used to make job submission easier. This command also allows an easier way of setting the job parameters. Job submission panel of QMON Click on Job Submission Icon Click to browse for the job script test2 Job queues • Unlike the traditional batch queue systems, users do not need to select the queue they are submitting to. Instead SGE uses the resource needs as specified by the user to determine the best queue for the job. • In Sheffield and Leeds the underlying queues are setup according to memory size and cpu time requirements and also numbers of multiple cpu’s needed (for mpi & openmp jobs ) • qstat –F displays full queue information, Also qmon (Task- Queue_Control) will allow information to be distilled about the queue limits. Job queue configuration Normally you will not need to know the details of each queue, as the Grid Engine will make the decisions for you in selecting a suitable queue for your job. If you feel the need to find out how the job queues are configured, perhaps to aid you in specifying the appropriate resources, you may do so by using the qconf system administrator command. • qconf –sql will give a list of all the queues • qconf –sq queue_name will list details of a specific queue’s configuration Monitoring the progress of your jobs • The commands qstat and the XWindows based qmon can be used to check on the progress of your jobs through the system. • We recommend that you use the qmon command if your terminal has X capability as this makes it easier to view your jobs progress and also cancel or abort it, if it becomes necessary to do so. Checking the progress of jobs with QMON Click on Job Control Icon Click on Running Jobs tab Managing Jobs monitoring and controlling your jobs http://www.sheffield.ac.uk/cics/research/hpc/using/runbatch/sge • There are a number of commands for querying and modifying the status of a job running or waiting to run. These are; – qstat or Qstat (query job status) • qstat –u username – qdel (delete a job) • qdel jobid – qmon ( a GUI interface for SGE ) Practice Session: Submitting Jobs To Iceberg (Problem 2 & 3) • Patient Inflammation Study run the R example as a batch job • Case Study – Fish population simulation • Submitting jobs to Sun Grid Engine • Instructions are in the readme file in the sge folder of the course examples – From an interactive session • Load the compiler module • Compile the fish program • Run test1, test2 and test3 Managing Jobs: Reasons for job failures http://www.shef.ac.uk/cics/research/hpc/using/requirements – SGE cannot find the binary file specified in the job script – You ran out of file storage. It is possible to exceed your filestore allocation limits during a job that is producing large output files. Use the quota command to check this. – Required input files are missing from the startup directory – Environment variable is not set correctly (LM_LICENSE_FILE etc) – Hardware failure (eg communication equipment failure for mpi jobs) Finding out the memory requirements of a job • Virtual Memory Limits: – Default virtual memory limits for each job is 6 GBytes – Jobs will be killed if virtual memory used by the job exceeds the amount requested via the –l mem= parameter. • Real Memory Limits: – Default real memory allocation is 2 GBytes – Real memory resource can be requested by using –l rmem= – Jobs exceeding the real memory allocation will not be deleted but will run with reduced efficiency and the user will be emailed about the memory deficiency. – When you get warnings of that kind, increase the real memory allocation for your job by using the –l rmem= parameter. – rmem must always be less than mem Determining the virtual memory requirements for a job; – qstat –f –j jobid | grep mem – The reported figures will indicate - the currently used memory ( vmem ) - Maximum memory needed since startup ( maxvmem) - cumulative memory_usage*seconds ( mem ) – When you run the job next you need to use the reported value of vmem to specify the memory requirement Remote Visualisation • See – Specialist High Speed Visualization Access to iceberg – http://www.sheffield.ac.uk/cics/research/hpc/using/access/intro • Undertake visualisation using thin clients accessing remote high quality visualisation hardware • Remote visualisation removes the need to transfer data and allows researchers to visualise data sets on remote visualisation servers attached to the high performance computer and its storage facility VirtualGL • VirtualGL is an open source package which gives any UNIX or Linux remote display software the ability to run 3D applications with full hardware accelerations. • VirtualGL can also be used in conjunction with remote display software such as VNC to provide 3D hardware accelerated rendering for OpenGL applications. • VirtualGL is very useful in providing remote display to thin clients which lack the 3D hardware acceleration. Client Access to Visualisation Cluster Iceberg – Campus Compute Cloud VirtualGL Server (NVIDIA GPU) VirtualGL Client Remote Visualisation Using SGD • Star a browser and goto – https://myapps.shef.ac.uk – login to Sun Global Desktop • Under Iceberg Applications start the Remote visualisation session • This opens a shell with instructions to either – Open a browser and enter the address • http://iceberg.shef.ac.uk:XXXX – Start Tiger VNCViewer on your desktop • Use the address iceberg.shef.ac.uk:XXXX • XXXX is a port address provided on the iceberg terminal • When requested use your usual iceberg user credentials Remote Desktop Through VNC Remote Visualisation Using Tiger VNC and the Putty SHH Client • Login in to iceberg using putty • At the prompt type qsh-vis • This opens a shell with instructions to either – Open a browser and enter the address • http://iceberg.shef.ac.uk:XXXX – Start Tiger VNCViewer on your desktop • Use the address iceberg.shef.ac.uk:XXXX • XXXX is a port address provided on the iceberg terminal • When requested use your usual iceberg user credentials Beyond Iceberg http://www.sheffield.ac.uk/cics/research/hpc/iceberg/costs • Iceberg OK for many compute problems • Purchasing dedicated resource • N8 tier 2 facility for more demanding compute problems • Hector/Archer Larger facility for grand challenge problems (pier review process to access) High Performance Computing Tiers • Tier 1 computing – Hector, Archer • Tier 2 Computing – Polaris • Tier 3 Computing – Iceberg Purchasing Resource http://www.sheffield.ac.uk/cics/research/hpc/iceberg/costs • Buying nodes using framework – Research Groups purchase HPC equipment against their research grant this hardware is integrated with Iceberg cluster • Buying slice of time – Research groups can purchase servers for a length of time specified by the research group (cost is 1.7p/core per hour) • Servers are reserved for dedicated usage by the research group using a provided project name • When reserved nodes are idle they become available to the general short queues. They are quickly released for use by the research group when required. • For information e-mail [email protected] The N8 Tier 2 Facility: Polaris http://www.shef.ac.uk/cics/research/hpc/polaris • Note N8 is for users whose research problems require greater resource than that available through Iceberg • Registration is through Projects – Authourisation by a supervisor or project leader to register project with the N8 – Users obtain a project code from supervisor or project leader – Complete online form provide an outline of work explaining why N8 resources are required Polaris: Specifications 5312 Intel Sandy Bridge cores Co-located with 4500-core Leeds HPC Purchased through Esteem framework agreement: SGI hardware #291 in June 2012 Top500 National HPC Services • Archer • UK National Supercomputing Service • Hardware – CRAY XC30 • 2632 Standard nodes • Each node contains two Intel E5-2697 v2 12-core processors • Therefore 2632*2*12 63168 cores. • 64 GB of memory per node • 376 high memory nodes with128GB memory • Nodes connected to each other via ARIES low latency interconnect • Research Data File System – 7.8PB disk • http://www.archer.ac.uk/ • EPCC • HPCC Facilities • http://www.epcc.ed.ac.uk/facilities/national-facilities • Training and expertise in parallel computing Sheffield University Web References • Interactive Jobs – http://www.sheffield.ac.uk/cics/research/hpc/using/interactive • Batch Jobs – http://www.sheffield.ac.uk/cics/research/hpc/using/runbatch Links for Software Downloads • Putty http://www.chiark.greenend.org.uk/~sgtatham/putty/ • WinSCP http://winscp.net/eng/download.php • TigerVNC http://sourceforge.net/projects/tigervnc/ http://sourceforge.net/apps/mediawiki/tigervnc/index.php?title= Main_Page