Using the IBM Opteron 1350 at OSC —Batch Processing October 19-20, 2010 Online Information • Technical information: – http://www.osc.edu/supercomputing/ • Hardware • Software • Environment • Training • Notices – http://www.osc.edu/supercomputing/computing/#batch • Contact information – [email protected] – 1-800-686-6472, 1-614-292-1800 2 Table of Contents • Batch Processing: Step by Step • Minimum Batch Files for Glenn Cluster • Advantages of Batch Processing • Useful Batch File Header Lines • Useful Batch Environment Variables • More PBS commands All files used in the examples can be retrieved from the Batch subdirectory of the svn repository: svn checkout http://svn.osc.edu/repos/softdevtools/trunk/Batch Batch 3 Interactive Processing • The way you are used to working on a workstation or laptop! • Enter a command, output returned to monitor. Based on output, enter a command, output returned to monitor, repeat. User is interacting in real-time with the computer. • Interactive use is easiest (and almost required) for tasks that involve user’s analysis of previous command’s output to determine the next command. • Common interactive tasks: file/directory searching, directory management (clean-up, reorganization, etc.), file editing, code debugging (in any manner), use of window-based software, use of performance tools (although most create a raw data file), and the list goes on … 4 Example Program Throughout this workshop the same tasks will be carried out in different ways. The tasks are: 1) Compiling a bug-free source code file 2) Running the executable produced 3) Examining the output 4) Removing unnecessary files 5) Changing from home directory to “work” directory 5 Example Serial Program svn checkout http://svn.osc.edu/repos/softdevtools/trunk/Batch Batch #include <stdio.h> This program ( ) #include <math.h> nest.c #include <stdlib.h> simply adds together int main(int argc, char* argv[]) { numbers whose values float x, rock = 0; int i, j, N, M; are created in a nested M = 8; N = 4; loop if (argc == 3) { N = atoi(argv[1]); // command arguments M = atoi(argv[2]); } else { // Exit if incorrect number of arguments printf("Usage : ./a.out int int\n"); exit(); } // Set Loop Counts for(i=1;i<=N;i=i+1) { for(j=1;j<=M;j=j+1) { x = log((float) ((10*i)+j)); // natural log rock = rock + x; // accumulate sum } } /* Output */ printf("For loop counts N=%d M=%d\n",N,M); printf("Sum=%f\n",rock); } 6 Example Parallel Program if (rank==0) { This MPI program ( ) for (i=0;i<N;++i) { search.c scanf("%d",&eleven[i]); divides an array into four arrays, } searches for a value, and reports MPI_Send(&eleven[1*(N/4)],N/4,MPI_INT,1,19,MPI_COMM_WORLD); MPI_Send(&eleven[2*(N/4)],N/4,MPI_INT,2,29,MPI_COMM_WORLD); when that value is found MPI_Send(&eleven[3*(N/4)],N/4,MPI_INT,3,39,MPI_COMM_WORLD); #include <stdio.h> for (i=0;i<N/4;++i) { #include <mpi.h> sub[i]=eleven[i]; #define N 16000 } int main(int argc, char *argv[]) { MPI_Send(&eleven[1*(N/4)],N/4,MPI_INT,1,19,MPI_COMM_WORLD); int rank,size; MPI_Send(&eleven[2*(N/4)],N/4,MPI_INT,2,29,MPI_COMM_WORLD); int i,j; MPI_Send(&eleven[3*(N/4)],N/4,MPI_INT,3,39,MPI_COMM_WORLD); int eleven[N],sub[N/4]; MPI_Status info; if (rank!=0) { MPI_Request ask,found; MPI_Recv(sub,N/4, MPI_INT, 0,MPI_ANY_TAG, int done=0; MPI_COMM_WORLD, &info); int index; } MPI_Init(&argc,&argv); MPI_Barrier(MPI_COMM_WORLD); MPI_Comm_rank(MPI_COMM_WORLD,&rank); MPI_Comm_size(MPI_COMM_WORLD,&size); MPI_Irecv(&index,1,MPI_INT,MPI_ANY_SOURCE,MPI_ANY_TAG, MPI_COMM_WORLD,&ask); printf("In node[%d], tmpdir is %s\n", rank, system("echo $TMPDIR")); [cont’d on next slide] 7 Example Parallel Program [cont’d from previous slide] i=0; MPI_Test(&ask,&done,&info); while (i<N/4 && !done) { if (sub[i]==11 ) { printf("P:%d 11 found at index=%d\n",rank,i); for (j=0; j<size; ++j) { MPI_Isend(&i,1,MPI_INT,j,55, MPI_COMM_WORLD,&found); } } ++i; MPI_Test(&ask,&done,&info); } MPI_Wait(&ask,&info); printf("P:%d I searched up to index %d\n", rank,(i-1)); MPI_Finalize(); } 8 Interactive Session This set of commands assumes you have downloaded nest.c and nest.pbs from the svn repository: opt-login1:$ cd Batch opt-login1:$ gcc nest.c -lm opt-login1:$ ./a.out 459 121 For loop counts N=459 M=121 Sum = 306505.406250 opt-login1:$ rm a.out 9 Batch Processing of SAME Tasks • Key fact: In the previous interactive session the user already knew before logging in what commands they were going to type. The code had been debugged, everything is ready for a “production” run. • NO REAL-TIME INTERACTION IS REQUIRED → Therefore, you can put the commands in a file (“batch file”) • A batch file is just a script (sequence of UNIX commands put into a file) that contains: 1) The EXACT SAME commands you typed on the keyboard during the interactive session. 2) Some lines at the beginning of the file that give the batch system software some parameters it needs to know. These opening lines are called the header of the batch file. 10 Structure of a Minimum Batch File #PBS -N nest Header #PBS -l walltime=00:05:00 #PBS -j oe • in /bin/ksh, this prints (echos) the executed commands in your output file #PBS –S /bin/ksh • in /bin/csh, this command is set echo set –x • environment variable which is automatically set when submitting a batch request • it is set to the directory path from where cd $PBS_O_WORKDIR the batch request was submitted gcc nest.c -lm • more on other environment variables later ./a.out 459 121 /bin/rm a.out Same Unix Commands this will remove the named file(s) immediately 11 Batch Processing Session -bash-3.2$ qsub nest.pbs 2289978.opt-batch.osc.edu -bash-3.2$ qstat -u yzhang opt-batch.osc.edu: Req'd Req'd Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time --------------- -------- -------- ---------- ------ --- --- ------ ----- - ----- 2289978.opt-bat yzhang serial nest -- 1 -- -- 00:05 Q -- -bash-3.2$ qstat -u yzhang 2289978.opt-bat yzhang serial nest -- 1 -- -- 00:05 Q -- -bash-3.2$ qstat -a | grep nest 2289978.opt-bat yzhang serial nest 402 1 -- -- 00:05 R -- -bash-3.2$ qstat -u yzhang 2289978.opt-bat yzhang serial nest 402 1 -- -- 00:05 R 00:01 12 Batch Processing Session Results: -bash-3.2$ ls -ltr total 108 -rw-r--r-- 1 yzhang G-3040 276 Oct 28 14:30 run-nest.job -rw-r--r-- 1 yzhang G-3040 295 Oct 28 14:30 run-nest3.job -rw-r--r-- 1 yzhang G-3040 140 Oct 28 14:30 nest.pbs -rw-r--r-- 1 yzhang G-3040 725 Oct 28 14:30 nest.c -rw------- 1 yzhang G-3040 139 Nov 1 22:17 nest.o2289978 -bash-3.2$ cat nest.o2289978 + cd /nfs/06/yzhang/workshops/softw09/Batch + gcc nest.c -lm + ./a.out 459 121 For loop counts N=459 M=121 Sum=306505.406250 + rm -f a.out 13 Description of Batch File Header Lines • Batch processing on OSC systems is implemented by the Portable Batch System. • Header lines begin with #PBS. The # symbol indicates that this line is just a comment from the shell's point of view. • #PBS -l walltime=00:01:00 – The character before the word walltime is a lower-case “el” not the numeric character “1”. The “el” stands for limit. – This header line tells PBS how much real time (the time on the clock on the wall) you expect the execution of the batch file commands to take. – Since the time format is hh:mm:ss, the limit on the wall clock time is 1 minute in this example – You can estimate wall clock time by running your program once and using external or internal timing commands (see later) 14 PBS at OSC • PBS is installed on Glenn cluster and Bale cluster • Glenn cluster has four partitions • Options may vary across systems/partitions , e.g., to take advantage of system-specific hardware • Review OSC web pages on the different computing systems/partitions. http://www.osc.edu/supercomputing/hardware/ 15 Description of Batch File Header Lines • #PBS –N nest – provides a name to your batch job (important!); here the name is nest – the name is used by PBS in several ways: • appears in status (qstat command) output • prefix for the output file name returned by PBS, e.g. nest.o2289978 – A PBS output log file can be thought of as a screen dump. The log file contains everything that would have appeared on your monitor if the commands had been run interactively. • #PBS –j oe – By default, PBS returns two log files: one for the standard output data stream, the other for the standard error data stream – This option joins these two into a single log file 16 Description of Batch File Header Lines • #PBS –S /bin/ksh – Sets the Linux shell which will interpret the shell commands in your batch script – In this example, the Korn shell is used • Can use any available shell for your command interpreter • The command cat /etc/shells shows the shells available on a system 17 Procedure to run a Batch job 1. Create a batch job-file (standard text file) 2. Use the qsub command to submit the job-file to PBS • When PBS begins executing batch file commands, the shell is invoked in your login directory (your home directory, $HOME). • Retain the job number identified in the return line! 3. Use the qstat command to check on your job status (tip: use qstat -u my_user_id to see the status of your jobs only) 4.