TORQUE® Administrator Guide version 3.0.2 TORQUE Admin Manual version 3.0.2 Legal Notices Preface Documentation Overview Introduction Glossary 1.0 Overview 1.1 Installation 1.2 Initialize/Configure TORQUE on the Server (pbs_server) 1.3 Advanced Configuration 1.4 Manual Setup of Initial Server Configuration 1.5 Server Node File Configuration 1.6 Testing Server Configuration 1.7 TORQUE on NUMA Systems 1.8 TORQUE Multi-MOM 2.0 Submitting and Managing Jobs 2.1 Job Submission 2.2 Monitoring Jobs 2.3 Canceling Jobs 2.4 Job Preemption 2.5 Keeping Completed Jobs 2.6 Job Checkpoint and Restart 2.7 Job Exit Status 2.8 Service Jobs 3.0 Managing Nodes 3.1 Adding Node 3.2 Configuring Node Properties 3.3 Changing Node State 3.4 Host Security 3.5 Linux Cpuset Support 3.6 Scheduling Cores 3.7 Scheduling GPUs 4.0 Setting Server Policies 4.1 Queue Configuration 4.2 Server High Availability 5.0 Interfacing with a Scheduler 5.1 Integrating Schedulers for TORQUE 6.0 Configuring Data Management 6.1 SCP/RCP Setup 6.2 NFS and Other Networked Filesystems 6.3 File Stage-In/Stage-Out 7.0 Interfacing with Message Passing 7.1 MPI (Message Passing Interface) Support 8.0 Managing Resources 8.1 Monitoring Resources 9.0 Accounting 9.1 Accounting Records 10.0 Logging 10.1 Job Logging 11.0 TroubleShooting 11.1 Troubleshooting 11.2 Compute Node Health Check 11.3 Debugging Appendices Appendix A: Commands Overview Client Commands momctl pbsdsh pbsnodes qalter qchkpt qdel qhold qmgr qrerun qrls qrun qsig qstat qsub qterm tracejob Server Commands pbs_mom pbs_server pbs_track Appendix B: Server Parameters Appendix C: MOM Configuration Appendix D: Error Codes and Diagnostics Appendix E: Considerations Before Upgrading Appendix F: Large Cluster Considerations Appendix G: Prologue and Epilogue Scripts Appendix H: Running Multiple TORQUE Servers and Moms on the Same Node Appendix I: Security Overview Appendix J: Submit Filter (aka qsub Wrapper) Appendix K: torque.cfg File Appendix L: TORQUE Quick Start Guide Changelog Legal Notices Copyright © 2011 Adaptive Computing Enterprises, Inc. All rights reserved. Distribution of this document for commercial purposes in either hard or soft copy form is strictly prohibited without prior written consent from Adaptive Computing Enterprises, Inc. Trademarks Adaptive Computing, Cluster Resources, Moab, Moab Workload Manager, Moab Cluster Manager, Moab Cluster Suite, Moab Grid Scheduler, Moab Grid Suite, Moab Access Portal, and other Adaptive Computing products are either registered trademarks or trademarks of Adaptive Computing Enterprises, Inc. The Adaptive Computing logo and the Cluster Resources logo are trademarks of Adaptive Computing Enterprises, Inc. All other company and product names may be trademarks of their respective companies. Acknowledgments TORQUE includes software developed by NASA Ames Research Center, Lawrence Livermore National Laboratory, and Veridian Information Solutions, Inc. Visit www.OpenPBS.org for OpenPBS software support, products and information. TORQUE is neither endorsed by nor affiliated with Altair Grid Solutions, Inc. TORQUE Administrator Guide Overview Advanced TORQUE Administration is a video tutorial of a session offered at Moab Con that offers further details on advanced TORQUE administration. This collection of documentation for TORQUE resource manager is intended as a reference for both users and system administrators. The 1.0 Overview section provides the details for installation and initialization, advanced configuration options, and (optional) qmgr options necessary to get the system up and running. System Testing is also covered. The 2.0 Submitting and Managing Jobs section covers different actions applicable to jobs. The first section, 2.1 Job Submission, details how to submit a job and request resources (nodes, software licenses, and so forth) and provides several examples. Other actions include monitoring, canceling, preemption, and keeping completed jobs. The 3.0 Managing Nodes section covers administrator tasks relating to nodes, which includes the following: adding nodes, changing node properties, and identifying state. Also an explanation of how to configure restricted user access to nodes is covered in section 3.4 Host Security. The 4.0 Setting Server Policies section details server side configurations of queue and high availability. The 5.0 Interfacing with a Scheduler section offers information about using the native scheduler versus an advanced scheduler. The 6.0 Configuring Data Management section deals with issues of data management. For non-network file systems, the SCP/RCP Setup section details setting up SSH keys and nodes to automate transferring data. The NFS and Other Networked File Systems section covers configuration for these file systems. This chapter also addresses the use of File Stage-In/Stage-Out using the stagein and stageout directives of the qsub command. The 7.0 Interfacing with Message Passing section offers details supporting MPI (Message Passing Interface). The 8.0 Managing Resources section covers configuration, utilization, and states of resources. The 9.0 Accounting section explains how jobs are tracked by TORQUE for accounting purposes. The 10.0 Troubleshooting section is a troubleshooting guide that offers help with general problems; it includes an FAQ (Frequently Asked Questions) list and instructions for how to set up and use compute node checks and how to debug TORQUE. The numerous appendices provide tables of commands, parameters, configuration options, error codes, the Quick Start Guide, and so forth. A. Commands Overview B. Server Parameters C. MOM Configuration D. Error Codes and Diagnostics E. Considerations Before Upgrading F. Large Cluster Considerations G. Prologue and Epilogue Scripts H. Running Multiple TORQUE Servers and Moms on the Same Node I. Security Overview J. Submit Filter (aka qsub Wrapper) K. torque.cfg File Introduction What is a Resource Manager? While TORQUE has a built-in scheduler, pbs_sched, it is typically used solely as a resource manager with a scheduler making requests to it. Resources managers provide the low-level functionality to start, hold, cancel, and monitor jobs. Without these capabilities, a scheduler alone can not control jobs. What are Batch Systems? While TORQUE is flexible enough to handle scheduling a conference room, it is primarily used in batch systems. Batch systems are a collection of computers and other resources (networks, storage systems, license servers, and so forth) that operate under the notion that the whole is greater than the sum of the parts. Some batch systems consist of just a handful of machines running single-processor jobs, minimally managed by the users themselves. Other systems have thousands and thousands of machines executing users' jobs simultaneously while tracking software licenses and access to hardware equipment and storage systems. Pooling resources in a batch system typically reduces technical administration of resources while offering a uniform view to users. Once configured properly, batch systems abstract away many of the details involved with running and managing jobs, allowing higher resource utilization. For example, users typically only need to specify the minimal constraints of a job and do not need to know the individual machine names of each host on which they are running. With this uniform abstracted view, batch systems can execute thousands and thousands of jobs simultaneously. Batch systems are comprised of four different components: (1) Master Node, (2) Submit/Interactive Nodes, (3) Compute Nodes, and (4) Resources. Master Node - A batch system will have a master node where pbs_server runs. Depending on the needs of the systems, a master node may be dedicated to this task, or it may fulfill the roles of other components as well. Submit/Interactive Nodes - Submit or interactive nodes provide an entry point to the system for users to manage their workload. For these nodes, users are able to submit and track their jobs. Additionally, some sites have one or more nodes reserved for interactive use, such as testing and troubleshooting environment problems. These nodes have client commands (such as qsub and qhold). Compute Nodes - Compute nodes are the workhorses of the system. Their role is to execute submitted jobs. On each compute node, pbs_mom runs to start, kill, and manage submitted jobs. It communicates with pbs_server on the master node. Depending on the needs of the systems, a compute node may double as the master node (or more). Resources - Some systems are organized for the express purpose of managing a collection of resources beyond compute nodes. Resources can include high-speed networks, storage systems, license managers, and so forth. Availability of these resources is limited and needs to be managed intelligently to promote fairness and increased utilization. Basic Job Flow The life cycle of a job can be divided into four stages: (1) creation, (2) submission, (3) execution, and (4) finalization. Creation - Typically, a submit script is written to hold all of the parameters of a job. These parameters could include how long a job should run (walltime), what resources are necessary to run, and what to execute. The following is an example submit file: #PBS -N localBlast #PBS -S /bin/sh #PBS -l nodes=1:ppn=2,walltime=240:00:00 #PBS -M [email protected] #PBS -m ea source ~/.bashrc cd $HOME/work/dir sh myBlast.sh -i -v This submit script specifies the name of the job (localBlast), what environment to use (/bin/sh), that it needs both processors on a single node (nodes=1:ppn=2), that it will run for at most 10 days, and that TORQUE should email [email protected] when the job exits or aborts. Additionally, the user specifies where and what to execute. Submission - A job is submitted with the qsub command. Once submitted, the policies set by the administration and technical staff of the site dictate the priority of the job and therefore, when it will start executing. Execution - Jobs often spend most of their lifecycle executing. While a job is running, its status can be queried with qstat.