What is TORQUE?

On this page:

• Introduction
• Policy
• Queues
• Jobs:
  • Scripts
  • Submission
  • Monitoring
  • Deleting
• Additional documentation

Introduction

TORQUE, also known by its historical name, Portable Batch System (PBS), is the resource manager on the Quarry system at Indiana University. Tools for job submission and management are available in /usr/local/bin; most tools have associated man pages. For detailed online information, such as user manuals and administrator guides, see the additional documentation below.

The PBS manages jobs that users submit to various queues on the system, each queue representing a group of resources with attributes necessary for the queue's jobs. Commonly used PBS tools include qsub, for job submission; qstat, for monitoring the status of jobs; and qdel, for terminating jobs prior to completion. More detailed information regarding these commands and others is available below, or in the documentation mentioned above.

Policy

Jobs that are run interactively on the user nodes are limited to 20 minutes of CPU time. Monitoring scripts on the user nodes will kill processes exceeding 20 minutes of wall clock time.

Run jobs that require more than 20 minutes of CPU time on the head nodes, b005-b008. To access one of the head nodes, you must first log into Quarry and from there use ssh to connect to b005, b006, b007, or b008.

Queues

Effective October 2, 2007, the batch queues have been renamed on Big Red and Quarry to align with TeraGrid partners. This should make it easier to transition between TeraGrid resources.

On Quarry:

• defaultq is now known as long.
• fastq is now known as debug.
• osgq is now known as osg.

Wall time limits and node limits per queue have not changed. Aliases for the old names are in place for job submissions, but when you use commands like llclass and showq -w class=XXXXX, you must use the new names.

If you have comments or suggestions, email Research Technologies.

The following queues are available on Quarry:

Note: Cluster-wide maximum number of tasks is 336.

SERIAL queue properties

• Nodes: 5 serial + 33 normal + 42 long = 80 blades
• Maximum walltime: 12 hours
• Maximum nodes per job: 1 node
• Maximum cores per job: 8 cores
• Maximum number of jobs per queue: 3500
• Maximum number of jobs per user: 1500
• Direct submission: No

NORMAL queue properties

• Nodes: 33 normal + 42 long = 75 blades
• Maximum walltime: 7 days
• Maximum nodes per job: 2 nodes
• Maximum cores per job: 16 cores
• Maximum number of jobs per queue: None
• Maximum number of jobs per user: None
• Direct submission: No

LONG queue properties

• Nodes: 42 long = 42 blades
• Maximum walltime: 14 days
• Maximum nodes per job: 42 nodes
• Maximum cores per job: 336 cores
• Maximum number of jobs per queue: None
• Maximum number of jobs per user: None
• Direct submission: No

DEBUG queue properties

• Nodes: 4 blades dedicated
• Maximum walltime: 30 min
• Maximum nodes per job: 2 nodes
• Maximum cores per job: 16 cores
• Maximum number of jobs per queue: 1
• Maximum number of jobs per user: 1
• Direct submission: Yes

OSG queue properties (group restricted access)

• Nodes: 16 blades dedicated
• Maximum walltime: 14 days
• Maximum nodes per job: None
• Maximum cores per job: None
• Maximum number of jobs per queue: None
• Maximum number of jobs per user: None
• Direct submission: Yes

Note: Access to the osg queue is restricted and jobs are routed to this queue via Globus. The debug queue is for debugging only; once code has been debugged, you may submit it to the long queue. If you have questions about the job queues, email High Performance Systems.

If you do not specify a queue when you submit a job, it will automatically go into the long queue. The debug queue is intended for test jobs requiring 30 minutes or less of wall clock time. To use this queue you must submit your job directly to it:

Jobs

Scripts

PBS most commonly handles job scripts, although interactive jobs are also supported. A job script may be as simple as a bash or tcsh shell script, but also may include a number of PBS job directives. You must always begin PBS job scripts, which will be executed under your preferred login shell, with a "sha-bang" line specifying which command interpreter it should run under, for example:

PBS directives, which are lines beginning with the string #PBS, include switches for specifying such useful information as wall clock time required to complete the job, number of nodes and processors necessary, and filenames for job output and errors. These directives must be at the top of the script following the "sha-bang" line. An example PBS job script might look like this:

Line by line, this script says:

• Use bash as the command interpreter for this script.
• Keep the job output.
• This job requires four nodes, two processors per node, and 30 minutes of wall clock time
• Send job-related email to  username@indiana.edu .
• Send email if the job is aborted (a), when it begins (b), and when it ends (e).
• The job name is JobName.
• Join standard output and standard error.
• Execute ~/bin/binaryname on eight processors from the machines in $PBS_NODEFILE using mpirun.

For additional details on PBS directives, view the man pages by entering man qsub .

Submission

Submit jobs with the qsub command. If the command exits successfully, a job ID will be returned to standard output, for example:

If you require attribute values different from the defaults, but less than the maximum allowed, specify these either in the job script with PBS directives, or on the command line with the  -l  switch. For example, to submit a job that needs more than the default two hours of walltime on Quarry, use:

Note that command-line arguments override directives in the job script, and that you may specify many attributes on the command line, either as comma-separated options following the  -l  switch, or each with its own  -l  switch. The following two commands are equivalent:

Useful qsub switches include:

See the qsub man page for more information.

Monitoring

The qstat command is useful for monitoring the status of a queued or running job. Switches include:

For example, to see all the running jobs in the Quarry long queue, at the Quarry shell prompt, enter:

Another useful command for monitoring jobs is the Moab Scheduler showq. To list the queued jobs in dispatch order, enter:

See the showq man page for more information.

Deleting

Use the qdel command to delete queued or running jobs. Occasionally, a node will become unresponsive to the point that it cannot respond to the PBS server's requests that a job be killed. In that case, try adding the -W (uppercase W) force option to qdel. If that doesn't work, contact High Performance Systems for help.

Additional documentation

• Moab Workload Manager User's Manual
• Moab Workload Manager Administrator's Guide
• Moab Workload Manager command overview
• TORQUE home page
• TORQUE Wiki
• TORQUE Admin Manual
• TORQUE command overview