ARCHIVED: Migration between IBM LoadLeveler and TORQUE

To help users migrate between the TORQUE job submission system (also called PBS, and used by Indiana University's Quarry cluster) and IBM LoadLeveler (used by IU's Big Red cluster), the following tables compare commands, environment variables, and resource specifications for both batch systems.

If you're not familiar with either LoadLeveler or TORQUE and new to Big Red, see At IU, what was Big Red?.

Commonly used commands


  LoadLeveler TORQUE
Job submission llsubmit <scriptfile> qsub <scriptfile>
Job deletion llcancel <job_id> qdel <job_id>
Job status (for user) llq -u <username> qstat -u <username>
Extended job status llq -l <job_id> qstat -f <job_id>
Hold a job temporarily llhold <job_id> qhold <job_id>
Resume job on hold llhold -r <job_id> qrls <job_id>
List usable queues llclass qstat -Q
GUI for batch system xloadl xpbs

Environment variables


  LoadLeveler TORQUE
Job ID $LOADL_STEP_ID $PBS_JOBID
Submission directory $LOADL_STEP_INITDIR $PBS_O_WORKDIR
Processor list
$LOADL_PROCESSOR_LIST (See Note below.)
$PBS_NODEFILE

Note: For jobs using more than 128 tasks (processors/cores), the value of $LOADL_PROCESSOR_LIST will be set to NULL. If your job uses more than 128 tasks, run:

  /opt/ibmll/LoadL/full/bin/ll_get_machine_list > /tmp/machinelist.$LOADL_STEP_ID

Then use the temporary file (/tmp/machinelist.$LOADL_STEP_ID) created to get the list of machines assigned to your job.

Resource specifications


  LoadLeveler TORQUE
Queue #@ class=<queue> #PBS -q <queue>
Nodes #@ node=<#> #PBS -l nodes=<#>
Processors #@ tasks_per_node=<#> #PBS -l ppn=<#>
Wall clock limit #@ wall_clock_limit=<hh:mm:ss> #PBS -l walltime=<hh:mm:ss>
Standard output file #@ output=<file> #PBS -o <file>
Standard error #@ error=<file> #PBS -e <file>
Copy environment #@ environment=COPY_ALL #PBS -V
Notification event #@ notification=start|error|complete|never|always #PBS -m abe
Email address #@ notify_user=<email> #PBS -M <email>
Job name #@ job_name=<name> #PBS -N <name>
Job restart #@ restart=<yes|no> #PBS -r <y|n>
Job type #@ job_type=<type> N/A
Initial directory #@ initialdir=<directory> N/A
Node usage #@ node_usage=not_shared #PBS -l naccesspolicy=singlejob
Memory requirement #@ requirements=(Memory >= NumKiloBytes) #PBS -l mem=NumBytes

For more, see the LLGuide documents for Big Red and What is TORQUE, and how do I use it to submit and manage jobs on high-performance computing systems?

This is document axpz in the Knowledge Base.
Last modified on 2012-10-25 00:00:00.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.