ARCHIVED: Migration between IBM LoadLeveler and TORQUE

This content has been archived, and is no longer maintained by Indiana University. Information here may no longer be accurate, and links may no longer be available or reliable.

To help users migrate between the TORQUE job submission system (also called PBS, and used by Indiana University's Quarry cluster) and IBM LoadLeveler (used by IU's Big Red cluster), the following tables compare commands, environment variables, and resource specifications for both batch systems.

If you're not familiar with either LoadLeveler or TORQUE and new to Big Red, see ARCHIVED: At IU, what was Big Red?.

Commonly used commands

	LoadLeveler	TORQUE
Job submission	`llsubmit <scriptfile>`	`qsub <scriptfile>`
Job deletion	`llcancel <job_id>`	`qdel <job_id>`
Job status (for user)	`llq -u <username>`	`qstat -u <username>`
Extended job status	`llq -l <job_id>`	`qstat -f <job_id>`
Hold a job temporarily	`llhold <job_id>`	`qhold <job_id>`
Resume job on hold	`llhold -r <job_id>`	`qrls <job_id>`
List usable queues	`llclass`	`qstat -Q`
GUI for batch system	`xloadl`	`xpbs`

Environment variables

	LoadLeveler	TORQUE
Job ID	`$LOADL_STEP_ID`	`$PBS_JOBID`
Submission directory	`$LOADL_STEP_INITDIR`	`$PBS_O_WORKDIR`
Processor list	`$LOADL_PROCESSOR_LIST` (See Note below.)	`$PBS_NODEFILE`

Note: For jobs using more than 128 tasks (processors/cores), the value of $LOADL_PROCESSOR_LIST will be set to NULL. If your job uses more than 128 tasks, run:

  /opt/ibmll/LoadL/full/bin/ll_get_machine_list > /tmp/machinelist.$LOADL_STEP_ID

Then use the temporary file (/tmp/machinelist.$LOADL_STEP_ID) created to get the list of machines assigned to your job.

Resource specifications

	LoadLeveler	TORQUE
Queue	`#@ class=<queue>`	`#PBS -q <queue>`
Nodes	`#@ node=<#>`	`#PBS -l nodes=<#>`
Processors	`#@ tasks_per_node=<#>`	`#PBS -l ppn=<#>`
Wall clock limit	`#@ wall_clock_limit=<hh:mm:ss>`	`#PBS -l walltime=<hh:mm:ss>`
Standard output file	`#@ output=<file>`	`#PBS -o <file>`
Standard error	`#@ error=<file>`	`#PBS -e <file>`
Copy environment	`#@ environment=COPY_ALL`	`#PBS -V`
Notification event	`#@ notification=start\|error\|complete\|never\|always`	`#PBS -m abe`
Email address	`#@ notify_user=<email>`	`#PBS -M <email>`
Job name	`#@ job_name=<name>`	`#PBS -N <name>`
Job restart	`#@ restart=<yes\|no>`	`#PBS -r <y\|n>`
Job type	`#@ job_type=<type>`	N/A
Initial directory	`#@ initialdir=<directory>`	N/A
Node usage	`#@ node_usage=not_shared`	`#PBS -l naccesspolicy=singlejob`
Memory requirement	`#@ requirements=(Memory >= NumKiloBytes)`	`#PBS -l mem=NumBytes`

For more, see the LLGuide documents for Big Red and ARCHIVED: Use TORQUE to submit and manage jobs on high performance computing systems