ARCHIVED: Using LoadLeveler on Libra at IU

• Submitting batch jobs to LoadLeveler
• LoadLeveler batch job classes
• Sample LoadLeveler scripts
  • Serial job examples
  • Nonshared job example
  • Parallel job example
• LoadLeveler keywords
• Submitting and monitoring your job

Submitting batch jobs to LoadLeveler

To optimize job throughput and minimize paging on the Libra cluster at Indiana University, you must submit all CPU-intensive jobs (i.e., jobs requiring more than 20 minutes of CPU time) as batch jobs to a batch scheduler. Libra uses IBM's LoadLeveler, which dispatch jobs to the least busy host by means of a load-balancing algorithm. In addition, LoadLeveler provides job prioritizing, hold/release, and an easy-to-use graphical user interface for creating, submitting, and manipulating jobs. LoadLeveler's functionality is extended via its interface to an external scheduler, the Maui Scheduler, which offers superior backfill scheduling, quality of service control, advanced reservations, and more.

Your home directory resides on the UITS EMC storage system, and is accessible from all nodes. You may submit jobs to LoadLeveler from the interactive nodes, libra01 or libra02, and may read and write your input and output files from/to your home directory or from/to the local /tmp and /scr directories, or to the shared scratch file systems, /scratch1 and /scratch2. When your job is started, your default login shell is invoked, along with your .login and .cshrc files (for C shell users) or your .profile and .kshrc files (for Korn and Bourne shell users). Be sure that these files do not contain any commands that would not work in a batch environment (e.g.,the stty command, which expects stdout to be a terminal).

LoadLeveler batch job classes

Before you can submit your job to LoadLeveler, you must first determine its requirements in terms of CPU time, memory, and software and submit it to the appropriate class. The following classes are available on Libra:

• serial: 1 CPU, less than 2GB memory, 14-day wall clock time
• nonshared: 1-2 CPUs, less than 4GB memory, 14-day wall clock time
• smp: 1-16 CPUs, more than 4GB memory, 14-day wall clock time

For Fortran and C jobs and other serial applications, such as SAS, SPSS, Matlab, and Mathematica, use the class "serial". If your job requires more than 2GB but less than 4GB of memory, use the class "nonshared". If your job requires multiple CPUs and/or more than 4GB of memory, use class "smp". You must also specify a feature code if the software you require is not available on all nodes that run the desired job class.

For information about the LoadLeveler node configuration, see ARCHIVED: LoadLeveler configuration on Libra.

Sample LoadLeveler scripts

To communicate with LoadLeveler, you can use the X Window graphical user interface, xloadl, or LoadLeveler commands (e.g., llsubmit, llq, and llcancel). The xloadl interface allows you to create, submit, query, and manipulate your batch jobs. If you do not have an X-capable display or you prefer LoadLeveler command mode, to submit jobs you must first create a shell script that contains all the necessary information to set up and run the desired tasks. Within the LoadLeveler job script, specify job options using keywords on a line beginning with the special characters #@. Job options include job class, working directory, executable name, software requirements, where to redirect stdin, stdout, and stderr, and notification type.

Serial job examples

  #@ class = serial
  #@ initialdir = /N/u/joedoe/libra
  #@ input = myprogram.input
  #@ output = myprogram.output
  #@ error = myprogram.error
  #@ arguments = arg1 arg2 arg3
  #@ executable = myprogram
  #@ queue

Important: You must assign values to "output" and "error" if your program writes to stdout and/or stderr. If not specified, these default to /dev/null.

To ensure that the output from each job goes to a unique file, use the values assigned by LoadLeveler to the Executable, Cluster, and Process values. The Cluster value is a unique jobid; a Process value is assigned to each process queued within a script. Following is an example script that executes the myjob program twice, using different arguments and creating two unique output files:

  #@ class = serial
  #@ initialdir = /N/u/joedoe/libra/bin
  #@ executable = myprogram
  #@ output = $(Executable).$(Cluster).$(Process).
  #@ arguments = arg1 arg2 arg3
  #@ queue
  #@ arguments = arg4 arg5 arg6
  #@ queue

If you do not specify an executable name, LoadLeveler assumes that anything following the #@ queue statement consists of commands to be executed. Use this format if you want to run a shell script rather than a compiled program, or if several commands need to be run in sequence. Following is an example:

  #@ class = serial
  #@ initialdir = /N/u/joedoe/libra
  #@ output = myprog.$(Cluster).out
  #@ error = myprog.$(Cluster).err
  #@ queue
  xlf -o myprog myprog.f            
  myprog
  rm /scr/joedoe/myprog.workfile

The following script submits a SAS job:

  #@ class = serial
  #@ initialdir = /N/u/joedoe/libra/sasjobs
  #@ error = mysasjob.err
  #@ output = mysasjob.out
  #@ queue
  sas mysasjob.sas

The following script submits a Mathematica job:

  #@ class = serial
  #@ initialdir = /N/u/joedoe/libra/math
  #@ input = math.input
  #@ output = math.output
  #@ error = math.err
  #@ queue
  math 

The following script submits an Splus job. Splus is a statistics package that has a node-locked license on libra05; target your job to the correct node by requesting the defined feature code, in this case splus:

  #@ class = serial
  #@ requirements = (Feature == "splus")
  #@ initialdir = /N/u/joedoe/libra/Splus
  #@ input = Splus.commandfile
  #@ output = Splus.output 
  #@ error = Splus.error
  #@ queue
  Splus 

Note: If you enter an unavailable requirement, you will not receive a message. LoadLeveler will hold your job in the queue until the resource becomes available. Be sure to specify the resource correctly.

To see the list of defined feature codes, enter llconfig.

Nonshared job example

If you use the Gaussian03 software package, which can use more than one CPU, or other programs that require one or two CPUs and more than 2GB of memory, use the "nonshared" queue to submit your jobs to nodes that are not shared by other user programs:

  #@ class = nonshared
  #@ initialdir = /N/u/joedoe/libra/gaussian03
  #@ output = run2.output
  #@ error = run2.error
  #@ queue
  setenv g03root /software/sciapps/g03
  source \$g03root/g03/bsd/g03.login
  setenv GAUSS_SCRDIR /scr/\$USER
  /bin/mkdir $GAUSS_SCRDIR
  g03 run2.com run2.log
  /bin/rm -R \$GAUSS_SCRDIR

Parallel job example

The following script submits a 16-CPU OpenMP parallel job. Note that although you specify the number of CPUs to reserve using the tasks_per_node keyword, your OpenMP program will spawn only the number of threads specified by the value of the OMP_NUM_THREADS environment variable, so you must set the value in the batch script, as follows:

  #@ class = smp
  #@ job_type = parallel
  #@ node = 1
  #@ tasks_per_node = 16
  #@ initialdir = /N/u/joedoe/libra/myprogs
  #@ executable = myopenmpcode
  #@ input = inputfile1
  #@ output = $(Executable).$(Cluster).output
  #@ error = $(Executable).$(Cluster).error 
  #@ environment = COPY_ALL; OMP_NUM_THREADS=16
  #@ queue

Libra does not have a high-speed low-latency interconnect, and does not support MPI (distributed memory parallel) jobs.

LoadLeveler keywords

Submitting and monitoring the job

Once you've created your job script, enter chmod u+x to make the script executable. To submit the job script to LoadLeveler, enter:

 llsubmit  scriptname

If the submission is successful, LoadLeveler returns a jobid. For some useful LoadLeveler commands for tracking your job, see ARCHIVED: Monitoring LoadLeveler jobs on Big Red at IU.

LoadLeveler has been set up on Libra so that you cannot run more than four jobs simultaneously. If you submit more than eight jobs to the queue at one time, LoadLeveler will not consider the additional jobs eligible for queueing until previous jobs have completed. The default LoadLeveler FIFO Scheduler has been replaced by the Maui Scheduler, which uses a fairshare algorithm to determine the dispatch order of submitted jobs. Thus Maui considers both the submit time as well as the users' recent past use of batch cycles to prioritize jobs. To display fairshare usage statistics, enter showfairshare.