ARCHIVED: On Quarry at IU, what is the paralleljob script, and how do I use it to submit jobs?

On Quarry at Indiana University, the paralleljob script provides a convenient method for submitting parallel (multiple-processor) programs to the TORQUE resource manager. The paralleljob script works only for parallel applications that are "single-program, multiple-data" (i.e., a single binary). It does not work for parallel applications that consist of more than one binary (i.e., "multiple-programs, multiple-data"). For complete documentation, enter man paralleljob.

• Setting up your user environment
• Using the script

Setting up your user environment

To use the paralleljob script, your user environment must include the local-utils/20130301 module. To determine whether the local-utils/20130301 is already added, on the command line, enter:

  module list 

If you don't see local-utils/20130301 listed, use the module load command to add it to your environment:

  module load local-utils/20130301

For example, to automatically load the local-utils/20130301 module every time you log into Quarry, add the following line to your ~/.modules file:

  module load local-utils/20130301

Using the script

When you use paralleljob to submit a job on Quarry, you can specify the number of processes to start, how long the job should be allowed to run, and the queue to which the job should be submitted. By default, the script launches four processes for up to two hours. The syntax is paralleljob is (the flags in brackets are optional):

  paralleljob program_name program_options [-CPUS np] [-wallhours n] [-jobname my_job]

In the example above, replace:

• program_name with the name of the program you want to run
• program_options with any command-line options you want passed to the program
• np with the number of processes to start
• n with the number of hours the job should be allowed to run
• my_job with the name of your job

For example, to run a program called speedster on an input file called my_data.dat), setting the application's -speed option to super and launching four processes for up to two hours, on the command line, you would enter:

  paralleljob speedster -speed super mydata.dat

To run the same application (with the same options on the same input file), but launch 16 processes and run for up to 48 hours, you would enter:

  paralleljob speedster -speed super mydata.dat -CPUS 16 -wallhours 48

To run the same application (with the same options on the same input file), but launch 512 processes, run for up to 10 hours, and label the job my_job, you would enter:

  paralleljob speedster -speed super mydata.dat -CPUS 512 -wallhours 10 -jobname my_job

Note the following:

• When your job runs, the current working directory of your program will be the directory where you invoked paralleljob. If the program you want to run is not in your your default path, you must specify the full path to the program.
• The paralleljob script accepts only double-quoted arguments; it treats single quotes as double quotes.
• An appropriate version of mpirun must be in your path or specified by the MPIRUN environment variable. Each flavor of MPI has its own version of mpirun. If you have compiled your parallel application using mpicc or another appropriate compiler, mpirun should already be in your path.

If you need help or have questions about using the paralleljob script on Quarry, contact the UITS Scientific Applications and Performance Tuning (SciAPT) team.

If you have system-specific questions about Quarry, contact the UITS High Performance Systems group.