On Quarry, how do I run MPI jobs using a TORQUE script?

Note: After seven years of production service, Indiana University's Quarry research computing cluster was decommissioned January 30, 2015. IU students, faculty, and staff can request accounts on Karst, IU's newest high-throughput computing cluster; for instructions, see Requesting an account. User data on Quarry has not be removed; you can access your old home directory data on Quarry from your account on any other IU research computing resource (see Accessing one home directory from another). All software modules that were available on Quarry are available on Karst. If you have questions or concerns about Quarry's retirement, or need help, contact the High Performance Systems group.

To run parallel MPI jobs on the Quarry cluster at Indiana University, you must first add the necessary modules to your user environment. Once your user environment is configured properly, you can compile your code and submit your jobs to the TORQUE resource management system.

On this page:


Setting up your user environment using Modules

Open MPI wrapper compilers and libraries for the Intel Compiler Suite and GNU Compiler Collection (GCC) are available on Quarry. MPICH wrapper compilers and libraries for the Intel compilers are also available. The wrapper compilers add the necessary flags for compiling and linking MPI programs, and then invoke the respective back-end compilers to perform the actual commands.

You must add the appropriate compiler collection to your user environment on Quarry before adding the corresponding Open MPI or MPICH library:

  • For the Intel Open MPI or MPICH library:
    1. Add the Intel Compiler Collection to your user environment; on the command line, enter:
        module load intel
    2. Add the Intel Open MPI or MPICH library to your user environment. To load the desired module, on the command line, enter one of the following:
      • Open MPI (version 1.6.3):
          module load openmpi/intel
      • MPICH (version 3.0.1):
          module load mpich
  • For the GCC Open MPI library:
    1. Add the GNU Compiler Collection (GCC) to your user environment; on the command line, enter:
        module load gcc
    2. Add the GCC Open MPI library to your user environment; on the command line, enter:
        module load openmpi/gnu

To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

For more about Modules, see On Big Red II, Karst, and Mason at IU, how do I use Modules to manage my software environment?

Compiling your code and submitting jobs to TORQUE

After adding the necessary modules to your user environment, you can compile your code and submit parallel MPI jobs to the TORQUE resource management system on Quarry:

  1. Compile your MPI code using the appropriate MPI wrapper compiler.
    MPI wrapper compilers
    Language Wrapper compiler
    C mpicc
    C++ mpic++ or
    mpicxx
    Fortran 77 mpif77
    Fortran 90 mpif90

    Note: Manual pages are available for the Open MPI wrapper compilers; for example, to view the manual page for the Open MPI Fortran 90 wrapper compiler, on the command line, enter:

      man mpif90

    For example:

    • To compile and link an MPI program written in C, on the command line, enter:
        mpicc -o myprog myprog.c
    • To compile and link an MPI program written in C++, on the command line, enter:
        mpicxx -o myprog myprog.cxx
    • To compile and link an MPI program written in Fortran 90, on the command line, enter:
        mpif90 -o myprog myprog.f
  2. Create a TORQUE script (e.g., submit_parallel.sh) that includes the correct number of nodes and tasks, and the appropriate output/error files; for example:
      #PBS -l nodes=2:ppn=8,walltime=5:00
      #PBS -m ae
      #PBS -N job_myprog
    
      mpirun -machine $PBS_NODEFILE -np 16  ${HOME}/simple_quarry_jobs/myprog
  3. Submit the program to TORQUE. Use the qsub command from the command line; for example:
      [jjbinks@q0141 ~]$ qsub submit_parallel.sh

For more on TORQUE, see What is TORQUE, and how do I use it to submit and manage jobs on high-performance computing systems?

Getting help

If you have questions about compilers, libraries, or programming on Quarry, email the Scientific Applications and Performance Tuning group for help.

This is document avpc in the Knowledge Base.
Last modified on 2015-08-20.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.