DRAFT: On Karst or Carboonate at IU, how do I run parallel MPI jobs using a TORQUE script?

To run parallel MPI jobs on Karst or Carbonate at Indiana University, first add the necessary modules to your user environment. Once your user environment is configured properly, compile your code, and then prepare and submit your job script to the TORQUE resource management system.

On this page:


Setting up your user environment

Default versions of the GNU Compiler Collection (GCC) and the Intel Compiler Suite are automatically added to your user environment on Karst and Carbonate. Alternative versions may be available, depending on the system; to check for alternative versions, use the module avail command; for example, to check whether alternative versions of the Intel compilers are available, on the command line, enter:

  module avail intel

To replace the default compiler collection with an alternative version, use the module swap command; for example, to replace the default Intel compilers (e.g., intel/default_version) with an alternative version (e.g., intel/alt_version), on the command line, enter:

  module swap intel/default_version intel/alt_version

Open MPI wrapper compilers and libraries for the Intel and GNU compilers are available on Karst and Carbonate. Additionally, MPICH wrapper compilers and libraries for the Intel compilers are available on both systems; GCC MPICH wrapper compilers are available on Carbonate.

Wrapper compilers add the necessary flags for compiling and linking MPI programs, and then invoke the respective back-end compilers to perform the actual commands.

Use the module avail command to check for a preferred version of the openmpi or mpich module, and then use the module load command to add it to your user environment.

To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

For more about using Modules to configure your user environment, see On the research computing systems at IU, how do I use Modules to manage my software environment?

Compiling your code

Compile your MPI code using the appropriate MPI wrapper compiler:

MPI/MPICH wrapper compilers
Language Wrapper compiler Manual page
C mpicc man mpicc
C++ mpic++ man mpic++
Fortran mpifort man mpifort

For example:

  • To compile and link an MPI program written in C (e.g., my_mpi_program.c), on the command line, enter:
  •   mpicc my_mpi_program.c -o my_mpi_program.exe
    
  • To compile and link an MPI program written in C++ (e.g., my_mpi_program.cxx), on the command line, enter:
  •   mpicxx my_mpi_program.cxx -o my_mpi_program.exe
    
  • To compile and link an MPI program written in Fortran (e.g., my_mpi_program.f), on the command line, enter:
  •   mpiffort my_mpi_program.f -o my_mpi_program.exe
    

Submitting jobs to TORQUE

Create a TORQUE script (e.g., submit_parallel.sh) that indicates the correct number of nodes and tasks, the required wall-clock time, and the appropriate output/error files; for example:

  #!/bin/bash 
  #PBS -k o 
  #PBS -l nodes=2:ppn=6,walltime=30:00
  #PBS -M jthutt@tatooine.net
  #PBS -m abe 
  #PBS -N JobName 
  #PBS -j oe 

  mpiexec -np 12 -machinefile $PBS_NODEFILE ~/bin/binaryname
  • Submit the program to TORQUE. Use the qsub command from the command line; for example:
  •   qsub submit_parallel.sh
    

    For more on TORQUE, see What is TORQUE, and how do I use it to submit and manage jobs on high-performance computing systems?

    Getting help

    Support for IU research computing systems, software, and services is provided by various UITS Research Technologies units. For help, see Research computing support at IU.

    This is document avpc in the Knowledge Base.
    Last modified on 2017-08-31 18:32:21.

    • Fill out this form to submit your issue to the UITS Support Center.
    • Please note that you must be affiliated with Indiana University to receive support.
    • All fields are required.

    Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

    • Fill out this form to submit your comment to the IU Knowledge Base.
    • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

    Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.