On Big Red II at IU, how do I run OpenMP or hybrid OpenMP/MPI jobs?

On this page:


Overview

On Big Red II at Indiana University, the Extreme Scalability Mode (ESM) execution environment supports C, C++, and Fortran applications that use shared memory (OpenMP) parallelism, and hybrid OpenMP/MPI applications that use OpenMP threading to reduce the amount of memory needed to perform MPI tasks.

To use OpenMP on Big Red II, you must build your code with a compiler-specific command-line flag that enables OpenMP. Additionally, your TORQUE job script must set the OpenMP environment variable OMP_NUM_THREADS and use the aprun application launch command to instruct the runtime environment how to distribute processing elements.

Although you'll use the same compiler driver commands ( cc to compile C code, CC to compile C++ code, and ftn to compile Fortran code) for whichever programming environment module that's added to your user environment, the command-line flag used to enable OpenMP will depend on which programming environment module (i.e., compiler collection) is currently loaded in your Big Red II user environment:

Compiler collection
Programming environment module
OpenMP command-line flag
Cray Compiler Environment PrgEnv-cray (see note below)
GNU Compiler Collection (GCC) PrgEnv-gnu -fopenmp
Intel Compiler Suite PrgEnv-intel -openmp
Portland Group (PGI) compilers
PrgEnv-pgi -mp
Note:
The Cray compilers have OpenMP support enabled by default.

Running OpenMP applications

To run an OpenMP application in the ESM execution environment on Big Red II, first compile your code:

  1. Verify that the desired programming environment module is loaded. The Cray programming environment module ( PrgEnv-cray) is the default on Big Red II. To see which programming environment module is currently loaded, run module list on the command line, and then review the list of currently loaded modules. If necessary, use the module swap command to replace the currently loaded programming environment module (e.g., PrgEnv-cray) with the desired one (e.g., PrgEnv-gnu); for example, on the command line, enter:
      module swap PrgEnv-cray PrgEnv-gnu

    Alternatively, use the module unload command to remove the currently loaded programming environment:

      module unload PrgEnv-cray

    Then, add the desired programming environment module (e.g., PrgEnv-gnu) by adding this line, for example, to your ~/.modules file:

      module load PrgEnv-gnu

    To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

    For more about using Modules to configure your user environment, see On the research computing systems at IU, how do I use Modules to manage my software environment?

  2. Execute the language-specific compiler driver command with the compiler-specific OpenMP flag added. For example:
    • If you're using the GNU programming environment (i.e., the PrgEnv-gnu module), to compile the C program my_sourcefile.c with OpenMP support enabled, on the command line, enter:
        cc -fopenmp -o my_binary my_sourcefile.c
    • If you're using the PGI programming environment (i.e., the PrgEnv-pgi module), to compile the C++ application my_sourcefile.CC with OpenMP support enabled, on the command line, enter:
        CC -mp -o my_binary my_sourcefile.CC
    Note:
    Within a given programming environment, consult the specific compiler's manual page (e.g., man gcc or man pgCC) for any options its OpenMP flag may take.
  3. Prepare a TORQUE script that includes PBS directives (specifying the resources required to run your application) and the aprun application launch command (instructing the runtime environment how to distribute processing elements among those resources).

    The following example job script runs my_binary on all 32 cores of a XE6 node on Big Red II:

      #!/bin/bash
      #PBS -l nodes=1:ppn=32
      #PBS -l walltime=00:10:00
      #PBS -N my_job 
      #PBS -q cpu
      
      export OMP_NUM_THREADS=32
      aprun -n 1 -d 32 my_binary

    In this example:

    • The -n option tells aprun to dispatch one processing element for the job.
    • The -d option tells aprun each processing element should have a depth of 32 CPU cores.

Running hybrid OpenMP/MPI applications

To run an MPI application with OpenMP threading added:

  1. Make sure the desired programming environment module is added to your user environment (refer to step 1 in the previous section).
  2. Compile your code using the language-specific compiler driver command with the compiler-specific OpenMP flag added.

    For example:

    • If you're using the GNU programming environment (i.e., the PrgEnv-gnu module), to compile the C MPI program my_mpi_sourcefile.c with OpenMP support enabled, on the command line, enter:
        cc -fopenmp -o my_mpi_binary my_mpi_sourcefile.c
    • If you're using the PGI programming environment (i.e., the PrgEnv-pgi module), to compile the C++ MPI program my_mpi_sourcefile.CC with OpenMP support enabled, on the command line, enter:
        CC -mp -o my_mpi_binary my_mpi_sourcefile.CC
    Note:
    Within a given programming environment, consult the specific compiler's manual page (e.g., man gcc or man pgCC) for any options its OpenMP flag may take.
  3. Prepare a TORQUE job script to run the application using any combination of MPI ranks and threads, on any group of nodes.

    The following example job script runs a job across 32 nodes with four MPI ranks per node, each having eight OpenMP threads:

      #!/bin/bash
      #PBS -l nodes=32:ppn=32
      #PBS -l walltime=00:10:00
      #PBS -N my_mpi_job 
      #PBS -q cpu
      
      export OMP_NUM_THREADS=8
      aprun -n 128 -N 4 -d 8 my_mpi_binary

    In the above example:

    • The -n option tells aprun to dispatch a total of 128 processing elements (equivalent to MPI ranks, for the purposes of this example).
    • The -N option tells aprun to dispatch four processing elements per node.
    • The -d option tells aprun each processing element should have a depth of eight CPU cores (i.e., each processing element can accommodate a total of eight OpenMP threads).

Getting help

Support for IU research computing systems, software, and services is provided by various UITS Research Technologies units. For help, see Research computing support at IU.

Back to top

This is document bdos in the Knowledge Base.
Last modified on 2017-11-21 13:41:27.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.