Indiana University
University Information Technology Services
  
What are archived documents?
Login>>
Login

Login is for authorized groups (e.g., UITS, OVPIT, and TCC) that need access to specialized Knowledge Base documents. Otherwise, simply use the Knowledge Base without logging in.

Close

ARCHIVED: On Big Red, how do I run MPI jobs?

Note: Big Red, originally commissioned in 2006, was retired from service on September 30, 2013. Its replacement, Big Red II, is a hybrid CPU/GPU Cray XE6/XK7 system capable of achieving a theoretical peak performance (Rpeak) of 1 petaFLOPS, making it one of the two fastest university-owned supercomputers in the US. For more, see Getting started on Big Red II. If you have questions or concerns about the retirement of Big Red, contact the High Performance Systems group.

On this page:


MPI libraries available through softenv

To display the MPI libraries available on Big Red, use the softenv command, and use grep to search for mpi.

The MPICH library is installed on Big Red. Among the other keys listed, you should see packages named +mpich-mx-ibm-32 and mpich-mx-ibm-64:

janedoe@BigRed:~/> softenv | grep mpi . . . +mpich-mx-ibm-32 MPICH MX xlc/xlf 32bit +mpich-mx-ibm-64 MPICH MX xlc/xlf 64bit . . . +openmpi-1.2.6-ibm-32 Openmpi-1.2.6 Ibm compilers 32-bit +openmpi-1.2.6-ibm-64 Openmpi-1.2.6 Ibm compilers 64-bit +openmpi-1.3.3-ibm-64 Openmpi-1.3.3 Ibm compilers 64-bit +openmpi-1.4.1-ibm-32 OpenMPI v1.4.1, IBM compilers +openmpi-1.4.1-ibm-64 OpenMPI v1.4.1, IBM compilers . . .

The key mpich-mx-ibm-32, for example, indicates that it is an MPICH library compiled with the IBM XL compiler in 32-bit mode, and that it communicates with the mx protocol. The key openmpi-1.2-mx-ibm-64 indicates that it is an Open MPI library (version 1.2) that was compiled with the IBM XL compilers in 64-bit mode. For more, see Understanding MPI package names (softenv keys) on Big Red below.

Back to top

Add the MPICH or Open MPI key to .soft

To use MPICH-MX, add the MPICH-MX key to softenv. Add +mpich-mx-ibm-32 or +mpich-mx-ibm-64 to your ~/.soft file, depending on whether you need a 32-bit or a 64-bit MPICH library. Your ~/.soft file should look similar to this:

+mpich-mx-ibm-32 ## Some more keys you may have @bigred

To use Open MPI, add the Open MPI key to softenv. Add +openmpi-1.4.1-ibm-64 to your ~/.softfile. Your ~/.soft file should look similar to this:

+openmpi-1.4.1-ibm-64 ## Some more keys you may have @bigred

If you are not familiar with the softenv system, see ARCHIVED: On Big Red at IU, how can I use SoftEnv to customize my software environment? You may also enter man softenv-intro at the prompt on Big Red.

Back to top

Compile your MPI code

To compile a 32-bit parallel C program (assuming you have +mpich-mx-ibm-32 in your ~/.soft file), you would use something like:

janedoe@BigRed:~/MPI_source_code> mpicc -q32 -o myprog myprog.c -L/opt/mx/lib32

To compile a 64-bit parallel Fortran 77 program (assuming you have +mpich-mx-ibm-64 in your ~/.soft file), you would use something like:

janedoe@BigRed:~/MPI_source_code> mpif77 -q64 -o myprog myprog.f -L/opt/mx/lib64

To compile a 32-bit parallel C program using Open MPI 1.1.1 (assuming you have +openmpi-1.1.1-xlc-8.0-32 in your ~/.soft file), you would use something like:

janedoe@BigRed:~/MPI_source_code> mpicc -q32 -o myprog myprog.c

Important: UITS strongly recommends using either the -q32 or the -q64 switch, depending on whether your code is 32- or 64-bit, when you compile and link your program. For more, see the Warning about the $OBJECT_MODE environment variable below.

Back to top

Submitting a parallel job

You have three options for submitting a parallel job:

  • Use paralleljob for simple MPI applications.
  • Run your job interactively for testing and debugging.
  • Use LoadLeveler if you need the advanced mpirun options.

Submitting parallel jobs using the paralleljob script

The paralleljob script provides a convenient method for submitting parallel (multiple-processor) programs to the LoadLeveler batching and queuing system. Programs must consist of just one executable file, in contrast to some master/worker programs in which the master and workers are different executable files. For more, see ARCHIVED: On Big Red at IU, how do I use the paralleljob script to submit jobs? or enter man paralleljob on Big Red. The general form of the command is:

pw@BigRed:~/MPI_source_code> paralleljob ./hello

Running your job interactively with mpirun

To access one of the interactive nodes, you must first log into Big Red, and from there use ssh to connect to b509, b510, b511, or b512. Using your favorite editor, create a machinefile that looks similar to:

janedoe@BigRed:~/MPI_source_code> cat mfile b509 b510 b509 b510 b509 b510 b509 b510

Then, use mpirun to run your parallel job:

janedoe@BigRed:~/MPI_source_code> mpirun -np 8 -machinefile mfile program-name

For np, use the number of processes to start. For mfile, use the name of the machinefile that you created with the editor. For program-name, substitute the name of the program to submit.

Note: The interactive nodes mentioned above are open to all users. If you get the MX error below, try again:

MX:Aborting
MX:s9c4b7:send req(already completed):req status 8:Remote endpoint is closed

If the problem persists, email High Performance Systems for help.

Submitting a parallel job with a LoadLeveler script

To submit a parallel job that runs your MPI program, edit the sample LoadLeveler script shown below (or create your own) with the correct number of nodes and tasks, and the appropriate output/error files.

The sample script, parallel_job.sh, follows:

#! /bin/bash -l ## LoadLeveler script to submit 2 node, 4 task MPI program: hello # @ job_type = MPICH # @ class = LONG # @ account_no = NONE # @ node = 2 # @ tasks_per_node = 4 # @ wall_clock_limit = 10:00:00 # @ notification = always # @ notify_user = <email_id> # @ environment=COPY_ALL; # @ output = hello.$(cluster).$(process).out # @ error = hello.$(cluster).$(process).err # @ queue ## Users should always cd into their execution directory due to ## a bug within LoadLeveler in dealing with the initialdir keyword. # cd <execution directory> cd ${HOME}/my_project ## Use mpirun to execute your MPICH program in parallel; ## $LOADL_TOTAL_TASKS and $LOADL_HOSTFILE are defined by ## LoadLeveler for jobs of type MPICH. mpirun -np $LOADL_TOTAL_TASKS -machinefile $LOADL_HOSTFILE ./hello

Use llsubmit to submit parallel_job.sh:

pw@BigRed:~/MPI_source_code> llsubmit parallel_job.sh

You will see text similar to the following, which confirms that your job has been submitted:

llsubmit: Processed command file through Submit Filter: "/home/load. . . llsubmit: The job "s10c2b5.dim.2577" has been submitted.

If you are not familiar with job submission on Big Red, see the "Submitting batch jobs to LoadLeveler" section of At IU, what was Big Red?.

Back to top

Warning about the $OBJECT_MODE environment variable

Note: If you use the -q32 or the -q64 switch as recommended above, you may safely ignore this section.

If you do not use the -q32 or -q64 switch, be aware of the $OBJECT_MODE environment variable, because it also indicates to the IBM XL compilers whether an application you compile is 32-bit or 64-bit.

The OBJECT_MODE variable has the following effects when you compile your program:

  • Setting OBJECT_MODE to 64 will force the compiler to compile 64-bit object files. It is equivalent to explicitly specifying the -q64 switch to the compiler.

  • Not setting OBJECT_MODE or setting it to 32 will force the compiler to compile 32-bit object files. It is equivalent to explicitly specifying the -q32 switch to the compiler.

UITS recommends using the -q32 or -q64 switch, depending on whether you want to compile a 32-bit or a 64-bit application, to avoid potential problems.

The OBJECT_MODE environment variable does not affect the execution of a program.

Back to top

Understanding MPI package names (softenv keys) on Big Red

Big Red is a 64-bit system that supports both 32-bit and 64-bit applications. However, 32-bit object files cannot be linked with 64-bit ones.

The MPICH libraries and the Open MPI libraries on Big Red are compiled with IBM's XL compiler suite in both 32- and 64-bit modes. Communication protocols you can use include mx (Myrinet Express) and TCP/IP over the Myrinet high-performance network.

Note: Applications using the mx protocol usually get better performance.

Back to top

This is document autn in domain all.
Last modified on October 09, 2013.

I need help with a computing problem

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.



Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

I have a comment for the Knowledge Base

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.