ARCHIVED: On Big Red, how do I run MPI jobs?
Note: Big Red, originally commissioned in 2006, was retired from service on September 30, 2013. Its replacement, Big Red II, is a hybrid CPU/GPU Cray XE6/XK7 system capable of achieving a theoretical peak performance (Rpeak) of 1 petaFLOPS, making it one of the two fastest university-owned supercomputers in the US. For more, see ARCHIVED: Get started on Big Red II. If you have questions or concerns about the retirement of Big Red, contact the High Performance Systems group.
On this page:
- MPI libraries available through
softenv
- Add the MPICH or Open MPI key to
.soft
- Compile your MPI code
- Submitting a parallel job
- Warning about the $OBJECT_MODE environment variable
- Understanding MPI package names
(
softenv
keys) on Big Red
MPI libraries available through softenv
To display the MPI libraries available on Big Red, use the
softenv
command, and use grep
to search for
mpi
.
The MPICH library is installed on Big Red. Among the other keys
listed, you should see packages named +mpich-mx-ibm-32
and mpich-mx-ibm-64
:
janedoe@BigRed:~/> softenv | grep mpi . . . +mpich-mx-ibm-32 MPICH MX xlc/xlf 32bit +mpich-mx-ibm-64 MPICH MX xlc/xlf 64bit . . . +openmpi-1.2.6-ibm-32 Openmpi-1.2.6 Ibm compilers 32-bit +openmpi-1.2.6-ibm-64 Openmpi-1.2.6 Ibm compilers 64-bit +openmpi-1.3.3-ibm-64 Openmpi-1.3.3 Ibm compilers 64-bit +openmpi-1.4.1-ibm-32 OpenMPI v1.4.1, IBM compilers +openmpi-1.4.1-ibm-64 OpenMPI v1.4.1, IBM compilers . . .
The key mpich-mx-ibm-32
, for example, indicates that
it is an MPICH library compiled with the IBM XL compiler in 32-bit
mode, and that it communicates with the mx
protocol. The
key openmpi-1.2-mx-ibm-64
indicates that it is an Open
MPI library (version 1.2) that was compiled with the IBM XL compilers
in 64-bit mode. For more, see Understanding MPI
package names (softenv
keys) on Big Red below.
Add the MPICH or Open MPI key to .soft
To use MPICH-MX, add the MPICH-MX key to softenv
. Add
+mpich-mx-ibm-32
or +mpich-mx-ibm-64
to your
~/.soft
file, depending on whether you need a 32-bit or a
64-bit MPICH library. Your ~/.soft
file should look
similar to this:
+mpich-mx-ibm-32 ## Some more keys you may have @bigred
To use Open MPI, add the Open MPI key to softenv
. Add
+openmpi-1.4.1-ibm-64
to your ~/.soft
file.
Your ~/.soft
file should look similar to this:
+openmpi-1.4.1-ibm-64 ## Some more keys you may have @bigred
If you are not familiar with the softenv
system, see
ARCHIVED: On Big Red at IU, how can I use SoftEnv to customize my
software environment? You may also enter man softenv-intro
at the prompt on Big Red.
Compile your MPI code
To compile a 32-bit parallel C program (assuming you have
+mpich-mx-ibm-32
in your ~/.soft
file), you
would use something like:
janedoe@BigRed:~/MPI_source_code> mpicc -q32 -o myprog myprog.c -L/opt/mx/lib32
To compile a 64-bit parallel Fortran 77 program (assuming you have
+mpich-mx-ibm-64
in your ~/.soft
file), you
would use something like:
janedoe@BigRed:~/MPI_source_code> mpif77 -q64 -o myprog myprog.f -L/opt/mx/lib64
To compile a 32-bit parallel C program using Open MPI 1.1.1
(assuming you have +openmpi-1.1.1-xlc-8.0-32
in your
~/.soft
file), you would use something like:
janedoe@BigRed:~/MPI_source_code> mpicc -q32 -o myprog myprog.c
Important: UITS strongly recommends using either
the -q32
or the -q64
switch, depending on
whether your code is 32- or 64-bit, when you compile and link your
program. For more, see the Warning about the
$OBJECT_MODE environment variable below.
Submitting a parallel job
You have three options for submitting a parallel job:
- Use
paralleljob
for simple MPI applications. - Run your job interactively for testing and debugging.
- Use LoadLeveler if you need the advanced
mpirun
options.
Submitting parallel jobs using the paralleljob
script
The paralleljob
script provides a convenient method
for submitting parallel (multiple-processor) programs to the
LoadLeveler batching and queuing system. Programs must consist of just
one executable file, in contrast to some master/worker programs in
which the master and workers are different executable files. For more,
see ARCHIVED: On Big Red at IU, how do I use the paralleljob script to submit
jobs? or enter man paralleljob
on Big
Red. The general form of the command is:
pw@BigRed:~/MPI_source_code> paralleljob ./hello
Running your job interactively with mpirun
To access one of the interactive nodes, you must first log into Big
Red, and from there use ssh
to connect to
b509
, b510
, b511
, or
b512
. Using your favorite editor, create a
machinefile
that looks similar to:
janedoe@BigRed:~/MPI_source_code> cat mfile b509 b510 b509 b510 b509 b510 b509 b510
Then, use mpirun
to run your parallel job:
janedoe@BigRed:~/MPI_source_code> mpirun -np 8 -machinefile mfile program-name
For np
, use the number of processes to start. For
mfile
, use the name of the machinefile
that
you created with the editor. For program-name
, substitute
the name of the program to submit.
Note: The interactive nodes mentioned above are open to all users. If you get the MX error below, try again:
MX:Aborting
MX:s9c4b7:send req(already completed):req status 8:Remote endpoint is closed
If the problem persists, email High Performance Systems for help.
Submitting a parallel job with a LoadLeveler script
To submit a parallel job that runs your MPI program, edit the sample LoadLeveler script shown below (or create your own) with the correct number of nodes and tasks, and the appropriate output/error files.
The sample script, parallel_job.sh
, follows:
#! /bin/bash -l ## LoadLeveler script to submit 2 node, 4 task MPI program: hello # @ job_type = MPICH # @ class = LONG # @ account_no = NONE # @ node = 2 # @ tasks_per_node = 4 # @ wall_clock_limit = 10:00:00 # @ notification = always # @ notify_user = <email_id> # @ environment=COPY_ALL; # @ output = hello.$(cluster).$(process).out # @ error = hello.$(cluster).$(process).err # @ queue ## Users should always cd into their execution directory due to ## a bug within LoadLeveler in dealing with the initialdir keyword. # cd <execution directory> cd ${HOME}/my_project ## Use mpirun to execute your MPICH program in parallel; ## $LOADL_TOTAL_TASKS and $LOADL_HOSTFILE are defined by ## LoadLeveler for jobs of type MPICH. mpirun -np $LOADL_TOTAL_TASKS -machinefile $LOADL_HOSTFILE ./hello
Use llsubmit
to submit
parallel_job.sh
:
pw@BigRed:~/MPI_source_code> llsubmit parallel_job.sh
You will see text similar to the following, which confirms that your job has been submitted:
llsubmit: Processed command file through Submit Filter: "/home/load. . . llsubmit: The job "s10c2b5.dim.2577" has been submitted.
If you are not familiar with job submission on Big Red, see the "Submitting batch jobs to LoadLeveler" section of ARCHIVED: At IU, what was Big Red?.
Warning about the $OBJECT_MODE environment variable
Note: If you use the -q32
or the
-q64
switch as recommended above, you may safely ignore
this section.
If you do not use the -q32
or -q64
switch, be aware of the $OBJECT_MODE environment variable, because it
also indicates to the IBM XL compilers whether an application you
compile is 32-bit or 64-bit.
The OBJECT_MODE variable has the following effects when you compile your program:
- Setting OBJECT_MODE to 64 will force the compiler to compile
64-bit object files. It is equivalent to explicitly specifying the
-q64
switch to the compiler. - Not setting OBJECT_MODE or setting it to 32 will force the
compiler to compile 32-bit object files. It is equivalent to
explicitly specifying the
-q32
switch to the compiler.
UITS recommends using the -q32
or -q64
switch, depending on whether you want to compile a 32-bit or a 64-bit
application, to avoid potential problems.
The OBJECT_MODE environment variable does not affect the execution of a program.
Understanding MPI package names (softenv
keys) on Big
Red
Big Red is a 64-bit system that supports both 32-bit and 64-bit applications. However, 32-bit object files cannot be linked with 64-bit ones.
The MPICH libraries and the Open MPI libraries on Big Red are
compiled with IBM's XL compiler suite in both 32- and 64-bit
modes. Communication protocols you can use include mx
(Myrinet Express) and TCP/IP over the Myrinet high-performance
network.
Note: Applications using the mx
protocol usually get better performance.
This is document autn in the Knowledge Base.
Last modified on 2023-04-21 16:58:24.