ARCHIVED: Use NAMD on Big Red II at IU

• Overview
• User environment setup
• GPU-accelerated batch job script preparation
• CPU-only batch job script preparation
• Job submission and monitoring
• Getting help

Overview

At Indiana University, Big Red II has MPI and GPU-accelerated versions of NAMD installed for running parallel batch jobs on the compute and hybrid CPU/GPU nodes in the Extreme Scalability Mode (ESM) execution environment. To avoid performance issues, run the GPU-accelerated version on Big Red II's CPU/GPU nodes (submit your job to the gpu queue), and run the MPI version on the compute nodes (submit your job to the cpu queue).

User environment setup

To add either version of NAMD to your Big Red II user environment, you first must add the GNU Compiler Collection (load the gcc module). To determine whether the gcc module is loaded already, on the command line, enter:

  module list

If gcc isn't listed, load the default gcc module; on the command line, enter:

  module load gcc

Once you have the gcc module added:

1. Check which versions of NAMD are available; on the command line, enter:

     module avail namd
   

2. Add the desired version to your user environment; on the command line, enter (replace version with gpu or mpi):

     module load namd/gnu/version
   

For example, to automatically load upon login the modules needed to run a GPU-accelerated NAMD job, add the following lines to your ~/.modules file:

  module load gcc
  module load namd/gnu/gpu

For more, see Use modules to manage your software environment on IU research supercomputers.

GPU-accelerated batch job script preparation

To run NAMD in batch mode on the hybrid CPU/GPU nodes in Big Red II's native ESM execution environment, prepare a TORQUE job script (for example, ~/work_directory/my_job_script.pbs) that specifies the resource requirements and other parameters appropriate for your job, and invokes the aprun command to properly launch the namd2 executable.

Following is a sample batch job script for running GPU-accelerated NAMD on four Big Red II CPU/GPU nodes:

  #!/bin/bash

  #PBS -l nodes=4:ppn=16,walltime=15:00
  #PBS -o  gpu_test_16_namd.out
  #PBS -e  gpu_test_16_namd.err
  #PBS -q gpu
  #PBS -N job_name
  #PBS -m ae
    
  cd  $PBS_O_WORKDIR
  aprun -n 4 -N 1 -d 16 namd2 +idlepoll +ppn 15 config_file

In the sample script above:

• The -q gpu TORQUE directive routes the job to the gpu queue.
• The cd $PBS_O_WORKDIR line changes the current working directory to the directory from which you submitted the script.
• If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
• When invoking aprun on the CPU/GPU nodes, the -n argument specifies the total number of nodes (not the total number of processing elements), and the -N argument specifies the number of GPUs per node, which on Big Red II is one (for example, -N 1).
• The +ppn switch indicates the number of processing elements per process.
• The -d 16 argument indicates each process has a depth of 16 threads, allowing each process one communication thread and 15 worker threads (+ppn 15).
• When you run GPU-accelerated NAMD, you must include the +idlepoll switch to poll the GPU for results while it is idle.
• Substitute config_file with the file containing your NAMD configuration parameters; see NAMD configuration parameters in the NAMD user guide.
  Note:

  To offset the speed difference between non-bonded force evaluations (which are carried out on the GPUs) and bonded forces and Particle Mesh Ewald (PME) electrostatic calculations that run on the CPUs, in your configuration file, set the value of the outputEnergies parameter to 100 or higher, thereby increasing the number of timesteps between each energy output.

Alternatively, to run one process with one communication thread and three worker threads (a depth of four threads) on all four nodes, change the aprun line to:

  aprun -n 4 -N 1 -d 4 namd2 +idlepoll +ppn 3 config_file

Each namd2 thread can use only one GPU, so you need to run at least one thread for each GPU you want to use. Multiple threads can share a single GPU, usually with an increase in performance. The GPU version of NAMD automatically distributes threads equally among the available GPUs.

CPU-only batch job script preparation

To run NAMD in batch mode on the compute nodes in Big Red II's native ESM execution environment, prepare a TORQUE job script (e.g., ~/work_directory/my_job_script.pbs) that specifies the resource requirements and other parameters appropriate for your job, and invokes the aprun command to properly launch the namd2 executable.

Following is a sample job script for launching the MPI version of NAMD on two Big Red II compute nodes:

  #!/bin/bash 

  #PBS -l nodes=2:ppn=32,walltime=5:00 
  #PBS -o mpi_test_64_namd.out 
  #PBS -e mpi_test_64_namd.err 
  #PBS -q cpu 
  #PBS -N job_myprog  
  #PBS -m ae 
  
  cd $PBS_O_WORKDIR 
  aprun -n 64 namd2 config_file

In the sample script above:

• The -q cpu TORQUE directive routes the job to the cpu queue.
• The cd $PBS_O_WORKDIR line changes the current working directory to the directory from which you submitted the script.
• If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
• When invoking aprun on Big Red II's compute nodes, the -n argument specifies the total number of processors; here -n 64 represents two compute nodes with 32 processors each.
• Substitute config_file with the file containing your NAMD configuration parameters; see NAMD configuration parameters in the NAMD user guide.

Job submission and monitoring

  qsub [options] ~/work_directory/my_job_script.pbs

Getting help