At IU, how do I use NAMD on Big Red II?

At Indiana University, Big Red II has MPI and GPU-accelerated versions of NAMD installed for running parallel batch jobs on the compute and hybrid CPU/GPU nodes in the Extreme Scalability Mode (ESM) execution environment. To avoid performance issues, run the GPU-accelerated version on Big Red II's CPU/GPU nodes (submit your job to the gpu queue), and run the MPI version on the compute nodes (submit your job to the cpu queue).

Note:
Some features are unavailable in the GPU-accelerated version, including alchemical free energy perturbation and the Lowe-Andersen thermostat.

On this page:


Setting up your user environment

To add either version of NAMD to your Big Red II user environment, you first must add the GNU Compiler Collection (i.e., load the gcc module). To determine whether the gcc module is loaded already, on the command line, enter:

  module list

If gcc isn't listed, load the default gcc module; on the command line, enter:

  module load gcc

Once you have the gcc module added:

  1. Check which versions of NAMD are available; on the command line, enter:
      module avail namd
    
  2. Add the desired version to your user environment; on the command line, enter (replace version with gpu or mpi):
      module load namd/gnu/version
    

To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

For example, to automatically load upon login the modules needed to run a GPU-accelerated NAMD job, add the following lines to your ~/.modules file:

  module load gcc
  module load namd/gnu/gpu

For more about using Modules to configure your user environment, see On the research computing systems at IU, how do I use Modules to manage my software environment?

Preparing a batch job script for the CPU/GPU nodes

To run NAMD in batch mode on the hybrid CPU/GPU nodes in Big Red II's native ESM execution environment, prepare a TORQUE job script (e.g., ~/work_directory/my_job_script.pbs) that specifies the resource requirements and other parameters appropriate for your job, and invokes the aprun command to properly launch the namd2 executable.

Following is a sample batch job script for running GPU-accelerated NAMD on four Big Red II CPU/GPU nodes:

  #!/bin/bash

  #PBS -l nodes=4:ppn=16,walltime=15:00
  #PBS -o  gpu_test_16_namd.out
  #PBS -e  gpu_test_16_namd.err
  #PBS -q gpu
  #PBS -N job_name
  #PBS -m ae
    
  cd  $PBS_O_WORKDIR
  aprun -n 4 -N 1 -d 16 namd2 +idlepoll +ppn 15 config_file

In the sample script above:

  • The -q gpu TORQUE directive routes the job to the gpu queue.
  • The cd $PBS_O_WORKDIR line changes the current working directory to the directory from which you submitted the script.
  • If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
  • When invoking aprun on the CPU/GPU nodes, the -n argument specifies the total number of nodes (not the total number of processing elements), and the -N argument specifies the number of GPUs per node, which on Big Red II is one (e.g., -N 1).
  • The +ppn switch indicates the number of processing elements per process.
  • The -d 16 argument indicates each process has a depth of 16 threads, allowing each process one communication thread and 15 worker threads (+ppn 15).
  • When you run GPU-accelerated NAMD, you must include the +idlepoll switch to poll the GPU for results while it is idle.
  • Substitute config_file with the file containing your NAMD configuration parameters; see NAMD configuration parameters in the NAMD user guide.
    Note:
    To offset the speed difference between non-bonded force evaluations (which are carried out on the GPUs) and bonded forces and Particle Mesh Ewald (PME) electrostatic calculations that run on the CPUs, in your configuration file, set the value of the outputEnergies parameter to 100 or higher, thereby increasing the number of timesteps between each energy output.

Alternatively, to run one process with one communication thread and three worker threads (i.e., a depth of four threads) on all four nodes, change the aprun line to:

  aprun -n 4 -N 1 -d 4 namd2 +idlepoll +ppn 3 config_file

Each namd2 thread can use only one GPU, so you need to run at least one thread for each GPU you want to use. Multiple threads can share a single GPU, usually with an increase in performance. The GPU version of NAMD automatically distributes threads equally among the available GPUs.

Preparing a batch job script for the compute nodes

To run NAMD in batch mode on the compute nodes in Big Red II's native ESM execution environment, prepare a TORQUE job script (e.g., ~/work_directory/my_job_script.pbs) that specifies the resource requirements and other parameters appropriate for your job, and invokes the aprun command to properly launch the namd2 executable.

Following is a sample job script for launching the MPI version of NAMD on two Big Red II compute nodes:

  #!/bin/bash 

  #PBS -l nodes=2:ppn=32,walltime=5:00 
  #PBS -o mpi_test_64_namd.out 
  #PBS -e mpi_test_64_namd.err 
  #PBS -q cpu 
  #PBS -N job_myprog  
  #PBS -m ae 
  
  cd $PBS_O_WORKDIR 
  aprun -n 64 namd2 config_file

In the sample script above:

  • The -q cpu TORQUE directive routes the job to the cpu queue.
  • The cd $PBS_O_WORKDIR line changes the current working directory to the directory from which you submitted the script.
  • If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
  • When invoking aprun on Big Red II's compute nodes, the -n argument specifies the total number of processors; here -n 64 represents two compute nodes with 32 processors each.
  • Substitute config_file with the file containing your NAMD configuration parameters; see NAMD configuration parameters in the NAMD user guide.

Submitting and monitoring your job

To submit your job script (e.g., ~/work_directory/my_job_script.pbs), use the TORQUE qsub command; for example, on the command line, enter:

  qsub [options] ~/work_directory/my_job_script.pbs

For a full description of the qsub command and available options, see its manual page.

To monitor the status of your job, use any of the following methods:

  • Use the TORQUE qstat command; on the command line, enter (replace username with the IU username you used to submit the job):
      qstat -u username
    
  • For a full description of the qstat command and available options, see its manual page.

  • Use the Moab checkjob command; on the command line enter (replace job_id with the ID number assigned to your job):
      checkjob job_id
    
  • For a full description of the checkjob command and available options, see its manual page.

Getting help

For more about running batch jobs on Big Red II, see How do I run batch jobs on Big Red II at IU?

Support for IU research computing systems, software, and services is provided by various UITS Research Technologies units. For help, see Research computing support at IU.

This is document bdkr in the Knowledge Base.
Last modified on 2017-10-10 13:59:44.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.