ARCHIVED: Use Amber on Big Red II at IU

• Overview
• User environment setup
• Batch job script preparation
  • GPU-accelerated job
  • CPU-only job
• Job submission and monitoring
• Getting help

Overview

Assisted Model Building with Energy Refinement (Amber) is a suite of applications used for molecular modeling and performing molecular dynamics simulations. The Amber suite is distributed in two parts: AmberTools, a collection of independently developed, open source applications, and Amber, which is licensed software centered around the sander and pmemd simulation programs. Amber was developed originally under the leadership of Peter Kollman at the University of California, San Francisco, and is collaboratively maintained by a community of developers from several academic institutions. For more, see the Amber home page.

The term "amber" also refers to a family of empirical force fields for the simulation of biomolecules. The force fields, which are in the public domain, and the software suite are separate entities; other programs implement the Amber force fields, and the Amber applications can implement other force fields.

At Indiana University, Amber is available on Big Red II. You can use the GPU version to run GPU-accelerated parallel jobs on Big Red II's hybrid CPU/GPU nodes (support is provided for single-GPU and multiple-GPU runs), or use the MPI version to run parallel jobs on the compute nodes.

User environment setup

To run Amber on Big Red II, you first must add the GNU programming environment (load the PrgEnv-gnu module). To determine which programming environment is currently loaded, on the command line, enter:

  module list

If another programming environment module (for example, the PrgEnv-cray module) is listed, use the module swap command to replace it with the PrgEnv-gnu module:

  module swap PrgEnv-cray PrgEnv-gnu

Once you have the PrgEnv-gnu module loaded, you will need to add other prerequisite modules specific to the version of Amber you will be using; for example, to use the current default version (the amber/gnu/gpu/16 module), you also must add Python and the NVIDIA CUDA Toolkit to your user environment:

1. To see which versions of Amber are available; on the command line, enter:

     module avail amber
   

2. To see which modules are prerequisite for a particular version, on the command line, enter (replace version with the version number):

     module show amber/gnu/gpu/version
   

3. Use the module load command to add the required packages to your user environment.

   For example, to automatically load upon login the packages needed to run the default version of Amber, add the following lines (in this order) to your ~/.modules file:

     module swap PrgEnv-cray PrgEnv-gnu 
     module load python
     module load cudatoolkit 
     module load amber/gnu/gpu

For more, see Use modules to manage your software environment on IU research supercomputers.

Batch job script preparation

To run Amber simulations in batch mode on the hybrid CPU/GPU nodes in Big Red II's native ESM execution environment, prepare a TORQUE job script (for example, ~/work_directory/my_job_script.pbs) that specifies the application you want to run, sets the resource requirements and other parameters appropriate for your job, and invokes the aprun command to properly launch your executable.

GPU-accelerated job

Following are sample job scripts for running GPU-accelerated Amber simulations on Big Red II:

• PMEMD simulation with single-GPU acceleration: This sample script launches the single-GPU version of PMEMD ( pmemd.cuda) to run a single simulation on one CPU/GPU node:

    #!/bin/bash 
    
    #PBS -l nodes=1:ppn=16,walltime=3:00:00
    #PBS -q gpu
    #PBS -o out.log
    #PBS -e err.log
    
    cd $PBS_O_WORKDIR  
    aprun -n 1 -N 1 pmemd.cuda -O -i mdin -o mdout -p prmtop -c inpcrd -r restrt -x mdcrd
  

• PMEMD simulation with multiple-GPU acceleration: This sample script launches the multiple-GPU version of PMEMD ( pmemd.cuda.mpi) to run a single simulation on four CPU/GPU nodes:

    #!/bin/bash 
    
    #PBS -l nodes=4:ppn=16,walltime=3:00:00
    #PBS -q gpu
    #PBS -o out.log
    #PBS -e err.log
    
    cd $PBS_O_WORKDIR 
    aprun -n 4 -N 1 pmemd.cuda.MPI -O -i mdin -o mdout -p prmtop -c inpcrd -r restrt -x mdcrd
  

In each sample script above:

• The -q gpu TORQUE directive routes the job to the gpu queue.
• The cd $PBS_O_WORKDIR line changes the current working directory to the directory from which you submitted the script.
• If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
• When invoking aprun on the CPU/GPU nodes, the -n argument specifies the total number of nodes (not the total number of processing elements), and the -N argument specifies the number of GPUs per node, which on Big Red II is one (for example, -N 1).
• The -O flag allows the Amber program to overwrite the output file.
• The following Amber arguments control input:

  Argument	Function
  -i	Specifies the control data file (for example, mdin), which contains a series of namelists and control variables for determining type of simulation and options
  -p	Specifies the topology file (for example, prmtop), which contains a description of the molecular topology and the necessary force field parameters
  -c	Specifies the coordinate file (for example, inpcrd), which contains a description of the atom coordinates (and, optionally, velocities and periodic box dimensions)

• The following Amber arguments control output:

  Argument	Function
  -o	Specifies the output information file (e.g., mdout)
  -r	Specifies the file (for example, restrt) containing the final coordinates, velocity, and box dimensions for a restart run
  -x	Specifies the file (for example, mdcrd) to which trajectory coordinates are written

CPU-only job

Following is a sample script for launching the MPI version of sander on two compute nodes:

  #!/bin/bash 
  
  #PBS -l nodes=2:ppn=32,walltime=3:00:00
  #PBS -q cpu
  #PBS -o out.log
  #PBS -e err.log
  
  cd $PBS_O_WORKDIR
  aprun -n 64 sander.MPI -O -i mdin -o mdout -p prmtop -c inpcrd -r restrt

In the sample script above:

• The -q cpu TORQUE directive routes the job to the cpu queue on Big Red II.
• The cd $PBS_O_WORKDIR line changes the current working directory to the directory from which you submitted the script.
• If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
• When invoking aprun on compute nodes, the -n argument specifies the total number of processors; here -n 64 represents two compute nodes with 32 processors each.
• The -O flag allows the Amber program to overwrite the output file.
• The following Amber arguments control input:

  Argument	Function
  -i	Specifies the control data file (for example, mdin), which contains a series of namelists and control variables for determining type of simulation and options
  -p	Specifies the topology file (for example, prmtop), which contains a description of the molecular topology and the necessary force field parameters
  -c	Specifies the coordinate file (for example, inpcrd), which contains a description of the atom coordinates (and, optionally, velocities and periodic box dimensions)

• The following Amber arguments control output:

  Argument	Function
  -o	Specifies the output information file (e.g., mdout)
  -r	Specifies the file (for example, restrt) containing the final coordinates, velocity, and box dimensions for a restart run

Job submission and monitoring

  qsub [options] ~/work_directory/my_job_script.pbs

Getting help