At IU, how do I use Amber on Big Red II?

Assisted Model Building with Energy Refinement (Amber) is a suite of applications used for molecular modeling and performing molecular dynamics simulations. The Amber suite is distributed in two parts: AmberTools, a collection of independently developed, open source applications, and Amber, which is licensed software centered around the sander and pmemd simulation programs. Amber was developed originally under the leadership of Peter Kollman at the University of California, San Francisco, and is collaboratively maintained by a community of developers from several academic institutions. For more, see the Amber home page.

The term "amber" also refers to a family of empirical force fields for the simulation of biomolecules. The force fields, which are in the public domain, and the software suite are separate entities; other programs implement the Amber force fields, and the Amber applications can implement other force fields.

At Indiana University, Amber is available on Big Red II. You can use the GPU version to run GPU-accelerated parallel jobs on Big Red II's hybrid CPU/GPU nodes (support is provided for single-GPU and multiple-GPU runs), or use the MPI version to run parallel jobs on the compute nodes.

On this page:


Setting up your user environment

To run Amber on Big Red II, you first must add the GNU programming environment (i.e., load the PrgEnv-gnu module). To determine which programming environment is currently loaded, on the command line, enter:

  module list

If another programming environment module (e.g., the PrgEnv-cray module) is listed, use the module swap command to replace it with the PrgEnv-gnu module:

  module swap PrgEnv-cray PrgEnv-gnu

Once you have the PrgEnv-gnu module loaded, you will need to add other prerequisite modules specific to the version of Amber you will be using; for example, to use the current default version (i.e., the amber/gnu/gpu/16 module), you also must add Python and the NVIDIA CUDA Toolkit to your user environment:

  1. To see which versions of Amber are available; on the command line, enter:
  2.   module avail amber
    
  3. To see which modules are prerequisite for a particular version, on the command line, enter (replace version with the version number):
  4.   module show amber/gnu/gpu/version
    
  5. Use the module load command to add the required packages to your user environment.
  6. To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

    For example, to automatically load upon login the packages needed to run the default version of Amber, add the following lines (in this order) to your ~/.modules file:

      module swap PrgEnv-cray PrgEnv-gnu 
      module load python
      module load cudatoolkit 
      module load amber/gnu/gpu

For more about using Modules to configure your user environment, see On the research computing systems at IU, how do I use Modules to manage my software environment?

Preparing a batch job script

To run Amber simulations in batch mode on the hybrid CPU/GPU nodes in Big Red II's native ESM execution environment, prepare a TORQUE job script (e.g., ~/work_directory/my_job_script.pbs) that specifies the application you want to run, sets the resource requirements and other parameters appropriate for your job, and invokes the aprun command to properly launch your executable.

GPU-accelerated job

Following are sample job scripts for running GPU-accelerated Amber simulations on Big Red II:

  • PMEMD simulation with single-GPU acceleration: This sample script launches the single-GPU version of PMEMD (pmemd.cuda) to run a single simulation on one CPU/GPU node:
  •   #!/bin/bash 
      
      #PBS -l nodes=1:ppn=16,walltime=3:00:00
      #PBS -q gpu
      #PBS -o out.log
      #PBS -e err.log
      
      cd $PBS_O_WORKDIR  
      aprun -n 1 -N 1 pmemd.cuda -O -i mdin -o mdout -p prmtop -c inpcrd -r restrt -x mdcrd
    
  • PMEMD simulation with multiple-GPU acceleration: This sample script launches the multiple-GPU version of PMEMD (pmemd.cuda.mpi) to run a single simulation on four CPU/GPU nodes:
  •   #!/bin/bash 
      
      #PBS -l nodes=4:ppn=16,walltime=3:00:00
      #PBS -q gpu
      #PBS -o out.log
      #PBS -e err.log
      
      cd $PBS_O_WORKDIR 
      aprun -n 4 -N 1 pmemd.cuda.MPI -O -i mdin -o mdout -p prmtop -c inpcrd -r restrt -x mdcrd
    

In each sample script above:

  • The -q gpu TORQUE directive routes the job to the gpu queue.
  • The cd $PBS_O_WORKDIR line changes the current working directory to the directory from which you submitted the script.
  • If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
  • When invoking aprun on the CPU/GPU nodes, the -n argument specifies the total number of nodes (not the total number of processing elements), and the -N argument specifies the number of GPUs per node, which on Big Red II is one (e.g., -N 1).
  • The -O flag allows the Amber program to overwrite the output file.
  • The following Amber arguments control input:
  • Argument Function
    -i Specifies the control data file (e.g., mdin), which contains a series of namelists and control variables for determining type of simulation and options
    -p Specifies the topology file (e.g., prmtop), which contains a description of the molecular topology and the necessary force field parameters
    -c Specifies the coordinate file (e.g., inpcrd), which contains a description of the atom coordinates (and, optionally, velocities and periodic box dimensions)
  • The following Amber arguments control output:
  • Argument Function
    -o Specifies the output information file (e.g., mdout)
    -r Specifies the file (e.g., restrt) containing the final coordinates, velocity, and box dimensions for a restart run
    -x Specifies the file (e.g., mdcrd) to which trajectory coordinates are written
Note:
To run multiple independent GPU-accelerated simulations, UITS recommends using one GPU per job. For more about running GPU-accelerated PMEMD simulations, see GPU accelerated PMEMD in the Amber 12 Reference Manual.

CPU-only job

Following is a sample script for launching the MPI version of sander on two compute nodes:

  #!/bin/bash 
  
  #PBS -l nodes=2:ppn=32,walltime=3:00:00
  #PBS -q cpu
  #PBS -o out.log
  #PBS -e err.log
  
  cd $PBS_O_WORKDIR
  aprun -n 64 sander.MPI -O -i mdin -o mdout -p prmtop -c inpcrd -r restrt

In the sample script above:

  • The -q cpu TORQUE directive routes the job to the cpu queue on Big Red II.
  • The cd $PBS_O_WORKDIR line changes the current working directory to the directory from which you submitted the script.
  • If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
  • When invoking aprun on compute nodes, the -n argument specifies the total number of processors; here -n 64 represents two compute nodes with 32 processors each.
  • The -O flag allows the Amber program to overwrite the output file.
  • The following Amber arguments control input:
  • Argument Function
    -i Specifies the control data file (e.g., mdin), which contains a series of namelists and control variables for determining type of simulation and options
    -p Specifies the topology file (e.g., prmtop), which contains a description of the molecular topology and the necessary force field parameters
    -c Specifies the coordinate file (e.g., inpcrd), which contains a description of the atom coordinates (and, optionally, velocities and periodic box dimensions)
  • The following Amber arguments control output:
  • Argument Function
    -o Specifies the output information file (e.g., mdout)
    -r Specifies the file (e.g., restrt) containing the final coordinates, velocity, and box dimensions for a restart run

Submitting and monitoring your job

To submit your job script (e.g., ~/work_directory/my_job_script.pbs), use the TORQUE qsub command; for example, on the command line, enter:

  qsub [options] ~/work_directory/my_job_script.pbs

For a full description of the qsub command and available options, see its manual page.

To monitor the status of your job, use any of the following methods:

  • Use the TORQUE qstat command; on the command line, enter (replace username with the IU username you used to submit the job):
  •   qstat -u username
    

    For a full description of the qstat command and available options, see its manual page.

  • Use the Moab checkjob command; on the command line enter (replace job_id with the ID number assigned to your job):
  •   checkjob job_id
    

    For a full description of the checkjob command and available options, see its manual page.

Getting help

For more about running batch jobs on Big Red II, see How do I run batch jobs on Big Red II at IU?

Support for IU research computing systems, software, and services is provided by various UITS Research Technologies units. For help, see Research computing support at IU.

This is document betu in the Knowledge Base.
Last modified on 2017-06-21 15:24:29.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.