At IU, how do I use GROMACS on Big Red II?

On this page:


Overview

GROMACS (Groningen Machine for Chemical Simulations) is a full-featured molecular dynamics simulation suite designed primarily for studying proteins, lipids, and polymers, but useful for analyzing a wide variety of chemical and biological research questions, as well. Originally developed at the University of Groningen, GROMACS is open source software maintained by contributors at universities and research centers in Sweden, Germany, and the US. GROMACS is freely available under the GNU Lesser General Public License (LGPL). Serial and MPI versions are available with 64-bit addressing. Starting with version 4.6, GROMACS includes native, CUDA-based GPU acceleration support for NVIDIA Fermi and Kepler GPUs.

At Indiana University, GROMACS is installed on Big Red II. Single and double precision MPI versions are available on both systems; GPU-accelerated MPI versions are available on Big Red II.

Note: When working with GROMACS, you occasionally will need to modify Protein Data Bank (PDB) files. UITS recommends using DeepView - Swiss-PdbViewer, an application for viewing and editing PDB files available for free download from the Swiss Institute of Bioinformatics (SIB) via its Expert Protein Analysis System (ExPASy) Bioinformatics Resource Portal.

For more information about the availability of software on the Indiana University shared central systems, see At IU, what software is available on the research computing systems?

Using GROMACS on Big Red II

Big Red II has single precision and double precision versions of GROMACS for running parallel jobs on compute nodes in Big Red II's Extreme Scalability Mode (ESM) execution environment. GPU-accelerated versions also are available for running parallel jobs on the hybrid CPU/GPU nodes. The GPU-accelerated versions routinely run twice as fast as the CPU versions. To take advantage of GPU acceleration you must use a GPU-accelerated version of GROMACS, include the NVIDIA CUDA Toolkit in your user environment, and specify the GPU queue in your TORQUE script.

Setting up your user environment

On the research computing resources at Indiana University, the Modules environment management system provides a convenient method for dynamically customizing your software environment.

To add GROMACS to your Big Red II user environment, you first must add the GNU programming environment and the FFTW (Fastest Fourier Transforms in the West) subroutine library. To use a GPU-accelerated version of GROMACS, you also must add the NVIDIA CUDA Toolkit.

Note: Although the GROMACS documentation directs you to edit your shell's login script to specify the location of the GROMACS commands and the GMXRC script (which identifies the shell you're running and sets GROMACS environment variables accordingly), on Big Red II all necessary user environment changes are handled automatically when you load any of the available GROMACS modules.

Use Module commands to add the necessary packages to your user environment:

  1. Verify that the PrgEnv-gnu module is added to your user environment; on the command line, enter module list to determine which modules are currently loaded.
  2. If another programming environment module (e.g., PrgEnv-cray) is loaded, use the module swap command to replace it with the PrgEnv-gnu module:

      module swap PrgEnv-cray PrgEnv-gnu
    
  3. If the FFTW library is not listed among the currently loaded modules, load the default fftw module; on the command line, enter:
  4.   module load fftw
    
  5. Check which versions of GROMACS are available; on the command line, enter:
  6.   module avail gromacs
    

    Note: Either of the single precision or double precision versions are suitable for running GROMACS jobs on Big Red II's compute nodes. Although these versions will run on the hybrid CPU/GPU nodes, they cannot take advantage of GPU acceleration. To run GPU-accelerated jobs on Big Red II's hybrid CPU/GPU nodes, choose a GPU-accelerated version.

  7. To load the version of your choice, on the command line, enter:
  8.   module load gromacs/version_path
    

    Replace version_path with the path corresponding to version you want to load; for example:

    • To load the double precision version, enter:
    •   module load gromacs/gnu/double/4.6.2
      
    • To load the GPU-accelerated version, enter:
    •   module load cudatoolkit
        module load gromacs/gnu/gpu/4.6.5
      

To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

For example, to automatically load upon login the modules needed to run a GPU-accelerated GROMACS job, add the following lines to your ~/.modules file:

  module swap PrgEnv-cray PrgEnv-gnu 
  module load fftw 
  module load cudatoolkit 
  module load gromacs/gnu/gpu/4.6.5 

For more about using Modules to configure your user environment, see On the research computing systems at IU, how do I use Modules to manage my software environment?

Preparing a batch job script for the CPU/GPU nodes

To run a parallel GROMACS job in batch mode on the hybrid CPU/GPU nodes in Big Red II's native ESM execution environment, prepare a TORQUE job script (e.g., ~/work_directory/my_job_script.pbs) that specifies the application you want to run, sets the resource requirements and other parameters appropriate for your job, and invokes the aprun command to properly launch your application.

On Big Red II:

  • To invoke the MPI version of a GROMACS command, append the _mpi suffix to its name (e.g., mdrun_mpi).
  • To invoke the double precision MPI version of a GROMACS command, append the _mpi_d suffix to its name (e.g., mdrun_mpi_d).

The following sample script executes the MPI version of mdrun on four Big Red II CPU/GPU nodes in the ESM execution environment:

  #!/bin/bash
  
  #PBS -l nodes=4:ppn=16,walltime=3:00:00
  #PBS -q gpu
  #PBS -o out.log
  #PBS -e err.log

  cd $PBS_O_WORKDIR  
  aprun -n 4 -N 1 mdrun_mpi <mdrun_mpi_options>

In the sample script above:

  • The -q gpu TORQUE directive routes the job to the gpu queue.
  • The cd $PBS_O_WORKDIR line changes to the directory from which the job was submitted and where the input files are.
  • If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
  • When invoking aprun on the CPU/GPU nodes, the -n argument specifies the total number of nodes (not the total number of processing elements), and the -N argument specifies the number of GPUs per node, which is one (e.g., -N 1).
  • Replace <mdrun_mpi_options> with mdrun_mpi options indicating your input and output files, and other parameters for controlling your simulation; for descriptions of available options, see the mdrun manual page (man mdrun).

Performance benefits of using GPU-accelerated GROMACS

The following graph compares the results of test simulations run using a CPU version of GROMACS with results of identical simulations run using a GPU-accelerated version. The test simulation involved more than 14 million atoms. Each run had identical simulation parameters and runtimes, and all simulations were performed on the hybrid CPU/GPU nodes. The data values reported represent the number of time steps evolved during the simulations (larger values indicate better performance). On each run, the GPU-accelerated version out-performed the CPU version by a significant margin.

The graph shows the performance improvements you can expect running GPU-accelerated GROMACS.

Preparing a batch job script for the compute nodes

To run a parallel GROMACS job in batch mode on the compute nodes in Big Red II's native ESM execution environment, prepare a TORQUE job script (e.g., ~/work_directory/my_job_script.pbs) that specifies the application you want to run, sets the resource requirements and other parameters appropriate for your job, and invokes the aprun command to properly launch your application.

On Big Red II:

  • To invoke the MPI version of a GROMACS command, append the _mpi suffix to its name (e.g., mdrun_mpi).
  • To invoke the double precision MPI version of a GROMACS command, append the _mpi_d suffix to its name (e.g., mdrun_mpi_d).

The following sample script executes the MPI version of mdrun on two Big Red II compute nodes in the ESM execution environment:

  #!/bin/bash 
  
  #PBS -l nodes=2:ppn=32,walltime=6:00:00
  #PBS -q cpu
  #PBS -o out.log
  #PBS -e err.log

  cd $PBS_O_WORKDIR
  aprun -n 64 mdrun_mpi <mdrun_mpi_options>

In the sample script above:

  • The -q cpu TORQUE directive routes the job to the cpu queue.
  • The cd $PBS_O_WORKDIR line changes to the directory from which the job was submitted and where the input files are.
  • If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
  • Replace <mdrun_mpi_options> with mdrun_mpi options indicating your input and output files, and other parameters for controlling your simulation; for descriptions of available options, see the mdrun manual page (man mdrun).

Submitting and monitoring your job

To submit your job script (e.g., ~/work_directory/my_job_script.pbs), use the TORQUE qsub command; for example, on the command line, enter:

  qsub [options] ~/work_directory/my_job_script.pbs

For a full description of the qsub command and available options, see its manual page.

To monitor the status of your job, use any of the following methods:

  • Use the TORQUE qstat command; on the command line, enter (replace username with the IU username you used to submit the job):
  •   qstat -u username
    

    For a full description of the qstat command and available options, see its manual page.

  • Use the Moab checkjob command; on the command line enter (replace job_id with the ID number assigned to your job):
  •   checkjob job_id
    

    For a full description of the checkjob command and available options, see its manual page.

Getting help

For more about running batch jobs on Big Red II, see How do I run batch jobs on Big Red II at IU?

For more about using TORQUE to submit batch jobs on high-performance computing clusters, see What is TORQUE, and how do I use it to submit and manage jobs on high-performance computing systems?

To learn more about GROMACS:

If you need help or have questions about running GROMACS jobs on Big Red II, contact the UITS Scientific Applications and Performance Tuning (SciAPT) team. If you have system-specific questions about Big Red II, or if you encounter problems running batch jobs, contact the UITS High Performance Systems group.

This is document besc in the Knowledge Base.
Last modified on 2017-07-25 09:08:02.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.