Use GROMACS on Big Red II at IU

On this page:

Note:

Big Red II will be retired from service on December 15, 2019. After that date, you will no longer be able to log into Big Red II; however, the data in your Big Red II home directory will remain accessible from your home directory on any of the other IU research supercomputers. New software requests for Big Red II will no longer be accepted after the October 13, 2019, maintenance window.

Beginning October 14, 2019, IU graduate students, faculty, and staff will be able to create accounts on Big Red III. Undergraduate students and affiliates will be able to get accounts if they are sponsored by full-time IU faculty or staff members. Grand Challenge users who create Big Red III accounts will be able to request exclusive access to a portion of the system for running jobs.

For more, see Upcoming changes to research supercomputers at IU.


Overview

GROMACS (Groningen Machine for Chemical Simulations) is a full-featured molecular dynamics simulation suite designed primarily for studying proteins, lipids, and polymers, but useful for analyzing a wide variety of chemical and biological research questions, as well. Originally developed at the University of Groningen, GROMACS is open source software maintained by contributors at universities and research centers in Sweden, Germany, and the US. GROMACS is freely available under the GNU Lesser General Public License (LGPL). Serial and MPI versions are available with 64-bit addressing. Starting with version 4.6, GROMACS includes native, CUDA-based GPU acceleration support for NVIDIA Fermi and Kepler GPUs.

At Indiana University, Big Red II has single precision and double precision versions of GROMACS for running parallel jobs on compute nodes in the Extreme Scalability Mode (ESM) execution environment. GPU-accelerated versions also are available for running parallel jobs on Big Red II's hybrid CPU/GPU nodes. The GPU-accelerated versions routinely run twice as fast as the CPU versions. To take advantage of GPU acceleration you must use a GPU-accelerated version of GROMACS, include the NVIDIA CUDA Toolkit in your user environment, and specify the GPU queue in your job script.

Note:
When working with GROMACS, you occasionally will need to modify Protein Data Bank (PDB) files. UITS recommends using DeepView - Swiss-PdbViewer, an application for viewing and editing PDB files available for free download from the Swiss Institute of Bioinformatics (SIB) via its Expert Protein Analysis System (ExPASy) Bioinformatics Resource Portal.

Set up your user environment

To add GROMACS to your Big Red II user environment, you first must add the GNU programming environment and the FFTW (Fastest Fourier Transforms in the West) subroutine library. To use a GPU-accelerated version of GROMACS, you also must add the NVIDIA CUDA Toolkit.

Note:
Although the GROMACS documentation directs you to edit your shell's login script to specify the location of the GROMACS commands and the GMXRC script (which identifies the shell you're running and sets GROMACS environment variables accordingly), on Big Red II all necessary user environment changes are handled automatically when you load any of the available GROMACS modules.

Use Module commands to add the necessary packages to your user environment:

  1. Verify that the PrgEnv-gnu module is added to your user environment; on the command line, enter module list to determine which modules are currently loaded.

    If another programming environment module (for example, PrgEnv-cray) is loaded, use the module swap command to replace it with the PrgEnv-gnu module:

    module swap PrgEnv-cray PrgEnv-gnu
    
  2. If the FFTW library is not listed among the currently loaded modules, load the default fftw module; on the command line, enter:
    module load fftw
    
  3. Check which versions of GROMACS are available; on the command line, enter:
    module avail gromacs
    
    Note:
    Either of the single precision or double precision versions are suitable for running GROMACS jobs on Big Red II's compute nodes. Although these versions will run on the hybrid CPU/GPU nodes, they cannot take advantage of GPU acceleration. To run GPU-accelerated jobs on Big Red II's hybrid CPU/GPU nodes, choose a GPU-accelerated version.
  4. To load the version of your choice, on the command line, enter:
    module load gromacs/version_path
    

    Replace version_path with the path corresponding to version you want to load; for example:

    • To load the double precision version, enter:
      module load gromacs/gnu/double/4.6.2
      
    • To load the GPU-accelerated version, enter:
      module load cudatoolkit
      module load gromacs/gnu/gpu/4.6.5
      

To make permanent changes to your environment, edit your ~/.modules file. For more, see Use a .modules file in your home directory to save your user environment on an IU research supercomputer.

For example, to automatically load upon login the modules needed to run a GPU-accelerated GROMACS job, add the following lines to your ~/.modules file:

module swap PrgEnv-cray PrgEnv-gnu
module load fftw
module load cudatoolkit
module load gromacs/gnu/gpu/4.6.5

For more about using Modules to configure your user environment, see Use Modules to manage your software environment on IU's research computing systems.

Prepare a GPU-accelerated batch job script

To run a parallel GROMACS job in batch mode on the hybrid CPU/GPU nodes in Big Red II's native ESM execution environment, prepare a TORQUE job script (for example, ~/work_directory/my_job_script.pbs) that specifies the application you want to run, sets the resource requirements and other parameters appropriate for your job, and invokes the aprun command to properly launch your application.

On Big Red II:

  • To invoke the MPI version of a GROMACS command, append the _mpi suffix to its name (for example, mdrun_mpi).
  • To invoke the double precision MPI version of a GROMACS command, append the _mpi_d suffix to its name (for example, mdrun_mpi_d).

The following sample script executes the MPI version of mdrun on four Big Red II CPU/GPU nodes in the ESM execution environment:

#!/bin/bash

#PBS -l nodes=4:ppn=16,walltime=3:00:00
#PBS -q gpu
#PBS -o out.log
#PBS -e err.log

cd $PBS_O_WORKDIR
aprun -n 4 -N 1 mdrun_mpi <mdrun_mpi_options>

In the sample script above:

  • The -q gpu TORQUE directive routes the job to the gpu queue.
  • The cd $PBS_O_WORKDIR line changes to the directory from which the job was submitted and where the input files are.
  • If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
  • When invoking aprun on the CPU/GPU nodes, the -n argument specifies the total number of nodes (not the total number of processing elements), and the -N argument specifies the number of GPUs per node, which is one (for example, -N 1).
  • Replace <mdrun_mpi_options> with mdrun_mpi options indicating your input and output files, and other parameters for controlling your simulation; for descriptions of available options, see the mdrun manual page (man mdrun).

Performance benefits of using GPU-accelerated GROMACS

The following graph compares the results of test simulations run using a CPU version of GROMACS with results of identical simulations run using a GPU-accelerated version. The test simulation involved more than 14 million atoms. Each run had identical simulation parameters and runtimes, and all simulations were performed on the hybrid CPU/GPU nodes. The data values reported represent the number of time steps evolved during the simulations (larger values indicate better performance). On each run, the GPU-accelerated version out-performed the CPU version by a significant margin.

The graph shows the performance improvements you can expect running GPU-accelerated GROMACS.

Prepare a CPU-only batch job script

To run a parallel GROMACS job in batch mode on the compute nodes in Big Red II's native ESM execution environment, prepare a TORQUE job script (for example, ~/work_directory/my_job_script.pbs) that specifies the application you want to run, sets the resource requirements and other parameters appropriate for your job, and invokes the aprun command to properly launch your application.

On Big Red II:

  • To invoke the MPI version of a GROMACS command, append the _mpi suffix to its name (for example, mdrun_mpi).
  • To invoke the double precision MPI version of a GROMACS command, append the _mpi_d suffix to its name (for example, mdrun_mpi_d).

The following sample script executes the MPI version of mdrun on two Big Red II compute nodes in the ESM execution environment:

#!/bin/bash 

#PBS -l nodes=2:ppn=32,walltime=6:00:00
#PBS -q cpu
#PBS -o out.log
#PBS -e err.log

cd $PBS_O_WORKDIR
aprun -n 64 mdrun_mpi <mdrun_mpi_options>

In the sample script above:

  • The -q cpu TORQUE directive routes the job to the cpu queue.
  • The cd $PBS_O_WORKDIR line changes to the directory from which the job was submitted and where the input files are.
  • If you did not previously add the required modules to your user environment (as described in the preceding section), you must include the necessary module load commands in your script before invoking the aprun command.
  • Replace <mdrun_mpi_options> with mdrun_mpi options indicating your input and output files, and other parameters for controlling your simulation; for descriptions of available options, see the mdrun manual page (man mdrun).

Submit and monitor your job

To submit your job script (e.g., ~/work_directory/my_job_script.pbs), use the TORQUE qsub command; for example, on the command line, enter:

  qsub [options] ~/work_directory/my_job_script.pbs

For a full description of the qsub command and available options, see its manual page.

To monitor the status of your job, use any of the following methods:

  • Use the TORQUE qstat command; on the command line, enter (replace username with the IU username you used to submit the job):
      qstat -u username
    

    For a full description of the qstat command and available options, see its manual page.

  • Use the Moab checkjob command; on the command line enter (replace job_id with the ID number assigned to your job):
      checkjob job_id
    

    For a full description of the checkjob command and available options, see its manual page.

Get help

To learn more about GROMACS:

If you need help or have questions about running GROMACS jobs on Big Red II, contact the UITS Research Applications and Deep Learning team. If you have system-specific questions about Big Red II, or if you encounter problems running batch jobs, contact the UITS High Performance Systems group.

Related documents

This is document besc in the Knowledge Base.
Last modified on 2019-08-01 13:03:12.

Contact us

For help or to comment, email the UITS Support Center.