About Carbonate's deep learning (DL) and GPU partitions

Below is an overview of the DL and GPU partitions:

The Carbonate DL partition is intended specifically for use by users with deep learning workloads, while the GPU partition is intended for use by users with any workloads that can benefit from GPUs. IU students, faculty, and staff can request accounts on the Carbonate DL and GPU partitions by following these steps:

1. If you don't already have an account on Carbonate, request one using the instructions in Get additional IU computing accounts.
2. Use IU HPC Projects to request access to either Carbonate Deep Learning Partition or Carbonate GPU Partition; for instructions, see Request and manage access to specialized high performance computing (HPC) resources.

The IU research supercomputers use module-based environment management systems that provide a convenient method for dynamically customizing your software environment. Carbonate uses the Modules module management system. For more, see Use modules to manage your software environment on IU research supercomputers.

Popular deep learning tools (including TensorFlow, scikit-learn, NumPy, SciPy, NLTK, Torch, and MXNet) are bundled together in deeplearning modules. Several versions of the deeplearning module are available on Carbonate:

• To list available versions, on the command line, enter:

  module avail deeplearning
  

• To see which packages are included in a version of the deeplearning module, on the command line, enter (replace x.x.x with the module's version number):

  module show deeplearning/x.x.x
  

• To add a deeplearning module to your user environment, on the command line, enter (replace x.x.x with the module's version number):

  module load deeplearning/x.x.x

Individual modules for GPU-capable applications also are available on Carbonate. Modules for GPU-capable applications will include gpu in the module name.

For a list of all packages available on Carbonate, see HPC Applications.

Carbonate's DL and GPU partitions use the Slurm workload manager to manage and schedule interactive sessions and batch jobs.

Carbonate DL partition

To use the Carbonate DL partition, you must:

• Specify the dl or dl-debug partition by including the -p flag either as an SBATCH directive in your batch job script or as an option in your srun command.
• Indicate the type of GPU (p100 or v100) and the number of GPUs per node (1 or 2) that should be allocated to your job by including the --gpus-per-node flag either as an SBATCH directive in your batch job script or as an option in your srun command.

Carbonate GPU partition

To use the Carbonate GPU partition, you must:

• Specify the gpu or gpu-debug partition by including the -p flag either as an SBATCH directive in your batch job script or as an option in your srun command.
• Indicate the number of GPUs per node (up to 4) that should be allocated to your job by including the --gpus-per-node flag either as an SBATCH directive in your batch job script or as an option in your srun command.

To request resources for an interactive job, use the srun command with the --pty option.

For example, to launch a Bash session that uses one V100 GPU on a node in the gpu partition, on the command line, enter:

srun -p gpu --gpus-per-node v100:1 --pty bash

To launch a Bash session that uses one V100 GPU on a node in the dl partition, on the command line, enter:

srun -p dl --gpus-per-node v100:1 --pty bash

To perform GPU debugging, submit an interactive job to the dl-debug or dl partition:

• To request 12 cores and one P100 GPU for four hours of wall time in the dl-debug partition, on the command line, enter:

  srun -p dl-debug -N 1 --ntasks-per-node=12 --gpus-per-node p100:1 --time=4:00:00 --pty bash
  

• To request 12 cores and one P100 GPU for 12 hours of wall time in the dl partition, on the command line, enter:

  srun -p dl -N 1 --ntasks-per-node=12 --gpus-per-node p100:1 --time=12:00:00 --pty bash
  

When the requested resources are allocated to your job, you will be placed at the command prompt on one of Carbonate's DL nodes. When you are finished with your interactive session, on the command line, enter exit to free the allocated resources.

For complete documentation about the srun command, see the srun manual page (on the web, see srun; on Carbonate, enter man srun).

To view current information about Carbonate's DL and GPU partitions, use the sinfo command. You should see output similar to the following:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
dl           up 2-00:00:00     11   idle dl[1-9,11-12]
dl-debug     up    8:00:00      1   idle dl10
gpu		up	2-00:00:00  22 idle g[1-22]
gpu-debug	up	1:00:00  	2  idle	g[23-24]

In the above sample output:

• The PARTITION column shows the names of the available partitions.
• The AVAIL column shows the status of each partition.
• The TIMELIMIT column shows the maximum wall time that users can request.
• The NODES column shows the number of nodes in each partition.
• The STATE column shows the current status of each partition.
• The NODELIST column shows the actual nodes that are part of each partition.

To submit a batch job to the dl partition, add the #SBATCH -p dl directive to your job script. To submit a batch job to the dl-debug partition, add the #SBATCH -p dl-debug directive to your job script.