Below is an overview of the DL and GPU partitions:

The Carbonate DL partition is intended specifically for use by users with deep learning workloads, while the GPU partition is intended for use by users with any workloads that can benefit from GPUs. IU students, faculty, and staff can request accounts on the Carbonate DL and GPU partitions by following these steps:

1. If you don't already have an account on Carbonate, request one using the instructions in Get additional IU computing accounts.
2. Use IU HPC Projects to request access to either Carbonate Deep Learning Partition or Carbonate GPU Partition; for instructions, see Request and manage access to specialized high performance computing (HPC) resources.

For a list of packages available on Carbonate, see HPC Applications.

Popular DL software has been bundled together on Carbonate via the "deep-learning" modules. Significant versions include:

• deeplearning/1.13.1 (based on Python 3 and TensorFlow 1.13.1)
• deeplearning/2.3.0 (based on Python 3 and TensorFlow 2.3.0)

Both modules contain frequently used deep learning tools including scikit-learn, NumPy, SciPy, NLTK, Torch, Caffe2, and MXNet. The deeplearning/1.13.1 and deeplearning/2.3.0 modules also include Keras.

Otherwise, individual modules are available for other GPU applications. The modules for applications that can take advantage of GPUs will be in the format xyz/gpu/version (for example, lammps/gpu/gnu/3Mar20).

Carbonate's DL and GPU partitions use the Slurm workload manager to manage and schedule interactive sessions and batch jobs.

Carbonate DL partition

To use the Carbonate DL partition, you must:

• Specify the dl or dl-debug partition by including the -p flag either as an SBATCH directive in your batch job script or as an option in your srun command.
• Indicate the type of GPU (p100 or v100) and the number of GPUs per node (1 or 2) that should be allocated to your job by including the --gpus-per-node flag either as an SBATCH directive in your batch job script or as an option in your srun command.

Carbonate GPU partition

To use the Carbonate GPU partition, you must:

• Specify the gpu or gpu-debug partition by including the -p flag either as an SBATCH directive in your batch job script or as an option in your srun command.
• Indicate the number of GPUs per node (up to 4) that should be allocated to your job by including the --gpus-per-node flag either as an SBATCH directive in your batch job script or as an option in your srun command.

To request resources for an interactive job, use the srun command with the --pty option.

For example, to launch a Bash session that uses one V100 GPU on a node in the gpu partition, on the command line, enter:

srun -p gpu --gpus-per-node v100:1 --pty bash

To launch a Bash session that uses one V100 GPU on a node in the dl partition, on the command line, enter:

srun -p dl --gpus-per-node v100:1 --pty bash

To perform GPU debugging, submit an interactive job to the dl-debug or dl partition:

• To request 12 cores and one P100 GPU for four hours of wall time in the dl-debug partition, on the command line, enter:

  srun -p dl-debug -N 1 --ntasks-per-node=12 --gpus-per-node p100:1 --time=4:00:00 --pty bash
  

• To request 12 cores and one P100 GPU for 12 hours of wall time in the dl partition, on the command line, enter:

  srun -p dl -N 1 --ntasks-per-node=12 --gpus-per-node p100:1 --time=12:00:00 --pty bash
  

When the requested resources are allocated to your job, you will be placed at the command prompt on one of Carbonate's DL nodes. When you are finished with your interactive session, on the command line, enter exit to free the allocated resources.

For complete documentation about the srun command, see the srun manual page (on the web, see srun; on Carbonate, enter man srun).

To view current information about Carbonate's DL and GPU partitions, use the sinfo command. You should see output similar to the following:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
dl           up 2-00:00:00     11   idle dl[1-9,11-12]
dl-debug     up    8:00:00      1   idle dl10
gpu		up	2-00:00:00  22 idle g[1-22]
gpu-debug	up	1:00:00  	2  idle	g[23-24]

In the above sample output:

• The PARTITION column shows the names of the available partitions.
• The AVAIL column shows the status of each partition.
• The TIMELIMIT column shows the maximum wall time that users can request.
• The NODES column shows the number of nodes in each partition.
• The STATE column shows the current status of each partition.
• The NODELIST column shows the actual nodes that are part of each partition.

To submit a batch job to the dl partition, add the #SBATCH -p=dl directive to your job script. To submit a batch job to the dl-debug partition, add the #SBATCH -p=dl-debug directive to your job script.