About Carbonate at Indiana University

• System overview
• System access
• Available software
• Set up your user environment
• File storage options
• Work with data containing PHI
• Run jobs on Carbonate
• Queue information
• Request single user time
• Acknowledge grant support
• Support

System overview

Carbonate has 72 general-purpose compute nodes, each with 256 GB of RAM, and eight large-memory compute nodes, each with 512 GB of RAM. Each node is a Lenovo NeXtScale nx360 M5 server equipped with two 12-core Intel Xeon E5-2680 v3 CPUs and four 480 GB solid-state drives. All nodes are housed in the IU Bloomington Data Center, run Red Hat Enterprise 7.x, and are connected to the IU Science DMZ via 10-gigabit Ethernet.

Carbonate uses the TORQUE resource manager integrated with Moab Workload Manager to coordinate resource management and job scheduling. The Data Capacitor II and Data Capacitor Wide Area Network (DC-WAN) parallel file systems are mounted for temporary storage of research data. The Modules environment management package on allows users to dynamically customize their shell environments.

Besides being available to IU students, faculty, and staff for standard, cluster-based, high-throughput computing, Carbonate offers two alternative service models to the IU community:

• Condominium computing: The condominium computing service model provides a way for IU schools, departments, and researchers to fund computational nodes for their own research purposes without shouldering the cost, overhead, and management requirements of purchasing individual systems. Condominium nodes are housed in the IU Bloomington Data Center, and are managed, backed up, and secured by UITS Research Technologies staff. Condominium nodes are available to "members" whenever they are needed, but when they are not in use, idle condominium nodes become available to other researchers and students on Carbonate. In this way, condominium computing promotes cost-effective expansion of IU's high-performance computing capabilities, enables efficient provisioning of computing resources to the entire IU research community, and helps conserve natural resources and energy.
• Dedicated computing: The dedicated computing service model lets schools and departments host nodes that are dedicated solely to their use within Carbonates's physical, electrical, and network framework. This provides 24/7 access for school or departmental use, while leveraging the network and physical components of Carbonate, and the security and energy efficiency benefits provided by location within the IU Data Center.

To inquire about the condominium computing or dedicated computing service models on Carbonate, email the UITS High Performance Systems (HPS) team.

System access

IU students, faculty, and staff can request accounts on Carbonate by following the instructions in Get additional IU computing accounts.

Non-IU collaborators must have IU faculty sponsors. For details, see the Research system accounts (all campuses) section of Computing accounts at IU.

NSF-funded life sciences researchers can apply to the National Center for Genome Analysis Support (NCGAS) allocations committee to request accounts on Carbonate. To request an allocation, submit the NCGAS Allocations Request Form. If you have questions, email NCGAS.

Once your account is created, you can use any SSH2 client to access carbonate.uits.iu.edu. Log in with your IU username and passphrase, and then confirm your identity with Duo two-step login.

Available software

For a list of packages available on Carbonate, see HPC Applications.

Set up your user environment

File storage options

Run jobs on Carbonate

User processes on the login nodes are limited to 20 minutes of CPU time. Processes on the login nodes that run longer than 20 minutes are terminated automatically (without warning). If your application requires more than 20 minutes of CPU time or has large memory requirements, submit a batch job or request an interactive session using the TORQUE qsub command.

Because of this limit:

• When running Java programs, add the -Xmx parameter (values must be multiples of 1,024 greater than 2 MB) on the command line to specify the Java Virtual Machine (JVM) maximum heap size. For example, to run a Java program (for example, Hello_DeathStar) with a maximum heap size of 640 MB , on the command line, enter:

    java -Xmx640m Hello_DeathStar

• Memory-intensive jobs started on the login nodes will be killed almost immediately. Submit debugging and testing jobs to the INTERACTIVE or DEBUG queue; for example:
  • INTERACTIVE queue:

     qsub -I -q interactive -l nodes=1:ppn=4,vmem=10gb,walltime=4:00:00

  • DEBUG queue:

     qsub -q debug -l nodes=1:ppn=4,vmem=32gb,walltime=00:30:00

Submit jobs

To submit a TORQUE job script (for example, job.script) on Carbonate, use the qsub command. If the command exits successfully, it will return a job ID; for example:

 [sgerrera@h1]$ qsub job.script
 123456.m1.Carbonate
 [sgerrera@h1]$

If your job has resource requirements that are different from the defaults (but not exceeding the maximums allowed), specify them either with TORQUE directives in your job script, or with the -l (a lower-case "L"; short for resource_list) option in your qsub command. For example:

• To submit a job (for example, job.script) that needs more than the default 60 minutes of walltime, use:

   qsub -l walltime=10:00:00 job.script

• To submit a job (for example, job.script) that needs more than the default 16 GB of virtual memory (for example, 200 GB), use:

   qsub -l nodes=1:ppn=4,vmem=200gb job.script

  If you don't provide a virtual memory resource (omit -l vmem=[n]gb), you will receive a warning, and the default (16 GB) virtual memory will be applied; for example:

   [ersojyn1@h1]$ qsub -l nodes=1:ppn=4 job.script
   warning:vmem resource not provided, default vmem of 16gb will be applied. See
   /etc/motd for details.
   36506.s1
   [ersojyn1@h1]$
  

On the command line, you can specify multiple attributes with either one -l switch followed by multiple comma-separated attributes, or multiple -l switches, one for each attribute. For example, to submit a job (for example, job.script) that requires 32 GB of virtual memory to run on 16 cores on one node, you may enter either of the following commands (they are equivalent):

 qsub -l nodes=1:ppn=16,vmem=32gb job.script
 qsub -l nodes=1:ppn=16 -l vmem=32gb job.script

If you need help determining how much memory your jobs are using, TORQUE will report this information when you add the -m e flag to qsub, or when you add the equivalent directive to your submit script:

 #PBS -m e

When you use the -m e parameter, TORQUE/PBS will send you an email message at job completion similar to the following:

 PBS Job Id: 16857.s1
 Job Name:   sleep
 Exec host:  c15/0-3
 Execution terminated
 Exit_status=0
 resources_used.cput=00:00:00
 resources_used.energy_used=0
 resources_used.mem=5464kb
 resources_used.vmem=470124kb
 resources_used.walltime=00:00:33

In the above example, resources_used.vmem=470124kb is the relevant line.

Useful qsub options include:

For more, see the qsub manual page.

Monitor jobs

To monitor the status of a queued or running job, use the TORQUE qstat command. Useful qstat options include:

For more, see the qstat manual page.

Delete jobs

To delete a queued or running job, use the qdel command.

For more, see the qdel manual page.

Queue information

Request single user time

Acknowledge grant support

Support

For an overview of Carbonate documentation, see Get started on Carbonate.