On this page:

Carbonate (carbonate.uits.iu.edu) is Indiana University's replacement for Mason and is available to IU students, faculty, and staff. Carbonate fully supports high-performance, data-intensive computing, and provides an eighteenfold increase in processing capability over Mason. Carbonate can handle computing tasks for researchers using genome assembly software, large-scale phylogenetic software, and other genome analysis applications that require large amounts of computer memory.

Carbonate has 72 general-purpose compute nodes, each with 256 GB of RAM, and eight large-memory compute nodes, each with 512 GB of RAM. Each node is a Lenovo NeXtScale nx360 M5 server equipped with two 12-core Intel Xeon E5-2680 v3 CPUs and four 480 GB solid-state drives. All nodes are housed in the IU Bloomington Data Center, run Red Hat Enterprise 7.x, and are connected to the IU Science DMZ via 10-gigabit Ethernet.

Carbonate uses the TORQUE resource manager integrated with Moab Workload Manager to coordinate resource management and job scheduling. The Data Capacitor II and Data Capacitor Wide Area Network (DC-WAN) parallel file systems are mounted for temporary storage of research data. The Modules environment management package on allows users to dynamically customize their shell environments.

Back to top

Back to top

Back to top

On the research computing resources at Indiana University, the Modules environment management system provides a convenient method for dynamically customizing your software environment.

For more about using Modules to configure your user environment, see On the research computing systems at IU, how do I use Modules to manage my software environment?

Back to top

Back to top

Working with data containing PHI

This system meets certain requirements established in the HIPAA Security Rule that enable its use for research involving data that contain protected health information (PHI). You may use this resource for research involving data that contain PHI only if you institute additional administrative, physical, and technical safeguards that complement those UITS already has in place. For more, see When using UITS Research Technologies systems and services, what are my legal responsibilities for protecting the privacy and security of data containing protected health information (PHI)? If you need help or have questions, contact UITS HIPAA Consulting.

IU's research computing clusters use the TORQUE resource manager (based on OpenPBS) and the Moab Workload Manager to manage and schedule batch jobs. Moab uses fairshare scheduling to track usage and prioritize jobs.

Back to top

Carbonate employs a default routing queue that funnels jobs, according to their resource requirements, into two execution queues configured to maximize job throughput and minimize wait times (i.e., the amount of time a job remains queued, waiting for required resources to become available). Depending on the resource requirements specified in either your batch job script or your qsub command, the routing queue (BATCH) automatically places your job into the NORMAL or LARGEMEMORY queue:

• NORMAL queue: Jobs requesting up to 251 GB of virtual memory (qsub -l vmem=251gb)
• LARGEMEMORY queue: Jobs requesting from 251 GB up to 503 GB of virtual memory (qsub -l vmem=503gb)

You do not have to specify a queue in your job script or in your qsub command to submit your job to one of the two batch execution queues; your job will run in the NORMAL or LARGEMEMORY queue unless you specifically submit it to the DEBUG or INTERACTIVE queue, the properties of which are as follows:

• DEBUG: The DEBUG queue is intended for short, quick-turnaround test jobs requiring less than 1 hour of wall time.

Maximum wall time:	1 hour
Maximum nodes per job:	2
Maximum cores per job:	48
Maximum number of jobs per user:	2
Direct submission:	Yes

To submit a batch job to the DEBUG queue, either add the #PBS -q debug directive to your job script, or enter qsub -q debug on the command line.

Note:
For longer debugging or testing sessions, submit an interactive job to the INTERACTIVE queue instead.
• INTERACTIVE: Interactive jobs submitted to the INTERACTIVE queue should experience less wait time (i.e., start sooner) than interactive jobs submitted to the batch execution queues.

Maximum wall time:	8 hours
Maximum cores per job:	8
Maximum number of jobs per queue:	128
Maximum number of jobs per user:	2
Direct submission:	Yes

To submit an interactive job to the INTERACTIVE queue, on the command line, enter qsub with the -I and -q interactive options added; for example:

  qsub -I -q interactive -l nodes=1:ppn=1,walltime=4:00:00

Note:
If you enter qsub without the -q interactive option, your interactive job will be placed in the routing queue for submission to the NORMAL or LARGEMEMORY batch execution queue, which most likely will entail a longer wait time for your job.

Back to top

Although UITS Research Technologies cannot provide dedicated access to an entire compute system during the course of normal operations, "single user time" is made available by request one day a month during each system's regularly scheduled maintenance window to accommodate IU researchers with tasks requiring dedicated access to an entire compute system. To request such single user time, complete and submit the Research Technologies Ask RT for Help form, requesting to run jobs in single user time on HPS systems. If you have questions, email the HPS team.

Back to top

Support for IU research computing systems, software, and services is provided by various UITS Research Technologies units. For help, see Research computing support at IU.

Back to top