About Carbonate at Indiana University

On this page:


System overview

Carbonate (carbonate.uits.iu.edu) at Indiana University is a large memory computer cluster configured to support high-performance, data-intensive computing. Carbonate can handle computing tasks for researchers using genome assembly software, large-scale phylogenetic software, and other genome analysis applications that require large amounts of computer memory. Accounts are available to IU students, faculty, and staff. Carbonate also serves as a "condominium cluster" environment for IU researchers, research labs, departments, and schools.

Carbonate has 72 general-purpose compute nodes, each with 256 GB of RAM, and eight large-memory compute nodes, each with 512 GB of RAM. Each node is a Lenovo NeXtScale nx360 M5 server equipped with two 12-core Intel Xeon E5-2680 v3 CPUs and four 480 GB solid-state drives. Carbonate also features 12 GPU-accelerated Lenovo ThinkSystem SD530 deep learning (DL) nodes, each equipped with two Intel Xeon Gold 6126 12-core CPUs, two NVIDIA GPU accelerators (eight with Tesla P100s; four with Tesla V100s), four 1.92 TB solid-state drives, and 192 GB of RAM. All Carbonate nodes are housed in the IU Bloomington Data Center, run Red Hat Enterprise 7.x, and are connected to the IU Science DMZ via 10-gigabit Ethernet. The Data Capacitor II, DC-WAN2, Slate, and Slate-Project file systems are mounted for temporary storage of research data. The Modules environment management package allows users to dynamically customize their shell environments.

Besides being available to IU students, faculty, and staff for standard, cluster-based, high-throughput computing, Carbonate offers two alternative service models to the IU community:

  • Condominium computing: The condominium computing service model provides a way for IU schools, departments, and researchers to fund computational nodes for their own research purposes without shouldering the cost, overhead, and management requirements of purchasing individual systems. Condominium nodes are housed in the IU Bloomington Data Center, and are managed, backed up, and secured by UITS Research Technologies staff. Condominium nodes are available to "members" whenever they are needed, but when they are not in use, idle condominium nodes become available to other researchers and students on Carbonate. In this way, condominium computing promotes cost-effective expansion of IU's high-performance computing capabilities, enables efficient provisioning of computing resources to the entire IU research community, and helps conserve natural resources and energy.
  • Dedicated computing: The dedicated computing service model lets schools and departments host nodes that are dedicated solely to their use within Carbonates's physical, electrical, and network framework. This provides 24/7 access for school or departmental use, while leveraging the network and physical components of Carbonate, and the security and energy efficiency benefits provided by location within the IU Data Center.

To inquire about the condominium computing or dedicated computing service models on Carbonate, email the UITS High Performance Systems (HPS) team.

System access

IU students, faculty, and staff can request accounts on Carbonate by following the instructions in Get additional IU computing accounts.

Non-IU collaborators must have IU faculty sponsors. For details, see the Research system accounts (all campuses) section of Computing accounts at IU.

NSF-funded life sciences researchers can apply to the National Center for Genome Analysis Support (NCGAS) allocations committee to request accounts on Carbonate. To request an allocation, submit the NCGAS Allocations Request Form. If you have questions, email NCGAS.

Once your account is created, you can use any SSH2 client to access carbonate.uits.iu.edu. Log in with your IU username and passphrase, and then confirm your identity with Duo two-step login.

Notes:
  • To set up SSH public-key authentication, you must submit the "SSH public-key authentication to HPS systems" user agreement (log into HPC everywhere beta using your IU username and passphrase), in which you agree to set a passphrase on your private key when you generate your key pair.
  • For enhanced security, SSH connections that have been idle for 60 minutes will be disconnected. To protect your data from misuse, remember to log off or lock your computer whenever you leave it.
  • The scheduled monthly maintenance window for IU's high-performance computing systems is the second Sunday of each month, 7am-7pm.

HPC software

The Research Applications and Deep Learning (RADL) group, within the Research Technologies division of UITS, maintains and supports the high-performance computing (HPC) software on IU's research supercomputers. For a list of HPC packages available on each system, see HPC Applications.

For information about adding packages to your user environment, see Use Modules to manage your software environment on IU's research computing systems.

To request software, submit the HPC Software Request form.

Note:
Carbonate users are free to install software in their home directories. Faculty and staff may request the installation of software for use by all users on the system. If students require software packages on Carbonate, their advisors must request it for them. For details, see IU policies relative to installing software on Carbonate.

Set up your user environment

On the research computing resources at Indiana University, the Modules environment management system provides a convenient method for dynamically customizing your software environment.

For more about using Modules to configure your user environment, see Use Modules to manage your software environment on IU's research computing systems.

File storage options

Before storing data on this system, make sure you understand the information in the Work with data containing PHI section (below).

Work with data containing PHI

The Health Insurance Portability and Accountability Act of 1996 (HIPAA) established rules protecting the privacy and security of individually identifiable health information. The HIPAA Privacy Rule and Security Rule set national standards requiring organizations and individuals to implement certain administrative, physical, and technical safeguards to maintain the confidentiality, integrity, and availability of protected health information (PHI).

This UITS system or service meets certain requirements established in the HIPAA Security Rule thereby enabling its use for work involving data that contain protected health information (PHI). However, using this system or service does not fulfill your legal responsibilities for protecting the privacy and security of data that contain PHI. You may use this system or service for work involving data that contain PHI only if you institute additional administrative, physical, and technical safeguards that complement those UITS already has in place.

Note:
Although PHI is one type of Critical data, other types of institutional data classified as Critical are not permitted on Research Technologies systems. For help determining which institutional data elements classified as Critical are considered PHI, see About protected health information (PHI) data elements in the classifications of institutional data.

For more, see Your legal responsibilities for protecting data containing protected health information (PHI) when using UITS Research Technologies systems and services.

UITS provides consulting and online help for Indiana University researchers, faculty, and staff who need help securely processing, storing, and sharing data containing protected health information (PHI). If you have questions about managing HIPAA-regulated data at IU, contact UITS HIPAA Consulting. To learn more about properly ensuring the safe handling of PHI on UITS systems, see the UITS IT Training video Securing HIPAA Workflows on UITS Systems. For additional details about HIPAA compliance at IU, see HIPAA Privacy and Security Compliance

Run jobs on Carbonate

Note:

The information in this section pertains only to Carbonate's general-purpose and large-memory nodes. For information about running jobs on Carbonate's DL nodes, see:

Carbonate uses the TORQUE resource manager integrated with Moab Workload Manager to coordinate resource management and job scheduling.

Carbonate employs a default routing queue that funnels jobs, according to their resource requirements, into two execution queues configured to maximize job throughput and minimize wait times (the amount of time a job remains queued, waiting for required resources to become available). Depending on your job's resource requirements, the routing queue (BATCH) automatically places your job into the NORMAL or LARGEMEMORY queue:

  • NORMAL: Jobs requesting up to 251 GB of virtual memory
  • LARGEMEMORY: Jobs requesting from 251 GB up to 503 GB of virtual memory
Note:
To best meet the needs of all research projects affiliated with Indiana University, UITS Research Technologies administers the batch job queues on IU's research supercomputers using resource management and job scheduling policies that optimize the overall efficiency and performance of workloads on those systems. If the structure or configuration of the batch queues on any of IU's supercomputing systems does not meet the needs of your research project, contact UITS Research Technologies.

For details on how to submit, monitor, and delete jobs on Carbonate, plus additional queue information, see Run jobs on Carbonate.

Request single user time

Although UITS Research Technologies cannot provide dedicated access to an entire compute system during the course of normal operations, "single user time" is made available by request one day a month during each system's regularly scheduled maintenance window to accommodate IU researchers with tasks requiring dedicated access to an entire compute system. To request "single user time" or ask for more information, contact UITS Research Technologies.

Acknowledge grant support

The Indiana University cyberinfrastructure, managed by the Research Technologies division of UITS, is supported by funding from several grants, each of which requires you to acknowledge its support in all presentations and published works stemming from research it has helped to fund. Conscientious acknowledgment of support from past grants also enhances the chances of IU's research community securing funding from grants in the future. For the acknowledgment statement(s) required for scholarly printed works, web pages, talks, online publications, and other presentations that make use of this and/or other grant-funded systems at IU, see Sources of funding to acknowledge in published work if you use IU's research cyberinfrastructure

Get help

For an overview of Carbonate documentation, see Get started on Carbonate.

Support for IU research computing systems, software, and services is provided by various teams within the Research Technologies division of UITS.

For general questions about research computing at IU, contact UITS Research Technologies.

For more options, see Research computing support at IU.

This is document aolp in the Knowledge Base.
Last modified on 2019-09-19 15:34:57.

Contact us

For help or to comment, email the UITS Support Center.