Available access to allocated and short-term storage capacity on IU's research systems

On this page:


Allocated storage and its uses

Important:
Before storing data on any of Indiana University's research computing or storage systems, make sure you understand the information in Types of sensitive institutional data appropriate for the research supercomputers and the RDC at IU.

Indiana University provides multiple storage options to store, share, and collaborate on files on IU research computing resources. As a member of the IU research community, you have access to allocated storage on the following Research Technologies systems.

Allocated resource Description Default capacity Default file limit
Home directory space on IU research supercomputers
  • Mounted on IU's research supercomputers
  • Allocated with research supercomputer accounts
  • Disk-based, replicated, and highly secure
  • Relatively slow I/O throughput
  • For more, see below.
100 GB 800,000 files
Scholarly Data Archive (SDA)
  • HPSS data archive
  • Tape-based, long-term storage
  • Allocations are available to IU graduate students, faculty, and staff (undergraduate students and sponsored affiliates need either faculty or staff sponsors to request accounts)
  • For more, see below.
50 TB 25,000 files
Slate
  • Persistent storage for scholarly data
  • Lustre-based, high-throughput, parallel file system
  • Disk-based, replicated, and highly secure
  • No purge policy
  • Allocations are available to all IU research supercomputer users
  • For more, see below.
800 GB 6.4 million objects
Slate-Project
  • Centralized project space supporting extreme, data-intensive, performance-demanding workflows running on IU research supercomputers
  • Lustre-based, high-throughput, parallel file system
  • Disk-based, replicated, and highly secure
  • No purge policy
  • Allocated with approval from UITS High Performance File Systems (HPFS) allocations committee
  • For more, see below.
Up to 15 TB No file limit
Geode-Project
  • Fee-based project space
  • Disk-based, replicated, and highly secure
  • Mounted on IU research supercomputers
  • Mountable/accessible from campus computers
  • A memo of understanding (MOU) with UITS Research Technologies is required
  • For more, see below.
No default (depends on MOU) 800,000 files per every 100 GB (up to a maximum of 80 million for a "standard" allocation)

Home directories

By default, you are allotted 100 GB of home directory disk storage (with a maximum file limit of 800,000 files) when you create an account on any of IU's research supercomputers.

To request an account on an Indiana University research system, see Get additional IU computing accounts. Account availability depends on your eligibility.

Home directories are neither intended for storing large data sets nor capable of handling data-intensive computational I/O; other Research Technologies systems are provided for those purposes. Home directories are relatively small by design, as they are intended for low-capacity, low-performance uses (such as storing parameter or configuration files and compiling binaries) only.

When in support of research activities, an increase in capacity (along with a proportional increase in the file limit) may be granted. If you need additional home directory disk space, and other allocated storage options do not meet your needs, email the UITS Research Storage team.

Scholarly Data Archive

An account on IU's Scholarly Data Archive (SDA) provides up to 50 TB of long-term, tape-based, fully-replicated archival storage. A maximum file limit of 25,000 files is enforced for new accounts.

The Indiana University Scholarly Data Archive (SDA) provides extensive capacity (approximately 79 PB of tape overall) for storing and accessing research data. The SDA is a distributed storage service co-located at IU data centers in Bloomington and Indianapolis, providing IU researchers with large-scale archival or near-line data storage, arranged in large files, with two copies of data made by default (for disaster recovery).

For eligibility requirements, see the "Research system accounts (all campuses)" section in Computing accounts at IU.

  • To request an individual (personal) account on the Indiana University Scholarly Data Archive (SDA), follow the instructions in Get additional IU computing accounts.

    If you are eligible to request an SDA account, but it is not listed among the accounts available to request, contact your campus Support Center for help.

  • To request to have an SDA account created for your IU group account, contact the Research Storage team.
    Note:
    In accordance with standards for access control mandated by the HIPAA Security Rule, you are not permitted to access data containing protected health information (PHI) using a group (or departmental) account. To ensure accountability and maintain appropriate levels of access control, all users must use an individual login for all work involving PHI.

After submitting your account request, UITS will notify you via email when your account is ready for use.

When in support of research activities, extensions beyond 50 TB may be granted for a nominal charge; for more, see Fee-based research storage below.

Slate

The default quota allotment is 800 GB per user. Upon request, your quota may be increased to a maximum of 1.6 TB. To request more space on Slate, contact the UITS High Performance File Systems (HPFS) team. Additionally, an inode quota (sometimes called "file quota") limits the number of objects a single user can create to 6.4 million.

Slate is a centralized, high-performance Lustre file system designed for the persistent storage of scholarly data to meet the needs of data-intensive workflows and analytics running on Indiana University's research supercomputers. Space on Slate is not subject to a purge policy. For more, see About Slate high-performance storage for research computation at IU.

Space on Slate is available to all IU research supercomputer users. To create a Slate account, follow the instructions in Get additional IU computing accounts.

Slate-Project

Slate-Project allocations are available to IU researchers who need shared/project spaces or more storage capacity than the 1.6 TB available per user on Slate. Requests for fewer than 15 TB are granted without fee.

The Slate-Project high-performance file system is a centralized storage environment supporting extreme, data-intensive, performance-demanding (Big Data) workflows running on IU's research supercomputers. For more, see About Slate-Project high-performance project space at IU.

To get an allocation on the Slate-Project file system, fill out and submit the Slate-Project request form. Once your request is approved, you will be able to create a Slate-Project account using the instructions in Get additional IU computing accounts.

Slate-Project allocations of 15 TB or more are available from UITS Research Technologies as a direct-bill service; for more, see Fee-based research storage below.

Geode-Project

Geode-Project is a fee-based Research Technologies service providing disk-based persistent storage allocations to research projects using Indiana University's research supercomputers. Geode-Project allocations are hosted on Geode, which is co-located at the IU data centers in Bloomington and Indianapolis. Files stored in Geode-Project space are replicated, by default, at each data center. Project space users can access Geode-Project directly from all IU research supercomputers and remotely from personal workstations connected to the IU campus network.

Geode-Project allocations require a memo of understanding (MOU) with UITS Research Technologies; for more, see Fee-based research storage below.

Unallocated temporary storage and its uses

As a member of the IU research community, you have access to unallocated temporary storage in local and shared scratch space managed by UITS Research Technologies. Scratch space is intended only for temporary storage of computational data. Data are not backed up. Files are regularly purged once they reach a certain age, and cannot be recovered. Users are responsible for moving data they want to keep to permanent storage solutions.

Local scratch space

The following amounts of local scratch space are accessible to users with accounts on IU research supercomputers:

System Local scratch space available
Carbonate 1.3 TB accessible at /tmp

Files in local scratch directories are automatically deleted once they are 14 days old; purged files cannot be recovered.

Fee-based research storage

Schools, departments, labs, and individual investigators with storage needs beyond those met by baseline (no fee) services may arrange direct-bill agreements with UITS Research Technologies for fee-based storage on Research Technologies systems:

To find fee information for Research Technologies services, see the "IU Internal Research Technologies Services" section of UITS Rates for Direct-Bill Services. If you are interested in fee-based storage services, contact UITS Research Technologies to request a meeting.

Access your allocated research storage space(s)

Home directories

Provided you have the required account(s), you can access your home directory space from any of IU's research supercomputers (in the following examples, replace username with the appropriate individual or group IU username):

System Home directory path
Big Red 3 /N/u/username/BigRed3
Carbonate /N/u/username/Carbonate
Quartz /N/u/username/Quartz

From your home directory space on one system, you can access your home directory space on another (provided you have accounts on both systems). For the following examples, assume you (username) have accounts on Big Red 3 and Carbonate:

  • To access your home directory on Carbonate from your home directory on Big Red 3, on the command line, enter :
    cd ../Carbonate
    

    As a result, the command line changes to reflect your present location within the file system:

    username@login1:/N/u/username/Carbonate>
    
  • To access a subdirectory on Carbonate (for example, ~/work_dir) from a subdirectory on Big Red 3 (for example, ~/working_directory), on the command line, enter:
    cd ../../Carbonate/work_dir
    

    As a result, the command line changes to reflect your present location within the file system:

    username@elogin2:/N/u/username/Carbonate/work_dir>
    

Slate accounts

Your Slate account is accessible from each IU research supercomputer at /N/slate/username (replace username with your IU username).

Slate-Project space

Your Slate-Project space is accessible from each IU research supercomputer at /N/project/project_name (replace project_name with the name of your project).

SDA accounts

Methods available for transferring data to and from the Indiana University Scholarly Data Archive (SDA) include secure FTP (SFTP), secure copy (SCP), GridFTP (via the IU Globus Web App), and Hierarchical Storage Interface (HSI). For instructions, see:

Read-only access is available via CIFS/Samba; see Use Samba or CIFS to access your SDA account from your personal workstation.

Important:
Files containing PHI must be encrypted when they are stored (at rest) and when they are transferred between networked systems (in transit). To ensure that files containing PHI are encrypted when they are stored, encrypt them before transferring them to storage. To ensure that files containing PHI remain encrypted during transit, use SFTP/SCP or the IU Globus Web App. For more, see Recommended tools for encrypting data containing HIPAA-regulated PHI.

This is document avkm in the Knowledge Base.
Last modified on 2021-07-09 15:05:07.