Available access to allocated and short-term storage capacity on IU's research systems

On this page:


Allocated storage and its uses

Important:
Before storing data on any of Indiana University's research computing or storage systems, make sure you understand the information in Types of sensitive institutional data appropriate for the research computing and database systems at IU.

Indiana University provides multiple storage options to store, share, and collaborate on files on IU research computing resources. As a member of the IU research community, you have access to allocated storage on the following Research Technologies systems.

Allocated resource Description Default quota
Research supercomputer home directories
  • Allocated with accounts on UITS research supercomputers
  • Disk-based, replicated, and highly secure
  • Relatively slow I/O throughput
100 GB
(see below)
Scholarly Data Archive (SDA)
  • Allocations are available to IU graduate students, faculty, and staff (undergraduate students and sponsored affiliates need either faculty or staff sponsors to request accounts)
  • HPSS data archive
  • Tape-based, long-term storage
Up to 50 TB
(see below)
Slate persistent storage for scholarly data
  • Allocations are available to all IU research supercomputer users
  • Disk-based, replicated, and highly secure
  • Lustre-based, high-throughput, parallel file system
  • No purge policy
Up to 1.6 TB
(see below)
Slate-Project project space
  • Allocated with approval from UITS High Performance File Systems (HPFS) allocations committee
  • Disk-based, replicated, and highly secure
  • Lustre-based, high-throughput, parallel file system
  • No purge policy
Up to 15 TB
(see below)

Home directories

By default, you are allotted 100 GB of home directory disk storage for use on IU research computing resources.

Home directories are neither intended for storing large data sets nor capable of handling data-intensive computational I/O; other Research Technologies systems are provided for those purposes. Home directories are relatively small by design, as they are intended for low-capacity, low-performance uses (such as storing parameter or configuration files and compiling binaries) only.

When in support of research activities, quota increases may be granted. If you need additional home directory disk space, and other allocated storage options do not meet your needs, email the UITS Research Storage team.

To request an account on an Indiana University research system, see Get additional IU computing accounts. Account availability depends on your eligibility.

Scholarly Data Archive accounts

An account on IU's Scholarly Data Archive (SDA) provides up to 50 TB of long-term, tape-based, fully-replicated archival storage.

Note:
On October 6, 2019, UITS Research Technologies implemented a quota to limit the number of files users can store in their SDA accounts. The file quota for new accounts is 25,000 files. To check your SDA quota and usage, log into HPC everywhere.

The Indiana University Scholarly Data Archive (SDA) provides extensive capacity (approximately 79 PB of tape overall) for storing and accessing research data. The SDA is a distributed storage service co-located at IU data centers in Bloomington and Indianapolis, providing IU researchers with large-scale archival or near-line data storage, arranged in large files, with two copies of data made by default (for disaster recovery).

When in support of research activities, extensions beyond 50 TB may be granted for a nominal charge; for help, email the UITS Research Storage team.

For eligibility requirements, see the "Research system accounts (all campuses)" section in Computing accounts at IU.

  • To request an individual (personal) account on the Indiana University Scholarly Data Archive (SDA), follow the instructions in Get additional IU computing accounts.

    If you are eligible to request an SDA account, but it is not listed among the accounts available to request, contact your campus Support Center for help.

  • To request to have an SDA account created for your IU group account, contact the Research Storage team.
    Note:
    In accordance with standards for access control mandated by the HIPAA Security Rule, you are not permitted to access data containing protected health information (PHI) using a group (or departmental) account. To ensure accountability and maintain appropriate levels of access control, all users must use an individual login for all work involving PHI.

After submitting your account request, UITS will notify you via email when your account is ready for use.

Slate accounts

The default Slate allocation is 800 GB per user. The default allocation may be increased to a maximum of 1.6 TB; to request more space on Slate, contact the High Performance File System (HPFS) team.

Slate is a centralized, high-performance Lustre file system designed for the persistent storage of scholarly data to meet the needs of data-intensive workflows and analytics running on Indiana University's research supercomputers. Unlike the Data Capacitor II (DC2) file system, Slate is not subject to a purge policy. Total capacity of Slate is 3.4 PB, and the measured aggregate write performance is 64 GB/s. For more, see About Slate high-performance storage for research computation at IU.

Space on Slate is available to all IU research supercomputer users. To create a Slate account, follow the instructions in Get additional IU computing accounts.

Slate-Project space

Slate-Project allocations are available to IU researchers who need shared/project spaces or more storage capacity than the 1.6 TB available per user on Slate. Requests for fewer than 15 TB are granted without fee; allocations of 15 TB or more are billed at $5.97 TB per month (an IU departmental account is required).

The Slate-Project high-performance file system is a centralized storage environment supporting extreme, data-intensive, performance-demanding (Big Data) workflows running on IU's research supercomputers. For more, see About Slate-Project high-performance project space at IU.

To get an allocation on the Slate-Project file system, fill out and submit the Slate-Project request form. Once your request is approved, you will be able to create a Slate-Project account using the instructions in Get additional IU computing accounts.

Unallocated temporary storage and its uses

As a member of the IU research community, you have access to unallocated temporary storage on local and shared, disk-based scratch file systems managed by UITS Research Technologies. No quotas are enforced on these scratch directories, but they are intended for temporary storage of research data only. Data are not backed up. Files are regularly purged once they reach a certain age, and cannot be recovered. Users are responsible for moving data they want to keep to permanent storage solutions.

Local scratch space

The following amounts of local scratch space are accessible to users with accounts on IU research computing systems:

System Local scratch space available
Carbonate 1.3 TB accessible at /tmp
Karst 180 GB accessible at /tmp

Files in local scratch directories are automatically deleted once they are 14 days old; purged files cannot be recovered.

DC2 shared scratch space

Shared scratch space is hosted on the DC2 file systems. When you create an account on one of IU's research computing resources, corresponding scratch directories are created for you on the DC2 file systems. Available capacity fluctuates based on project directory usage.

Your DC2 scratch directories are accessible from each research computing system at /N/dc2/scratch/username (replace username with the appropriate IU username).

Files in DC2 scratch directories may be purged if they have not been accessed for more than 60 days.

Purchase tape or condominium storage

Schools, departments, labs, and individual investigators with storage needs beyond those that can be met by baseline (no fee) services, may consider entering into fee-based cluster "condominium" hosting agreements with UITS Research Technologies. Storage space on the DC2/DC-WAN2 file systems and the Scholarly Data Archive may be purchased also. To find fee information, use the most recent link (for IU Bloomington or IUPUI) in the Rates for direct-bill services section of Rates and costs of UITS services, and look under "IU Internal Research Technologies Services".

If you are interested in fee-based storage services, contact UITS Research Technologies to request a meeting.

Access your allocated research storage space(s)

Home directories

Provided you have the required account(s), you can access your home directory and/or DC2/DC-WAN2 project directory from any of IU's research supercomputers (in the following examples, replace username with the appropriate individual or group IU username):

System Home directory path
Big Red II /N/u/username/BigRed2
Big Red 3 /N/u/username/BigRed3
Carbonate /N/u/username/Carbonate
Karst /N/u/username/Karst

From your home directory space on one system, you can access your home directory space on another (provided you have accounts on both systems). For the following examples, assume you (username) have accounts on Big Red II and Karst:

  • To access your home directory on Karst from your home directory on Big Red II, on the command line, enter :
    cd ../Karst
    

    As a result, the command line changes to reflect your present location within the file system:

    username@login1:/N/u/username/Karst>
    
  • To access a subdirectory on Karst (for example, ~/work_dir) from a subdirectory on Big Red II (for example, ~/working_directory), on the command line, enter:
    cd ../../Karst/work_dir
    

    As a result, the command line changes to reflect your present location within the file system:

    username@login1:/N/u/username/Karst/work_dir>
    

Slate accounts

Your Slate account is accessible from each IU research supercomputer at /N/slate/username (replace username with your IU username).

Slate-Project space

Your Slate-Project space is accessible from each IU research supercomputer at /N/project/project_name (replace project_name with the name of your project).

SDA accounts

Methods available for transferring data to and from the Indiana University Scholarly Data Archive (SDA) include Hierarchical Storage Interface (HSI), secure FTP (SFTP), secure copy (SCP), and GridFTP. For instructions, see:

Read-only access is available via CIFS/Samba; see Use Samba or CIFS to access your SDA account from your personal workstation.

Important:
Files containing PHI must be encrypted when they are stored (at rest) and when they are transferred between networked systems (in transit). To ensure that files containing PHI are encrypted when they are stored, encrypt them before transferring them to storage. To ensure that files containing PHI remain encrypted during transit, use SFTP/SCP or the IU Globus Web App. For more, see Recommended tools for encrypting data containing HIPAA-regulated PHI.

This is document avkm in the Knowledge Base.
Last modified on 2019-11-01 17:08:36.

Contact us

For help or to comment, email the UITS Support Center.