Available access to allocated and short-term storage capacity on IU's research systems

Important:
Before storing data on any of Indiana University's research computing or storage systems, make sure you understand the information in Types of sensitive institutional data appropriate for the research computing and database systems at IU.

On this page:


Allocated storage and its uses

As a member of the Indiana University research community, you have access to allocated storage on the following Research Technologies systems.

Allocated resource Description Default quota
Research supercomputer home directories
  • Allocated with accounts on UITS research supercomputers
  • Disk-based, replicated, and highly secure
  • Relatively slow I/O throughput
100 GB
(see below)
Data Capacitor II project space
  • Allocated with approval from UITS High Performance File Systems (HPFS) allocations committee
  • Disk-based, replicated, and highly secure
  • Lustre-based, high-throughput, parallel file system
  • Temporary storage only (files may be purged if they have not been accessed for more than 180 days)
  • Not disaster-resilient (files are not backed up; account owners are responsible for backing up important data to other storage resources)
Up to 10 TB on request
(see below)
Scholarly Data Archive (SDA)
  • Allocations are available to IU graduate students, faculty, and staff (undergraduate students and sponsored affiliates need either faculty or staff sponsors to request accounts)
  • HPSS data archive
  • Tape-based, long-term storage
Up to 50 TB
(see below)

Home directories

By default, you are allotted 100 GB of home directory disk storage for use on IU research computing resources.

Home directories are neither intended for storing large data sets nor capable of handling data-intensive computational I/O; other Research Technologies systems are provided for those purposes. Home directories are relatively small by design, as they are intended for low-capacity, low-performance uses (such as storing parameter or configuration files and compiling binaries) only.

When in support of research activities, quota increases may be granted. If you need additional home directory disk space, and other allocated storage options do not meet your needs, email the UITS Research Storage team.

To request an account on an Indiana University research system, see Get additional IU computing accounts. Account availability depends on your eligibility.

DC2 project directories

IU research projects may be allocated up to 10 TB of dedicated, disk-based storage on the Data Capacitor II (DC2) Lustre-based, high-throughput parallel file system.

DC2 project directories are temporary workspaces that are not intended for long-term storage. Data are not backed up, files may be purged if they have not been accessed for more than 180 days, and purged files cannot be recovered. Project owners are responsible for moving data they want to keep to long-term storage solutions.

DC2 project directories are allocated with approval from the UITS High Performance File System (HPFS) group. To request a DC2 project directory allocation, submit the Data Capacitor II project allocation request form. The HPFS allocation committee will review your request and contact you with more information.

When in support of research, quota increases are routinely granted. To request a quota increase for a DC2 project directory, the project owner should contact the HPFS team.

Scholarly Data Archive accounts

An account on IU's Scholarly Data Archive (SDA) provides up to 50 TB of long-term, tape-based, fully-replicated archival storage.

The Indiana University Scholarly Data Archive (SDA) provides extensive capacity (approximately 42 PB of tape overall) for storing and accessing research data. The SDA is a distributed storage service co-located at IU data centers in Bloomington and Indianapolis, providing IU researchers with large-scale archival or near-line data storage, arranged in large files, with two copies of data made by default (for disaster recovery).

When in support of research activities, extensions beyond 50TB may be granted at no charge; for help, email the UITS Research Storage team.

For eligibility requirements, see the "Research system accounts (all campuses)" section in Computing accounts at IU.

  • To request an individual (personal) account on the Indiana University Scholarly Data Archive (SDA), follow the instructions in Get additional IU computing accounts.

    If you are eligible to request an SDA account, but it is not listed among the accounts available to request, contact your campus Support Center for help.

  • To request to have an SDA account created for your IU group account, contact the Research Storage team.
    Note:
    In accordance with standards for access control mandated by the HIPAA Security Rule, you are not permitted to access data containing protected health information (PHI) using a group (or departmental) account. To ensure accountability and maintain appropriate levels of access control, all users must use an individual login for all work involving PHI.

After submitting your account request, UITS will notify you via email when your account is ready for use.

Unallocated temporary storage and its uses

As a member of the IU research community, you have access to unallocated temporary storage on local and shared, disk-based scratch file systems managed by UITS Research Technologies. No quotas are enforced on these scratch directories, but they are intended for temporary storage of research data only. Data are not backed up. Files are regularly purged once they reach a certain age, and cannot be recovered. Users are responsible for moving data they want to keep to permanent storage solutions.

Local scratch space

The following amounts of local scratch space are accessible to users with accounts on IU research computing systems:

System Local scratch space available
Carbonate 1.3 TB accessible at /tmp
Karst 180 GB accessible at /tmp

Files in local scratch directories are automatically deleted once they are 14 days old; purged files cannot be recovered.

DC2 shared scratch space

Shared scratch space is hosted on the DC2 file systems. When you create an account on one of IU's research computing resources, corresponding scratch directories are created for you on the DC2 file systems. Available capacity fluctuates based on project directory usage.

Your DC2 scratch directories are accessible from each research computing system at /N/dc2/scratch/username (replace username with the appropriate IU username).

Files in DC2 scratch directories may be purged if they have not been accessed for more than 60 days.

Purchase tape or condominium storage

Schools, departments, labs, and individual investigators with storage needs beyond those that can be met by baseline (no cost) services, may consider entering into fee-based cluster "condominium" hosting agreements with UITS Research Technologies. Storage space on the DC2/DC-WAN2 file systems and the Scholarly Data Archive may be purchased also. To find cost information, use the most recent link (for IU Bloomington or IUPUI) in the Rates for direct-bill services section of Rates and costs of UITS services, and look under "IU Internal Research Technologies Services".

If you are interested in fee-based storage services, contact UITS Research Technologies to request a meeting.

Access your allocated research storage space(s)

Home directories

Provided you have the required account(s), you can access your home directory and/or DC2/DC-WAN2 project directory from any of IU's research supercomputers (in the following examples, replace username with the appropriate individual or group IU username):

System Home directory path
Big Red II /N/u/username/BigRed2
Carbonate /N/u/username/Carbonate
Karst /N/u/username/Karst

From your home directory space on one system, you can access your home directory space on another (provided you have accounts on both systems). For the following examples, assume you (username) have accounts on Big Red II and Karst:

  • To access your home directory on Karst from your home directory on Big Red II, on the command line, enter :
    cd ../Karst
    

    As a result, the command line changes to reflect your present location within the file system:

    username@login1:/N/u/username/Karst>
    
  • To access a subdirectory on Karst (for example, ~/work_dir) from a subdirectory on Big Red II (for example, ~/working_directory), on the command line, enter:
    cd ../../Karst/work_dir
    

    As a result, the command line changes to reflect your present location within the file system:

    username@login1:/N/u/username/Karst/work_dir>
    

DC2/DC-WAN2 project directories

DC2/DC-WAN2 project directories are accessible from each IU research supercomputer at /N/dc2/projects/username (DC2) and /N/dcwan/projects/username (DC-WAN2); replace username with the project owner's Network ID username.

For more, see About your scratch or project directory on the Data Capacitor II or DC-WAN2 file system.

SDA accounts

Methods available for transferring data to and from the Indiana University Scholarly Data Archive (SDA) include Hierarchical Storage Interface (HSI), secure FTP (SFTP), secure copy (SCP), and GridFTP. For instructions, see:

Read-only access is available via CIFS/Samba; see Use Samba or CIFS to access your SDA account from your personal workstation.

Important:
Files containing PHI must be encrypted when they are stored (at rest) and when they are transferred between networked systems (in transit). To ensure that files containing PHI are encrypted when they are stored, encrypt them before transferring them to storage. To ensure that files containing PHI remain encrypted during transit, use SFTP/SCP or the IU Globus Web App. For more, see Recommended tools for encrypting data containing HIPAA-regulated PHI.

This is document avkm in the Knowledge Base.
Last modified on 2019-02-15 18:12:08.

Contact us

For help or to comment, email the UITS Support Center.