On IU's research systems, how much allocated and short-term storage capacity is available to me?

Important:
Before storing data on any of Indiana University's research computing or storage systems, make sure you understand the information in At IU, what types of sensitive institutional data are appropriate for the research computing systems?

On this page:


Allocated storage and its uses

As a member of the Indiana University research community, you have access to allocated storage on three systems managed by the Research Technologies division of UITS. The default allocated quota is 60.1 TB (of that, 10.1 TB is disk-based and 50 TB is tape-based).

Allocated resource Description Default quota
Research supercomputer home directories
  • Allocated with accounts on UITS research supercomputers
  • Disk-based, replicated, and highly secure
  • Relatively slow I/O throughput
100 GB
(see below)
Data Capacitor II project space
  • Allocated with approval from UITS High Performance File System (HPFS) allocations committee
  • Disk-based, replicated, and highly secure
  • Lustre-based, high-throughput, parallel file system
  • Temporary storage only (files may be purged if they have not been accessed for more than 180 days)
  • Not disaster-resilient (files are not backed up; account owners are responsible for backing up important data to other storage resources)
Up to 10 TB on request
(see below)
Scholarly Data Archive (SDA)
  • HPSS data archive
  • Tape-based, long-term storage
Up to 50 TB
(see below)

Home directories

By default, you are allotted 100 GB of home directory disk storage for use on IU research computing resources.

Home directories are neither intended for storing large data sets nor capable of handling data-intensive computational I/O; other Research Technologies systems are provided for those purposes. Home directories are relatively small by design, as they are intended for low-capacity, low-performance uses (e.g., storing parameter or configuration files, and compiling binaries) only.

When in support of authorized research activities, quota increases are routinely granted. If you need additional home directory disk space, specifically (i.e., other storage options do not meet your needs), email the UITS Research Storage team.

To request an account on an Indiana University research system, see At IU, if I already have some computing accounts, how do I get others? Account availability depends on your eligibility.

Note:
You can exceed your home directory quota for a "grace period" of seven days. If your home directory remains over quota beyond the grace period, you (and the jobs you run) will not be permitted access to additional storage space. After the grace period has ended, reaching or exceeding your home directory quota while a job is running will prevent output files from being appended or created, but will not interrupt the job.

DC2 project directories

IU research projects may be allocated up to 10 TB of dedicated, disk-based storage on the Data Capacitor II (DC2) Lustre-based, high-throughput parallel file system.

DC2 project directories are temporary workspaces that are not intended for long-term storage. Data are not backed up, files may be purged if they have not been accessed for more than 180 days, and purged files cannot be recovered. Project owners are responsible for moving data they want to keep to long-term storage solutions.

DC2 project directories are allocated with approval from the UITS High Performance File Systems (HPFS) group. To request a DC2 project directory allocation, submit the Request Form. The HPFS allocation committee will review your request and contact you with more information.

When in support of authorized research, quota increases are routinely granted. To request a quota increase for a DC2 project directory, the project owner should contact the HPFS team.

Scholarly Data Archive accounts

An account on IU's Scholarly Data Archive (SDA) provides up to 50 TB of long-term, tape-based, fully-replicated archival storage.

The Indiana University Scholarly Data Archive (SDA) provides extensive capacity (approximately 42 PB of tape overall) for storing and accessing research data. The SDA is a distributed storage service co-located at IU data centers in Bloomington and Indianapolis, providing IU researchers with large-scale archival or near-line data storage, arranged in large files, with two copies of data made by default (for disaster recovery).

When in support of authorized research activities, extensions beyond 50 TB are routinely granted at no charge; for help, email the UITS Research Storage team.

For eligibility requirements, see the "Research system accounts (all campuses)" section in What computing accounts are available at IU, and for whom?

If you (or your group) are eligible to request an SDA account, but it is not listed among the accounts available to request, contact your campus Support Center for help.

After submitting your account request, UITS will notify you via email when your account is ready for use.

Unallocated temporary storage and its uses

As a member of the IU research community, you have access to unallocated temporary storage on local and shared, disk-based scratch file systems managed by UITS Research Technologies. No quotas are enforced on these scratch directories, but they are intended for temporary storage of research data only. Data are not backed up. Files are regularly purged once they reach a certain age, and cannot be recovered. Users are responsible for moving data they want to keep to permanent storage solutions.

Local scratch space

The following amounts of local scratch space are accessible to users with accounts on Karst, Mason, or the Research Database Complex (RDC):

System Local scratch space available
Karst 180 GB accessible at /tmp
Mason 400 GB accessible at /scratch and /tmp
RDC 1 GB accessible at /tmp
10 GB accessible at/scr

Files in local scratch directories are automatically deleted once they are 14 days old; purged files cannot be recovered.

DC2 and DC-WAN shared scratch space

Shared scratch space is hosted on the DC2 and DC-WAN file systems. When you create an account on one of IU's research computing resources, corresponding scratch directories are created for you on the DC2 and DC-WAN file systems. Available capacity fluctuates based on project directory usage.

Your DC2 and DC-WAN scratch directories are accessible from each research computing system at /N/dc2/scratch/username and /N/dcwan/scratch/username, respectively (replace username with the appropriate IU Network ID username).

Files in DC2 or DC-WAN scratch directories may be purged if they have not been accessed for more than 60 days.

About purchasing tape or condominium storage

Schools, departments, labs, and individual investigators with storage needs beyond those that can be met by baseline (no cost) services, may consider entering into fee-based cluster "condominium" hosting agreements with UITS Research Technologies. Storage space on the DC2/DC-WAN file systems and the Scholarly Data Archive may be purchased also. To find cost information, use the most recent link (for IU Bloomington or IUPUI) in the Rates for direct-bill services section of Rates and costs of UITS services, and look under "IU Internal Research Technologies Services".

If you are interested in fee-based storage services, contact UITS Research Technologies to set up a meeting.

Accessing your allocated research storage space(s)

Home directories

Provided you have the required account(s), you can access your home directory and/or DC2/DC-WAN project directory from any of IU's research supercomputers (in the following examples, replace username with the appropriate individual or group Network ID username):

System Home directory path
Big Red II /N/u/username/BigRed2
Karst /N/u/username/Karst
Mason /N/u/username/Mason
Research Database Complex /N/u/username/RDC

From your home directory space on one system, you can access your home directory space on another (provided you have accounts on both systems). For the following examples, assume you (username) have accounts on Big Red II and Karst:

  • To access your home directory on Karst from your home directory on Big Red II, on the command line, enter :
  •   cd ../Karst
    

    As a result, the command line changes to reflect your present location within the file system:

      username@login1:/N/u/username/Karst>
    
  • To access a subdirectory on Karst (e.g., ~/work_dir) from a subdirectory on Big Red II (e.g., ~/working_directory), on the command line, enter:
  •   cd ../../Karst/work_dir
    

    As a result, the command line changes to reflect your present location within the file system:

      username@login1:/N/u/username/Karst/work_dir>
    

DC2/DC-WAN project directories

DC2/DC-WAN project directories are accessible from each IU research supercomputer at /N/dc2/projects/username (DC2) and /N/dcwan/projects/username (DC-WAN); replace username with the project owner's Network ID username.

For more, see:

SDA accounts

Methods available for transferring data to and from the Indiana University Scholarly Data Archive (SDA) include Hierarchical Storage Interface (HSI), secure FTP (SFTP), secure copy (SCP), https (via a web browser), and the IU Cyberinfrastructure Gateway (CI Gateway). For instructions, see:

Read-only access is available via CIFS/Samba; see At IU, how do I access the SDA via Samba?

This is document avkm in the Knowledge Base.
Last modified on 2016-11-15 08:44:52.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.