On the Scholarly Data Archive at IU, what are classes of service, and how do I use them?

Indiana University's HPSS archival storage system, the Scholarly Data Archive (SDA), uses classes of service to determine how data are cached and stored. The class of service (COS) assigned depends on the size of the file, how frequently it is accesses, and how many copies are stored, among other considerations.

SDA classes of service are organized by file size and the number of copies stored. Each COS has its own disk cache and tape pool. Other classes of service may be reserved for internal use. The SDA uses the following classes of service:

COS Description Number of copies Optimal file size Maximum file size
1 Small files 2 1 - 4 MB 10 GB
2 Medium files 2 4 - 64 MB 40 GB
3 Large files 2 64 MB - 1 TB 10 TB
4 Large files (parallel transfers) 2 500 GB - 50 TB 50 TB
13 Large files 1 64 MB - 1 TB 10 TB

Classes 1, 2, 3, and 4 are dual-copy, meaning files are stored on two tapes in a separate tape libraries located at IU Bloomington and IUPUI (storing copies at two geographically distant locations provides additional protection from disasters). If the primary tape is damaged and can no longer be read, data can be recovered from copies on the second tape.

By default, SDA accounts are assigned "class 0", and the SDA assigns an appropriate COS for each file depending on its size. Because the SDA picks classes from the beginning of the list, most automatically set classes are dual-copy. If the SDA cannot determine the size when creating a file (e.g., when data are piped to HSI for transfer to SDA), it sets the COS to 1.

Make sure to use a dual-copy COS for important files. If your files are easily regenerated or have a limited lifetime, or you have master copies stored elsewhere, use a single-copy COS (e.g., 13). For example, use COS 13 for intermediate data files, workstation backups, and copies of tapes or CD-ROMs.

The method you use to set a non-default COS depends on the application you are using; however, you must specify the COS before transferring your files to the SDA (the COS normally does not change after that).

Application Set COS to N
HSI cos=N for all subsequent transfers, or put cos=N ... for just the current transfer
sftp or SCP Change directory to current directory, prefixed with /cosN, as described in At IU, how do I use SFTP or SCP to access my SDA account?
CIFS/Samba Connect to the small (COS 1), medium (COS 2), or large (COS 3) share, as described in At IU, how do I access the SDA via Samba?
SDA web interface Not possible (always COS 1)
GridFTP Append ,,N to the target filename

To see the COS assigned to a file on the SDA, you must access your account using HSI. Once you are connected, change to the directory that contains the file, and then, at the HSI prompt, enter:

  ls -U filename

In the above example, replace filename with the name of your file, or omit it to see information about all the files in the directory. For example:

  ? cd archive
  ? ls -U r2.tar
  -rw-------  1 leorgana  hpss   1  36834 DISK   17408 Mar 31 16:07 r2.tar
  ? ls -U
  /hpss/p/a/pamidala/archive:
  -rw-------  1 leorgana  hpss   1  36834 DISK   17408 Mar 31 16:07 r2.tar
  -rw-------  1 leorgana  hpss   1  36834 DISK    2336 Mar 31 16:07 r2.tar.idx
  ?

In the above example, the files are assigned COS 1 (the number displayed to the right of hpss).

If you have questions about classes of service on the SDA, or need help, email the UITS Research Storage team.

This is document auli in the Knowledge Base.
Last modified on 2015-09-30 00:00:00.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.