On the Scholarly Data Archive at IU, what are classes of service, and how do I use them?

Important:
Files containing PHI must be encrypted when they are stored (i.e., at rest) and when they are transferred between networked systems (i.e., in transit). Do not use HSI, HTAR, or Samba to transfer data containing PHI unless those data are encrypted already; HSI, HTAR, and Samba do not encrypt data during transit. To ensure that files containing PHI remain encrypted during transit, use SFTP/SCP or the IU Globus Web App. To ensure that files containing PHI are encrypted when they are stored on the SDA, encrypt them before transferring them. For more, see Recommended tools for encrypting data containing HIPAA-regulated PHI.

Indiana University's HPSS archival storage system, the Scholarly Data Archive (SDA), uses classes of service to determine how data are cached and stored. The class of service (COS) assigned depends on the size of the file, how frequently it is accesses, and how many copies are stored, among other considerations.

SDA classes of service are organized by file size and the number of copies stored. Each COS has its own disk cache and tape pool. Other classes of service may be reserved for internal use. The SDA uses the following classes of service:

COS Description Number of copies Optimal file size Maximum file size
1 Small files 2 1 - 4 MB 10 GB
2 Medium files 2 4 - 64 MB 40 GB
3 Large files 2 64 MB - 1 TB 10 TB
4 Large files (parallel transfers) 2 500 GB - 50 TB 50 TB
13 Large files 1 64 MB - 1 TB 10 TB

Classes 1, 2, 3, and 4 are dual-copy, meaning files are stored on two tapes in a separate tape libraries located at IU Bloomington and IUPUI (storing copies at two geographically distant locations provides additional protection from disasters). If the primary tape is damaged and can no longer be read, data can be recovered from copies on the second tape.

By default, SDA accounts are assigned "class 0", and the SDA assigns an appropriate COS for each file depending on its size. Because the SDA picks classes from the beginning of the list, most automatically set classes are dual-copy. If the SDA cannot determine the size when creating a file (e.g., when data are piped to HSI for transfer to SDA), it sets the COS to 1.

Make sure to use a dual-copy COS for important files. If your files are easily regenerated or have a limited lifetime, or you have master copies stored elsewhere, use a single-copy COS (e.g., 13). For example, use COS 13 for intermediate data files, workstation backups, and copies of tapes or CD-ROMs.

The method you use to set a non-default COS depends on the application you are using; however, you must specify the COS before transferring your files to the SDA (the COS normally does not change after that).

Application Set COS to N
HSI cos=N for all subsequent transfers, or put cos=N ... for just the current transfer
sftp or SCP Change directory to current directory, prefixed with /cosN, as described in Use SFTP or SCP to access your SDA account at IU
CIFS/Samba Connect to the small (COS 1), medium (COS 2), or large (COS 3) share, as described in Use Samba or CIFS to access your SDA account from your personal workstation.
GridFTP Append ,,N to the target filename

To see the COS assigned to a file on the SDA, you must access your account using HSI. Once you are connected, change to the directory that contains the file, and then, at the HSI prompt, enter:

  ls -U filename

In the above example, replace filename with the name of your file, or omit it to see information about all the files in the directory. For example:

  ? cd archive
  ? ls -U r2.tar
  -rw-------  1 leorgana  hpss   1  36834 DISK   17408 Mar 31 16:07 r2.tar
  ? ls -U
  /hpss/p/a/pamidala/archive:
  -rw-------  1 leorgana  hpss   1  36834 DISK   17408 Mar 31 16:07 r2.tar
  -rw-------  1 leorgana  hpss   1  36834 DISK    2336 Mar 31 16:07 r2.tar.idx
  ?

In the above example, the files are assigned COS 1 (the number displayed to the right of hpss).

If you have questions about classes of service on the SDA, or need help, email the UITS Research Storage team.

This is document auli in the Knowledge Base.
Last modified on 2018-06-12 15:44:56.

Contact us

For help or to comment, email the UITS Support Center.