On IU HPSS, what are classes of service, and how do I use them?
Class of service (COS) is a mechanism used by HPSS to decide how user data will be cached and stored based on file size, required performance, storage location, and other considerations.
Indiana University HPSS supports the following classes of service:
| COS | Description | Optimal file size | # of copies | Maximum file size |
|---|---|---|---|---|
| 1 | Small files | 1 - 4MB | Dual copy | 10 GB |
| 2 | Medium files | 4 - 64 MB | Dual copy | 40 GB |
| 3 | Large files | 64 MB - 4TB | Dual copy | 10 TB |
| 4 | Large files, parallel transfers | 1 GB - 4 TB | Dual copy | 10 TB |
| 13 | Large files | 64 MB - 4 TB | Single copy | 10 TB |
IU HPSS classes of service are organized by file size and number of copies. Each COS has its own disk cache and tape pool. Other classes of service may be reserved for internal use.
Classes 1, 2, 3, and 4 store data on two tapes, each in a separate tape library, one at IU Bloomington and one at IUPUI. If the primary tape is damaged and can no longer be read, the data can be recovered from the second copy. In very rare circumstances, tapes may become damaged, so you should store important files on the dual copy classes of service. The dual-site nature of these copies also provides additional disaster resistance.
Use classes of service that do not make dual copies, such as 13, for storing data that can be easily regenerated or have a limited lifetime, or data whose master copy lives elsewhere. Examples are intermediate data files, workstation backups, and copies of tapes or CD-ROMs.
Users' accounts are assigned by default to "class 0". There is no such class, however; "class 0" means HPSS assigns an appropriate class of service to a given file, depending on its size. Because in this case classes of service are picked from the beginning of the list, the resulting automatically chosen classes of service default to dual copy. If the file size is not known by HPSS when creating the file, the default class of service will be class 1, for small files.
The way you set the class of service to something other than the default depends on the transfer method. You must specify the COS before transferring the file into HPSS; the COS will normally not change after that.
| Method | Set COS to N |
|---|---|
FTP, pftp_client
|
site setcos N for
all subsequent transfers
|
| HSI |
cos=N for all subsequent transfers, or
put cos=N ... for just the current transfer
|
| SFTP, SCP | Change directory to current
directory, prefixed with /cosN, as described in At IU, how do I use SFTP or SCP to access my SDA account?
|
| CIFS/SMB | Connect to the small (COS 1), medium (COS 2), or large (COS 3) share, as described in At IU, how do I map or mount my SDA account to my workstation? |
| Web interface | Not possible; always COS 1 |
| GridFTP | Append ,,N to the target filename
|
You can view the current COS only with FTP,
pftp_client, and HSI. These methods support an option to
the ls command that provides HPSS-specific
information.
| Method | List files with COS | Field displaying COS |
|---|---|---|
FTP, pftp_client
|
ls -lh or
dir -h
|
Third field (normally "group") |
| HSI | ls -U |
Fifth field |
This document was developed with support from the National Science Foundation (NSF) under Grant No. 0503697 to the University of Chicago and subcontracted to Indiana University. Additional support was provided by IU through its participation in the TeraGrid, which is supported by the NSF under Grants No. 0833618, SCI451237, SCI535258, and SCI504075. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.
Last modified on July 28, 2011.
