At IU, how do I transfer files larger than 500 GB to the SDA?

Files containing PHI must be encrypted when they are stored (at rest) and when they are transferred between networked systems (in transit). Do not use HSI, HTAR, or Samba to transfer data containing PHI unless those data are encrypted already; HSI, HTAR, and Samba do not encrypt data during transit. To ensure that files containing PHI remain encrypted during transit, use SFTP/SCP or the IU Globus Web App. To ensure that files containing PHI are encrypted when they are stored on the SDA, encrypt them before transferring them. For more, see Recommended tools for encrypting data containing HIPAA-regulated PHI.

To transfer a single file that's larger than 500 GB to the Indiana University Scholarly Data Archive (SDA), you can use HSI or the IU Globus Web App without any special options; see:

However, if you need to transfer large amounts of data using a pipe (e.g., using tar to aggregate data), use HSI with the following options to inform the system that the resulting file may be very large:

  cd /test/dir
  hsi 'put copies=2 coslist=large "|tar cf - ." : test.tar'

In the above example, the copies=2 coslist=large string changes the class of service to 3 (for large files) and warns the system to allocate as much disk cache space as possible for the file.

Do not use this method unless you are certain you will be creating files that are hundreds of gigabytes in size.

This is document axnn in the Knowledge Base.
Last modified on 2019-01-10 16:48:09.

Contact us

For help or to comment, email the UITS Support Center.