Transfer files larger than 500 GB to the SDA at IU

Important:

Files containing PHI must be encrypted when they are stored (at rest) and when they are transferred between networked systems (in transit). To ensure that files containing PHI are encrypted when they are stored, encrypt them before transferring them to storage. To ensure that files containing PHI remain encrypted during transit, use SFTP/SCP or the IU Globus Web App. For more, see Recommended tools for encrypting data containing HIPAA-regulated PHI.

To transfer a single file that's larger than 500 GB to the Indiana University Scholarly Data Archive (SDA), you can use HSI or the IU Globus Web App without any special options; see:

However, if you need to transfer large amounts of data using a pipe (for example, using tar to aggregate data), use HSI with the following options to inform the system that the resulting file may be very large:

hsi 'put copies=2 coslist=large "|tar cf - ." : test.tar'

In the above example, the copies=2 coslist=large string changes the class of service to 3 (for large files) and warns the system to allocate as much disk cache space as possible for the file.

Note:

Do not use this method unless you are certain you will be creating files that are hundreds of gigabytes in size.

For help with HSI commands and options, see the HSI Reference Manual. If you have questions or need help transferring large files to your SDA account, email the UITS Research Storage team (store-admin@iu.edu).