On Big Red II, how do I use the data transfer nodes to improve the efficiency of my data transfers?

Big Red II users can improve the efficiency and performance of their data transfers by using the system's dedicated data transfer nodes (DTNs). The DTNs feature hardware components that are tuned to optimize I/O throughput, enabling high-speed data transfers between Research Technologies resources at Indiana University and other remote hosts.

Important:
To proceed, you must have an on-campus connection to the IU network, or be connected from off campus via the SSL VPN.

Big Red II users can log into the DTNs (dataxfer.bigred2.uits.iu.edu) via an SSH2 client; for example:

  ssh dataxfer.bigred2.uits.iu.edu

When prompted, log in with your IU username and passphrase (or a pre-configured SSH public-key authentication password).

When you authenticate, you are placed in your Big Red II home directory (e.g., /N/u/username/BigRed2, where username is your IU username); for example:

  username@dataxfer1:~> pwd
  /N/u/username/BigRed2

Local file systems on IU's research computing systems are mounted at /N/u and the Data Capacitor II (DC2) high-perform ace file system is mounted at /N/dc2. Users with accounts on those systems can navigate between them using cd on the command line; for example, to change from your home directory on Big Red II to your home directory on Karst, on the command line, enter the following (replace username with your IU username):

  cd /N/u/username/Karst

Use the same method to change to your DC2 scratch /N/dc2/scratch/username or project /N/dc2/project/username directory; for example (replace username with your IU username):

  cd /N/dc2/scratch/username

To transfer data between these file systems, use the cp or mv commands. For example, in your Big Red II home directory, to copy a file from your DC2 scratch space (e.g., app_data/my_input.txt) enter the following (replace username with your IU username):

  cp /N/dc2/scratch/username/app_data/my_input.txt ~/my_input.txt

For data transfers to or from your account on the IU Scholarly Data Archive (SDA), use HSI or HTAR; for instructions, see:

For data transfers to or from a remote host, use SCP or SFTP; for instructions, see:

Note:
The DTNs are intended only for running data transfer commands (e.g., cp, mv, scp, sftp, or hsi), or data compression and archiving commands (e.g., tar, htar, gzip, or cpio). Any other use of these nodes will cause your process to be terminated.

If you need help, or have a question about the DTNs, contact the UITS High Performance Systems (HPS) team.

This is document bfvm in the Knowledge Base.
Last modified on 2017-07-18 10:09:16.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.