How do I use the IU Cyberinfrastructure Gateway to transfer data?

On this page:

• Overview
• About endpoints
  • Endpoint names
  • IU endpoints
  • XSEDE endpoints
  • Personal computer endpoints (using Globus Connect)
• Accessing the file transfer interface
• Selecting and activating endpoints
• Transferring files
• Monitoring progress
• Getting help

Overview

You can use the Indiana University Cyberinfrastructure Gateway (CI Gateway) to transfer data between your accounts on IU research computing and storage systems. You can also use the IU CI Gateway to move data to an IU research system from a local desktop or laptop workstation (or vice versa). You can also use the IU CI Gateway to transfer data to and from other networked hosts, such as digital services on the Extreme Science and Engineering Discovery Environment ( XSEDE). For example, you can use the IU CI Gateway to:

• Transfer research data from jobs run on Big Red II to your archival storage space on the Scholarly Data Archive (SDA).
• Transfer data from your laptop to your scratch or project space on Data Capacitor II for use in applications running on Big Red II, Karst, or Mason.
  Note:

  Mason, Indiana University's large memory computer cluster, will be retired on January 1, 2018. For more, see About the Mason retirement.
• Transfer output data from a Big Red II job from your SDA account to your account on an XSEDE visualization system, such as Maverick (TACC).

The IU CI Gateway data transfer interface is based on Globus Online, a fast, reliable, high-performance Globus service designed to provide researchers a tool for secure data movement between networked hosts (i.e., endpoints). Although the IU CI Gateway data transfer interface uses the GridFTP protocol, the service automates many of the complex, time-consuming tasks associated with managing large data transfers, saving you time and delivering a marked improvement in performance over other transfer methods, such as SCP.

About endpoints

Endpoint names

The IU CI Gateway file transfer interface allows you to set up transfers between networked endpoints. Globus Online (the back-end GridFTP engine) uses a unique naming convention to assign registered endpoints short names, so users don't have to remember their fully domain names. Each of these short names includes a hash mark ( # ) flanked on one side by text signifying the name of an institution, department, group, or individual, and on the other side by text signifying a hostname or a personal computer's nickname; for example:

  galactic_empire#deathstar.gov 
  RA-jedi#r2d2-hpss
  plorgana#leia_laptop

IU endpoints

IU resources have endpoint names that start with iu# . For example:

• The SDA endpoint is iu#SDA-CI .
• The Data Capacitor II endpoint is iu#dc2.

XSEDE endpoints

All XSEDE digital services have endpoint names that begin with xsede# . For example, the Gordon (SDSC) endpoint is xsede#gordon . To transfer files to or from an XSEDE endpoint, you need an XSEDE allocation.

Personal computer endpoints (using Globus Connect)

To use your personal computer as an endpoint, you must install the Globus Connect application on your computer. To download the Globus Connect installer for OS X, Linux, or Windows, click Get Globus Connect (above the transfer window on the right). To run the installer, you'll need to be logged into your computer as an administrator.

When you set up your personal computer as an endpoint, you will most likely name the endpoint something that combines your IU username with an informal name for your computer, for example:

  username#my_laptop

Accessing the file transfer interface

To access IU CI Gateway file transfer interface:

1. In your web browser, go to the IU Cyberinfrastructure Gateway, and then click Login.
2. On the IU CAS login page, enter your IU Network ID username and passphrase, and then click Login.
3. From the menu bar at the top (or the pull-down menu in the top right corner, depending on the size of your browser window), click File Transfer.

   Note: If your browser displays a warning about an untrusted connection, choose the option that adds an exception, so you can proceed to the Globus Sign In page.

4. To proceed, you need a Globus Online account:
   • If you don't have a Globus Online account, follow these steps to create one:
     1. In the top right corner, click Sign Up with Globus Online, fill out the brief registration form, and then click Register.
     2. Check your email for a request from support@globus.org to confirm your email address. Click the link provided in the message to confirm your email address.
     3. Check your email inbox for a "Welcome to Globus" message from support@globus.org. In the message, click the link that mentions transferring files using your web browser. This should take you to the Globus Sign In page.
   • If you have a Globus Online account (or just created one), skip to the next step.
5. On the Globus Sign In page, click alternate login, and then from the "Select Identity Provider" menu, select InCommon/CILogon.
6. Click Proceed.
7. Under "Select An Identity Provider", use the scrolling menu to select Indiana University, click to check the Remember this selection checkbox, and then click Log On.
8. If prompted, log in with your IU Network ID credentials.

You should now be on the Transfer Files page; here you can begin to select and activate endpoints (see the next section).

Selecting and activating endpoints

To set up a data transfer, you must select and activate the desired endpoints. On the Transfer Files page, use either transfer window for the source or destination endpoint; when both endpoints are activated, you'll be able to move data from one to the other in either direction.

To select and activate endpoints:

1. In one of the Endpoint fields, begin typing an endpoint name, or to the right of the Endpoint field, click the ... icon. This will reveal a drop-down list of available endpoints.
2. Browse to find the desired endpoint, click its name, and then click Go to activate it.

   Note: Activating the iu#dc2 endpoint will place you in the system's top-level directory ( /). To access your personal scratch or project space, you must manually navigate to it:

   1. In the "Path" field, enter one of the following (replace username with your IU username):
      • For your scratch space, enter:

          /~/N/dc2/scratch/username

      • For your project space, enter:

          /~/N/dc2/project/username

   2. Click Go.
3. Depending on the endpoint, you may (or may not) need to authenticate with another set of credentials to activate that endpoint. For example, to activate an XSEDE endpoint, you will be prompted to authenticate with your XSEDE-wide login credentials. However, when activating an IU endpoint (e.g., the SDA or Data Capacitor II), you won't be prompted to log in; the IU CI Gateway will automatically pass your IU credentials to the endpoint.

When an endpoint is activated, its directory tree will appear in the transfer window, and you can browse existing folders, create new folders, show hidden files, and delete files. For example, activating the iu#SDA-CI endpoint places you in your home directory on the SDA. From there, you should be able to see, browse, and manipulate all of your files and subdirectories.

When you have activated endpoints in both transfer windows, you can start transferring files between them.

Transferring files

To transfer files between endpoints:

1. In the source transfer window, select one or more files or folders to transfer.
2. To reveal optional transfer settings, near the bottom, click more options. Check the boxes to select the settings you prefer:
   • only transfer new or changed files
   • delete files on destination that do not exist on source
   • preserve source file modification times
   • verify file integrity after transfer
   • encrypt transfer

   Note: The verify file integrity after transfer setting is selected by default.

3. Optionally, in the "Label This Transfer" field, enter a short text description, which will be displayed in the log of transfer activity.
4. To start the transfer, near the top of the destination transfer window, click the highlighted arrow.

Monitoring progress

To monitor the progress of your transfer, near the top of the "Start Transfer" page, click View Transfer Activity to open the "View Transfers" page. Alternatively, from the top menu bar (or the touch-friendly drop-down menu), select Manage Data, and then View Transfers.

On the "View Transfers" page, you'll see a history of your data transfers. By default, 25 records are displayed per page, with the most recent transfer requests at the top:

• To change the sort order, click the gear-shaped icon next to right of Completion Time or Status.
• To change how many records are displayed per page, use the View Records drop-down list to select 5, 10, 50, or 100.
• To navigate through the list of records, use the arrows at the bottom of the page. Click the single arrows to go one page forward or backward; click the double arrows to skip to the end or beginning of the list.

The "Task Progress" column will display a progress bar for each transfer. While a transfer is in progress, its progress bar will indicate how many files have been successfully transferred (e.g., 0/5, 2/5). When a transfer is complete, its progress bar will turn green and indicate that all requested files have been transferred (e.g., 5/5).

To cancel one or more transfers that are in progress, click the checkbox(es) to select the transfer(s) you want to cancel, and then click Cancel.

To return to the "Start Transfer" page, from the top menu bar (or the touch-friendly drop-down menu), select Manage Data, and then Start Transfer.

Getting help