Use the IU Globus Web App for SDA and DC2 file transfers

Note:
UITS plans to retire the SDA web interface on October 21, 2018. For web-based access to your SDA data, UITS recommends using the IU - Scholarly Data Archive endpoint in the IU Globus Web App. If you have questions or concerns, contact the UITS Research Storage team.

On this page:


Overview

You can use the IU Globus Web App to transfer data between your accounts on IU research computing and storage systems. You can also use it to move data to an IU research system from a personal computer (or vice versa). You can also use the IU Globus Web App to transfer data to and from other networked hosts, such as digital services on the Extreme Science and Engineering Discovery Environment (XSEDE). For example, you can use the IU Globus Web App to:

  • Transfer output data from an application running on an IU research supercomputer to your archival storage space on the Scholarly Data Archive (SDA).
    Important:
    Before storing data on the SDA, make sure you understand the information in Types of sensitive institutional data appropriate for the Scholarly Data Archive at IU.
  • Transfer data from your laptop to your Data Capacitor II (DC2) scratch space or DC-WAN2 project space for use with an application running on an IU research supercomputer.
  • Transfer your application's output data from your SDA account to your account on an XSEDE visualization system.

The IU Globus Web App is based on Globus Online, a fast, reliable, high-performance Globus service designed to provide researchers a tool for secure data movement between networked hosts (i.e., endpoints). Although the IU Globus Web App uses the GridFTP protocol, it automates many of the tasks associated with managing large data transfers, saving you time and delivering better performance than other transfer methods (e.g., SCP).

Access the IU Globus Web App

To access the IU Globus Web App:

  1. In your web browser, go to the IU Globus Web App.
  2. Under "Use your existing organizational login", use the drop-down list to find and select Indiana University.
  3. Click Continue, and then log into IU CAS and authenticate via Duo two-step authentication.
  4. To proceed, you need a Globus ID:
    • If you don't have a Globus ID, go to Create a Globus ID and follow the on-screen instructions. (Use your IU email address; next to "Organization" enter Indiana University.)
    • If you have a Globus ID (or just created one), skip to the next step.
  5. On the Globus Sign In page, click alternate login, and then from the "Select Identity Provider" menu, select InCommon/CILogon.
  6. Click Proceed.
  7. Under "Select An Identity Provider", use the scrolling menu to select Indiana University, click to check the Remember this selection checkbox, and then click Log On.
  8. If prompted, log in with your IU username and passphrase.

You should now be on the Transfer Files page, where you can begin to select and activate endpoints (see the next section).

Set up a transfer between two endpoints

The IU Globus Web App allows you to set up transfers between networked endpoints. To set up a data transfer, you must select and activate the desired endpoints.

Endpoint names

The IU Globus Web App uses Globus Online's naming convention to assign short names to registered endpoints (so users don't have to remember fully qualified domain names). Each endpoint's name includes a hash mark ( # ) and text signifying the name of the host institution, department, group, or individual, and a hostname or a personal computer's nickname. IU resources have endpoint names that start with iu# :

IU resource Globus endpoint
SDA IU - Scholarly Data Archive
DC2 iu#dc2
Note:
To use the IU - Scholarly Data Archive endpoint, you must have an SDA account. To use the iu#dc2 endpoint, you must have an account on one of IU's research supercomputers. For eligibility and account creation information, see the Research system accounts (all campuses) section of Computing accounts at IU.

XSEDE digital services have endpoint names that begin with xsede#. To transfer files to or from an XSEDE endpoint, you need an XSEDE allocation.

Activate endpoints

To set up a transfer between two endpoints, on the Transfer Files page, use either transfer window for the source or destination endpoint; when both endpoints are activated, you'll be able to move data from one to the other in either direction.

To select and activate endpoints:

  1. In one of the Endpoint fields, begin typing an endpoint name, or to the right of the Endpoint field, click the ... icon. This will reveal a drop-down list of available endpoints.
  2. Browse to find the desired endpoint, click its name, and then click Go to activate it.
    Note:
    Activating the iu#dc2 endpoint will place you in the system's top-level directory ( /). To access your personal scratch space, in the "Path" field, enter /~/N/dc2/scratch/username (replace username with your IU username), and then click Go.
  3. When activating an IU endpoint (e.g., IU - Scholarly Data Archive), you won't be prompted to log in; the IU Globus Web App will automatically pass your IU credentials to the endpoint. When activating an XSEDE endpoint, you will be prompted to authenticate with your XSEDE-wide login credentials.

Activating the IU - Scholarly Data Archive endpoint places you in your home directory on the SDA. From there, you should be able to see, browse, and manipulate all of your files and subdirectories.

When you have activated endpoints in both transfer windows, you can start transferring files between them.

Transfer files

To transfer files between endpoints:

  1. In the source transfer window, select one or more files or folders to transfer.
  2. To reveal optional transfer settings, near the bottom, click more options. Check the boxes to select the settings you prefer:
    • only transfer new or changed files
    • delete files on destination that do not exist on source
    • preserve source file modification times
    • verify file integrity after transfer
    • encrypt transfer
    Note:
    The verify file integrity after transfer setting is selected by default.
  3. Optionally, in the "Label This Transfer" field, enter a short text description, which will be displayed in the log of transfer activity.
  4. To start the transfer, near the top of the destination transfer window, click the highlighted arrow.

To monitor the progress of your transfer, near the top of the "Start Transfer" page, click View Transfer Activity to open the "View Transfers" page. Alternatively, from the top menu bar (or the touch-friendly drop-down menu), select Manage Data, and then View Transfers.

On the "View Transfers" page, you'll see a history of your data transfers. By default, 25 records are displayed per page, with the most recent transfer requests at the top:

  • To change the sort order, click the gear-shaped icon next to right of Completion Time or Status.
  • To change how many records are displayed per page, use the View Records drop-down list to select 5, 10, 50, or 100.
  • To navigate through the list of records, use the arrows at the bottom of the page. Click the single arrows to go one page forward or backward; click the double arrows to skip to the end or beginning of the list.

The "Task Progress" column will display a progress bar for each transfer. While a transfer is in progress, its progress bar will indicate how many files have been successfully transferred (e.g., 0/5, 2/5). When a transfer is complete, its progress bar will turn green and indicate that all requested files have been transferred (e.g., 5/5).

To cancel one or more transfers that are in progress, click the checkbox(es) to select the transfer(s) you want to cancel, and then click Cancel.

To return to the "Start Transfer" page, from the top menu bar (or the touch-friendly drop-down menu), select Manage Data, and then Start Transfer.

Add your personal computer as an endpoint

To use your personal computer as an endpoint:

  1. Go to Add Globus Connect Personal Endpoint.
  2. Next to "Display Name", enter a human-friendly name for your personal endpoint. UITS recommends combining your IU username with an informal name for your computer, for example:
      username#my_laptop
    
  3. Under "Download & Install Globus Connect Personal, click to download the installer for your operating system (macOS, Linux, or Windows).
  4. Run the downloaded installer (you'll need to be logged into your computer as an administrator).
  5. When prompted, paste in your Setup Key to complete the installation.

Getting help

Support for IU research computing systems, software, and services is provided by the Research Technologies division of UITS. To ask a question or get help, contact UITS Research Technologies.

This is document bdqp in the Knowledge Base.
Last modified on 2018-09-05 09:34:21.

Contact us

For help or to comment, email the UITS Support Center.