How do I use the IU Cyberinfrastructure Gateway to transfer data?
On this page:
- About endpoints
- Accessing the file transfer interface
- Selecting and activating endpoints
- Transferring files
- Monitoring progress
- Getting help
You can use the Indiana University Cyberinfrastructure Gateway (CI Gateway) to transfer data between your accounts on IU research computing and storage systems. You can also use the IU CI Gateway to move data to an IU research system from a local desktop or laptop workstation (or vice versa). You can also use the IU CI Gateway to transfer data to and from other networked hosts, such as digital services on the Extreme Science and Engineering Discovery Environment (XSEDE). For example, you can use the IU CI Gateway to:
- Transfer research data from jobs run on Big Red II to your archival storage space on the Scholarly Data Archive (SDA).
- Transfer data from your laptop to your scratch or project space on Data Capacitor II for use in applications running on Big Red II, Karst, or Mason.
- Transfer output data from a Big Red II job from your SDA account to your account on an XSEDE visualization system, such as Maverick (TACC).
The IU CI Gateway data transfer interface is based on Globus Online, a fast, reliable, high-performance Globus service designed to provide researchers a tool for secure data movement between networked hosts (i.e., endpoints). Although the IU CI Gateway data transfer interface uses the GridFTP protocol, the service automates many of the complex, time-consuming tasks associated with managing large data transfers, saving you time and delivering a marked improvement in performance over other transfer methods, such as SCP.
The IU CI Gateway file transfer interface allows you to set up
transfers between networked endpoints. Globus Online (the back-end
GridFTP engine) uses a unique naming convention to assign registered
endpoints short names, so users don't have to remember their fully
domain names. Each of these short names includes a hash mark
# ) flanked on one side by text signifying
the name of an institution, department, group, or individual, and on
the other side by text signifying a hostname or a personal computer's
nickname; for example:
galactic_empire#deathstar.gov RA-jedi#r2d2-hpss plorgana#leia_laptop
IU resources have endpoint names that start with
iu# . For example:
- The SDA endpoint is
- The Data Capacitor II endpoint is
iu#SDA-CIendpoint, you must have an SDA account. To use the
iu#dc2endpoint, you must have an account on Big Red II, Karst, or Mason. For eligibility and account creation information, see the Research system accounts (all campuses) section of What computing accounts are available at IU, and for whom?
All XSEDE digital services have endpoint names that begin with
xsede# . For example, the Gordon
(SDSC) endpoint is
xsede#gordon . To
transfer files to or from an XSEDE endpoint, you need an XSEDE
To apply for an allocation on the Extreme Science and Engineering Discovery Environment (XSEDE), the principal investigator must be a researcher or an educator at a US-based institution, including federal research labs and commercial organizations (additional information may be needed from researchers not affiliated with academic or non-profit research institutions). Postdoctoral researchers are eligible to serve as principal investigators. In general, XSEDE follows guidelines described in the National Science Foundation (NSF) Grant Proposal Guide. Principal investigators with support from any funding sources (i.e., not just NSF-funded projects) are encouraged to apply. If your institution is not a university or a two- or four-year college, special rules may apply.
Personal computer endpoints (using Globus Connect)
To use your personal computer as an endpoint, you must install the Globus Connect application on your computer. To download the Globus Connect installer for OS X, Linux, or Windows, click administrator.(above the transfer window on the right). To run the installer, you'll need to be logged into your computer as an
When you set up your personal computer as an endpoint, you will most likely name the endpoint something that combines your IU username with an informal name for your computer, for example:
Accessing the file transfer interface
To access IU CI Gateway file transfer interface:
- In your web browser, go to the IU Cyberinfrastructure Gateway, and then click .
- On the IU CAS login page, enter your IU Network ID username and passphrase, and then click .
- From the menu bar at the top (or the pull-down menu in the top
right corner, depending on the size of your browser window), click
Note: If your browser displays a warning about an untrusted connection, choose the option that adds an exception, so you can proceed to the Globus Sign In page..
- To proceed, you need a Globus Online account:
- If you don't have a Globus Online account, follow these steps to
- In the top right corner, click , fill out the brief registration form, and then click .
- Check your email for a request from
email@example.com confirm your email address. Click the link provided in the message to confirm your email address.
- Check your email inbox for a "Welcome to Globus" message from
firstname.lastname@example.org. In the message, click the link that mentions transferring files using your web browser. This should take you to the Globus Sign In page.
- If you have a Globus Online account (or just created one), skip to the next step.
- If you don't have a Globus Online account, follow these steps to create one:
- On the Globus Sign In page, click , and then from the "Select Identity Provider" menu, select .
- Click .
- Under "Select An Identity Provider", use the scrolling menu to select , click to check the checkbox, and then click .
- If prompted, log in with your IU Network ID credentials.
You should now be on the Transfer Files page; here you can begin to select and activate endpoints (see the next section).
Selecting and activating endpoints
To set up a data transfer, you must select and activate the desired endpoints. On the Transfer Files page, use either transfer window for the source or destination endpoint; when both endpoints are activated, you'll be able to move data from one to the other in either direction.
To select and activate endpoints:
- In one of the fields, begin typing an endpoint name, or to the right of the field, click the icon. This will reveal a drop-down list of available endpoints.
- Browse to find the desired endpoint, click its name, and then
Note: Activating the
iu#dc2endpoint will place you in the system's top-level directory (
/). To access your personal scratch or project space, you must manually navigate to it:
to activate it.
- In the "Path" field, enter one of the following (replace
usernamewith your IU username):
- For your scratch space, enter:
- For your project space, enter:
- For your scratch space, enter:
- Click .
- In the "Path" field, enter one of the following (replace
- Depending on the endpoint, you may (or may not) need to authenticate with another set of credentials to activate that endpoint. For example, to activate an XSEDE endpoint, you will be prompted to authenticate with your XSEDE-wide login credentials. However, when activating an IU endpoint (e.g., the SDA or Data Capacitor II), you won't be prompted to log in; the IU CI Gateway will automatically pass your IU credentials to the endpoint.
When an endpoint is activated, its directory tree will appear in
the transfer window, and you can browse existing folders, create new
folders, show hidden files, and delete files. For example, activating
iu#SDA-CI endpoint places you in your home directory
on the SDA. From there, you should be able to see, browse, and
manipulate all of your files and subdirectories.
When you have activated endpoints in both transfer windows, you can start transferring files between them.
To transfer files between endpoints:
- In the source transfer window, select one or more files or folders to transfer.
- To reveal optional transfer settings, near the bottom, click
Note: Thesetting is selected by default. . Check the boxes to select the settings you
- Optionally, in the "Label This Transfer" field, enter a short text description, which will be displayed in the log of transfer activity.
- To start the transfer, near the top of the destination transfer window, click the highlighted arrow.
To monitor the progress of your transfer, near the top of the "Start Transfer" page, clickto open the "View Transfers" page. Alternatively, from the top menu bar (or the touch-friendly drop-down menu), select , and then .
On the "View Transfers" page, you'll see a history of your data transfers. By default, 25 records are displayed per page, with the most recent transfer requests at the top:
- To change the sort order, click the gear-shaped icon next to right of or .
- To change how many records are displayed per page, use the drop-down list to select , , , or .
- To navigate through the list of records, use the arrows at the bottom of the page. Click the single arrows to go one page forward or backward; click the double arrows to skip to the end or beginning of the list.
The "Task Progress" column will display a progress bar for each
transfer. While a transfer is in progress, its progress bar will
indicate how many files have been successfully transferred (e.g.,
2/5). When a transfer is complete, its
progress bar will turn green and indicate that all requested files
have been transferred (e.g.,
To cancel one or more transfers that are in progress, click the checkbox(es) to select the transfer(s) you want to cancel, and then click.
To return to the "Start Transfer" page, from the top menu bar (or the touch-friendly drop-down menu), select, and then .
- If you have a system-specific question about Big Red II, Karst, Mason, or the Research Database Complex (RDC) contact the High Performance Systems (HPS) team.
- If you have questions about the Scholarly Data Archive (SDA), contact the Research Storage team.
- If you have questions about shared scratch or project space on the Data Capacitor II or Data Capacitor Wide Area Network (DC-WAN) file system, contact the High Performance File Systems (HPFS) team.
- If you have questions about the development tools, compilers, scientific or numerical libraries, or debuggers available on the research computing system, contact the Scientific Applications and Performance Tuning (SciAPT) team.
- If you have questions about the statistical and mathematical applications available on the research computing systems, contact the Research Analytics group.
- If you have questions about the bioinformatics and genome analysis packages available on the research computing systems, email the National Center for Genome Analysis Support (NCGAS).
For general inquiries about UITS Research Technologies systems and services, complete and submit the Research Technologies request for help form.
This is document bdqp in the Knowledge Base.
Last modified on 2016-06-23 00:00:00.
- Fill out this form to submit your issue to the UITS Support Center.
- Please note that you must be affiliated with Indiana University to receive support.
- All fields are required.