ARCHIVED: Completed project: XSEDE (Extreme Science and Engineering Discovery Environment) - Supports EP Action 70

This content has been archived, and is no longer maintained by Indiana University. Resources linked from this page may no longer be available or reliable.
The original XSEDE award reached completion on August 31, 2016, at which time this project was superseded by XSEDE 2.0.

Primary UITS contact: Craig Stewart (Subcontract PI), Therese Miller (Project Manager)

Last update: September 2, 2016


Scientists, engineers, social scientists, and humanities experts around the world, many of them at colleges and universities, use advanced digital resources and services every day. Things like supercomputers, collections of data, and new tools are critical to the success of those researchers, who use them to make us all healthier, safer, and better off. The eXtreme Science and Engineering Discovery Environment (XSEDE) is creating an integrated national cyberinfrastructure that integrates these resources and services, makes them easier to use, and helps more people use them. XSEDE supports more than a dozen and a half supercomputers and high-end visualization and data analysis resources across the country and makes them available for use by the US open science and engineering research community. It is a successor to the 11-year-long TeraGrid facility funded by the NSF.

Indiana University is a key partner in XSEDE and is funded to deliver services as part of XSEDE via a subcontract from NCSA.

For more about XSEDE, see the project website.

Milestones and status:

  • XD operations failover: In the event of an emergency or an extended outage, the Indiana University Global Research Network Operations Center (GRNOC) will serve in the role of a backup XOC. The GRNOC is located on the IUPUI campus in Indianapolis, IN. GRNOC will be prepared to receive/send emails on behalf of XOC ( In addition, GRNOC has a dedicated phone line, 317-274-7782, where XOC calls can be forwarded to technicians at GRNOC should the primary XOC at NCSA become unreachable due to an emergency or outage.
  • Process and procedure documentation: GRNOC added a few XOC documents which will help GRNOC support XOC in their backup role. Training has been conducted to familiarize GRNOC technicians with XOC's daily activities/processes and procedures.

    Failover tests will occur quarterly to prepare and keep both XOC and GRNOC ready for a real failover event.

  • Systems operational support: Jetstream entered early operations on February 10, 2016. By the end of March, there were more than 240 users (representing more than 20 institutions in 16 states) on the system. In May 2016, Jetstream had its formal NSF review and was recommended for approval. It is expected to enter full operations once it is accepted.
  • User information and interfaces: XSEDE KB staff have 526 active documents and 125 archived documents as of the end of Q1 2016.
    • KB editing statistics for Q1 2016:
      • 1 new document
      • 14 modified documents
      • 0 archived documents
    • KB analytics for Q1 2016:
      Time frame Page
      Unique page
      Average time
      on page
      % Exit
      October 1, 2015 -
      December 31, 2015
      179,717 167,563 00:04:42 91.4% 89.20%
      January 1, 2016 -
      March 31, 2016
      208,753 192,715 00:04:27 91.39% 88.21%
      % Change +16.16% +15.01% -5.37% -0.61% -1.11%
  • User engagement: The 2015 XSEDE User Satisfaction Survey final report is complete and available for download from the Illinois Digital Environment for Access to Learning and Scholarship (IDEALS).

    UII continues to oversee the editing, entry, and maintenance of all technical documentation on the XSEDE website in addition to working with the User Engagement team to adapt to the ever-increasing needs of the community.

    User Engagement (UE) activities included planning, scheduling, and conducting all feedback activities (i.e., conference BoFs, focus groups, PI/user interviews), routinely contacting startup and XRAC new/renewal PIs, and analyzing old tickets to identify the areas of operations and support that need to be addressed to resolve issues encountered by the XSEDE user community.

    The 2015 XSEDE Staff Climate Study Report was published in December 2015 and is available for download from the IDEALS portal.

    PY4 EOT event totals

    Event name Location Attendees
    The Quilt Annual Meeting La Jolla, CA 40
    Oklahoma Supercomputing Symposium
    Norman, OK 70
    ACI-REF Meeting Clemson, SC 20
    BigGreen Cluster Installation
    Marshall University, Huntington, WV
    LCI Workshop Norman, OK 30
    ARCC Workshop Clemson, SC 50
    6 events   216
  • Science Gateways: The XSEDE Community account policy was drafted after receiving feedback from the Science Gateway team. Suggestions were made to the community account request process to provide more information up front, and to streamline account creation at the SPs. The security group is working on the SD&I activity for Science Gateways for collecting Gateway end user information, which can be used for security incident management.
  • Campus Bridging:

    Area metrics for E&O-Campus Bridging

    Area metric PY4 target PY4 Sub-goal supported
    Q3 201 Q4 201 Q1 201 Q2 201 PY4 total
    Number of campuses on which one or more CB tools have been adopted (presented as total unique campuses)
    100 39 59 81 89 89
    Advance-Create an open and evolving e-infrastructure (§3.2.1)
    Number of unique individuals who use one or more CB tools (Globus + XCBC) from start of year to quarter's end
    1,000 388 597 810 1,040 1,040
    Advance-Create an open and evolving e-infrastructure (§3.2.1)
    Number of Rocks rolls downloads
    100 35 89 42 59 225
    Advance-Create an open and evolving e-infrastructure (§3.2.1)

    PY4 XCBC and XNIT access metrics

    Repository Downloads Unique IPs
      Q3 2014 Q4 2014 Q1 2015 Q2 2015 Q3 2014 Q4 2014 Q1 2015 Q2 2015 35 89 42 59 None or no data available None or no data available None or no data available 35
    Yum Repository None or no data available None or no data available None or no data available 1001 None or no data available None or no data available None or no data available 79

    PY4 Globus statistics

      Year-end totals
    To/from XSEDE endpoint Files to XSEDE (millions) 317
    TB to XSEDE 5439.3
    Files from XSEDE (millions) 472
    TB from XSEDE 3968.7
    Users 952
    To/from XSEDE via Globus Connect
    Files to XSEDE (millions) 26.6
    TB to XSEDE 278
    Files from XSEDE (millions) 139.1
    TB from XSEDE 451.1
    Users 709
    To/from XSEDE from/to campuses
    TB to XSEDE 660.2
    TB from XSEDE 816.9
    Campuses 50
    Campus endpoints 89
    To/from campus TB to campuses 26,082
    TB from campuses 26,185
    Campuses 115
    Campus endpoints 372

    Indiana University brought up a 10-node XCBC (Rocks) cluster to use for education and testing purposes. To date, this has been used for developing a tutorial for the XSEDE community that will be unveiled in early 2016. This cluster, called Travertine, has also been used in the efforts to develop the Campus Queue in conjunction with OSG, as well as in developing science gateway services. Working with the XSEDE Science Gateways group, efforts are being made to create an easily packaged gateway framework to include in the XCBC build. The four-node cluster also allocated to Campus Bridging projects recently came online to test several different clustering alternatives. One of the projects was simply a CentOS 7-based update for Rocks. OpenHPC is also being explored. Using Xcat as a customized solution to replace Rocks is another alternative slated for review.

    The Campus Bridging team deployed two engineers to Marshall University in April 2015 to rebuild the BigGreen cluster. It was previously a commercial Rocks cluster distribution (Rocks+). Engineers rebuilt the head node with Rocks 6.2, which is based on CentOS 6.6, and installed the XSEDE roll. In addition, NVIDIA CUDA support was built to utilize the eight GPUs attached to the server. Once rebuilt, the server consisted of 22 Nahelem-series Dell servers with eight NVIDIA GPUs available. Results for the High-Performance Linpack (HPL) benchmark showed 3.85 TFLOPS of 5.6 TFLOPS (Rmax), or an efficiency of 68.5%; considering the factors, these results are quite good for this server. An independent analysis of the rebuild efforts is underway; results will be reported to XSEDE. Several other cluster build projects have also occurred; details will be forthcoming.

    The XSEDE Compatible Cluster project using Rocks rolls was announced at the IEEE Cluster meeting in Indianapolis in September 2013. A Knowledge Base document on how to get started with Rocks rolls was released just prior. An informational video on Campus Bridging and XSEDE Compatible Clusters was also posted on the PTI YouTube channel.

    Rich Knepper, Manager, Campus Bridging within RT, has taken on a leadership role within XSEDE in developing new campus bridging technologies and implementations for XSEDE. Campus bridging is a concept that IU pioneered, and the idea is very simple: to create technology bridges from local campus research computing systems to state, regional, and national systems so that all of the cyberinfrastructure to which a researcher has access is as easy to use as a peripheral attached to the researcher's laptop computer. Knepper is now leading a series of pilot studies related to campus bridging and XSEDE.

  • SP collaborations

    NCGAS was awarded an REU in order to create a virtual internship program with Clark State Community College. The REU students are compiling and benchmarking bioinformatics software on XSEDE resources and tuning software on Mason. This program helps create an internship opportunity for these students, by providing them with real-world hands-on experience.

Aids achievement of the following Empowering People Actions:

  • Recommendation A1
    • Action 4: Cyberinfrastructure. IU should continue to advance its local cyberinfrastructure, participation in national cyberinfrastructure, and its efforts to win federal funding of cyberinfrastructure programs that enhance IU's research capabilities.
    • Action 5: Philosophy of abundance. IU should pursue strategies that approximate a philosophy of abundance, within reason, towards unmetered availability of basic IT services, support, and infrastructure for creative activity, storage, computation, communication, and other activities fundamental to the work of the university via any appropriate sourcing strategy.
      • Participation in XSEDE creates local expertise that allows UITS to aid IU researchers in writing high-quality proposals, thus enabling a local abundance of supercomputing resources by adding availability of XSEDE supercomputers to IU systems such as Big Red II and Karst.
    • Action 6: Leveraging partnerships. IU should continue its highly successful program of relationships with hardware, software, and services vendors, and seek additional partnerships and creative exchanges that provide mutual benefits.
      • Participation in XSEDE identifies IU as among a very small number of key leaders in the US cyberinfrastructure community and aids our partnerships with vendors.
  • Recommendation A2
    • Action 9: Network partnerships. IU should continue to pursue opportunities for strategic partnerships that can provide services for advanced networks to further the missions of the university.
      • Through XSEDE, IU obtains funding for services delivered via the Global NOC.
  • Recommendation A4
    • Action 16: External funding. OVPIT should continue to lead and expand its efforts to effectively partner with academic units, campuses, administrative units, or individual investigators for external funding opportunities.
  • Recommendation A7
    • Action 24: World-class IT staff. IU should remain competitive with regard to compensation, benefits, facilities, workplace climate, and quality of life offerings through funding choices to attract, develop, and retain the very best technical and professional staff.
      • Five of UITS' world-class staff are funded by the XSEDE sub-award.
    • Action 25: Research into IT. IU should support and pursue research into information technology itself. IT Professionals and faculty should seek partnership opportunities for scholarly publication and invention disclosure that document meritorious research and discovery.
      • One of the aspects of IU's involvement in XSEDE is research and development of new IT facilities based on leadership and innovation of the staff of the UITS Research Technologies Division.
  • Recommendation B8
    • Action 27: Human-centered support. IU should continue to pioneer and provision effective means of user support through advanced tools for self-service and connection to IU experts to help faculty, staff, and students effectively use IT. IU should continue its work as a support infrastructure provider for national research projects and services.
      • IU's XSEDE sub-award funds the Support Division to do what is proposed in this Action.
  • Recommendation C15
    • Action 70: IT-enabled research. IU should purposefully select areas of great and timely promise for strategic development of IT-enabled research, scholarship, and/or creative activity.
    • Action 71: IT-enabled research resources. IU should identify a base of resources to provide both initial and sustained investments in selected areas for IT-enabled research, scholarship, and/or creative activity. This may include reallocating current resources and developing new ones, including endowments, grants, and/or additional fees.

Project team: All Research Technologies staff are involved directly or indirectly in the XSEDE project.


  • This project is led by PI John Towns (Director of Persistent Infrastructure at NCSA).
  • Craig Stewart (Associate Dean of Research Technologies, Executive Director of the Pervasive Technology Institute) is the local PI on IU's sub-award.
  • Therese Miller (Manager, Collaboration and Engagement Support, UITS Research Technologies) is the project manager for IU's participation in XSEDE.

This is document bbfc in the Knowledge Base.
Last modified on 2018-01-18 18:00:19.

Contact us

For help or to comment, email the UITS Support Center.