ARCHIVED: Project: Jetstream - A national science and engineering cloud

Primary UITS contacts: Craig Stewart (PI), Therese Miller (Project Manager)

Last update: December 7, 2017

Description: Jetstream, led by Indiana University's Pervasive Technology Institute (PTI), will add cloud-based computation to the national cyberinfrastructure. Researchers will be able to create virtual machines on the remote resource that look and feel like their lab workstations or home machines but can harness thousands of times the computing power.

Publications: List maintained at the Jetstream website

PI Stewart submits monthly status reports to National Science Foundation (NSF) program officer Bob Chadduck via email. In addition, biweekly telephone meetings with the IU project team and program officer are held. Also, Stewart holds a monthly key leads telephone meeting with co-PIs and delegate representatives from partner institutions to discuss issues and progress.

Milestones and status:

• November 2017: 387 active XSEDE projects, covering 66 fields of science, and 2261 active users, representing 195 institutions; more than 80 million CPU hours allocated to XSEDE projects since June 2016; 9 active science gateways; 38 active education/teaching allocations, serving more than 800 undergraduate and graduate students
• October 2017: 360 active XSEDE projects, covering 66 fields of science, and 2,180 active users, representing 191 institutions; more than 76 million CPU hours allocated to XSEDE projects since June 2016; 9 active science gateways; 42 active education/teaching allocations, serving nearly 800 undergraduate and graduate students
• September 2017: 357 active XSEDE projects, covering 66 fields of science, and more than 2,100 active users, representing 187 institutions; more than 73 million CPU hours allocated to XSEDE projects since June 2016; 9 active science gateways; 34 active education/teaching allocations, serving more than 700 undergraduate and graduate students
• August 2017: 322 active XSEDE projects, covering 58 fields of science, and more than 2,000 active users, representing 189 institutions; more than 70 million CPU hours allocated to XSEDE projects since June 2016; 9 active science gateways; 40 education/teaching allocations; more than 600 undergraduate and graduate students using Jetstream
• July 2017: Jetstream's yearly NSF review was held July 19-20, 2017, in Bloomington, IN. The presentations included updates on milestones from the Jetstream staff and science impact presentations from several key projects. The review panel recommended that Jetstream be approved for continued funding.

The IT Communications Office (ITCO) updated the Jetstream website to align its visual style with other IU sites.

• June 2017: 298 active XSEDE projects covering 54 fields of science and 1,900+ active users representing 165 institutions; over 65 million CPU hours allocated to XSEDE projects since June 2016; 9 active science gateways; 28 education/teaching allocations; almost 600 undergraduate and graduate students using Jetstream
• May 2017: 290 active XSEDE projects covering 55 fields of science and 1,700+ active users representing 167 institutions; over 56 million CPU hours allocated to XSEDE projects since June 2016; 13 science gateways; 24 education/teaching allocations; almost 600 undergraduate and graduate students using Jetstream
• April 2017: 305 active XSEDE projects covering 57 fields of science and 1,600+ active users representing 174 institutions; over 56 million CPU hours allocated to XSEDE projects since June 2016; 12 science gateways; 23 education/teaching allocations; almost 600 undergraduate and graduate students using Jetstream
• March 2017: 290 active XSEDE projects covering 53 fields of science and over 1,500 active users representing 165 institutions; over 56 million CPU hours allocated to XSEDE projects since June 2016; 12 science gateways and 12 education/teaching allocations serving almost 400 undergraduate and graduate students
• January - February 2017: The first Stakeholder Advisory Board (SAB) meeting was held February 7-8, 2017, in Chicago, IL. A full report from the Jetstream SAB has been delivered to the Jetstream PIs.

Other key performance indicators for Jetstream:

PY1 (June 2016 - May 2017)	Q1	Q2	Q3		Target
			December 2016	January 2017
Capacity allocated via allocation process	96.5%	> 100%	> 100%	> 100%	90%
Distinct users	593	1,463	1,547	1,386	1,000/year (goal exceeded)
Students using Jetstream in educational or training settings	36	281	336	208	100/year (goal exceeded)
Science gateways using Jetstream	5	5	9	9	2/year (goal exceeded)
SUs available to users	4.41M	4.41M	1.47M	1.47M	1.47M/month
% of available SUs used	--	79.81%	> 100% (2,227,411.54 SUs)	> 100% (2,875,275.96 SUs)	--
Publications facilitated by use of Jetstream	1	1	6	0	5/year
VM images and data sets published with DOIs in IUScholarWorks	9	1	1	1	10/year



• October - December 2016: For the December XRAC meeting, just over 14 M SUs were requested in the form of research allocation requests. With just a single request turned down as a result of a too hastily written proposal, more than 13 M SUs were allocated. Considerable credit goes to Rudi Eigenmann, Bob Chadduck, Dave Hart, and Ken Hackworth. Bob was of great help during the proposal generation phase and offered ongoing sound advice about encouraging adoption. Ken and Dave were extremely effective in communicating to XRAC members the importance of building up the user community for Jetstream. And Rudi was particularly helpful in offering guidance to the XRAC regarding handling Jetstream allocations. Dave Hart allowed Stewart some time on the Sunday evening agenda to talk about Jetstream goals for user community expansion, which was also helpful. All in all, this was the best-conducted XRAC meeting that PI Stewart has ever attended, and it had a very positive outcome in terms of progress in expanding use of Jetstream.

The Jetstream rack operating with a Ceph deployment is operational at Jetstream-AZ (and one OpenStack deployment). The current OpenStack deployment has been connected with the development Atmosphere instances for some time, but until the deployment is locked down and tested thoroughly, it is assigned to a subnet that is accessible only to the UA campus.

Updates to Atmosphere (Q release) were deployed on November 1, 2016.

Through October 27, 2016, there were over 20 million SUs allocated on Jetstream on 224 active allocations.

• July - September 2016: Updates to Atmosphere (R release - Ravenous Raven) were deployed on September 27, 2016.

Jetstream was officially declared production on September 1, 2016.

During July, the IU cluster was upgraded to the latest Ubuntu release (16.04) and OpenStack release (Mitaka). The TACC cloud was also upgraded in early August to Mitaka.

The Atmosphere updates for SU accounting to XSEDE have been implemented ahead of the September 1, 2016 production date.

• April - June 2016: As of June 30, there were 120 active allocations on Jetstream, totaling 12,620,740 SUs and 31.9 TB of allocated storage.

Thursdays are scheduled as maintenance days through the end of early operations. The admin team is using these scheduled maintenance windows to complete software updates in preparation for entering production.

The formal NSF review was held on May 3 and 4, 2016. These days were spent presenting the system to the NSF panel, reviewing acceptance criteria, and answering questions about Jetstream. Jetstream was recommended for acceptance by the program officer. Production phase is expected to start as soon as Jetstream's formal acceptance is completed.

The first major Atmosphere update was completed on May 24, 2016. This contained several bug fixes and UI system improvements.

• January - March 2016: Early operations started February 10. Several projects have been released to the system, including multiple Campus Champion allocations, several Startup allocations, and the iPlant and Galaxy research allocations. By March 31, there were 242 users (representing 25 institutions in 18 states) on Jetstream. All Startup, Champion, Education, and Research allocations previously approved by XSEDE were granted access to the system.

The test system is scheduled to be sent from Indiana to Arizona in June 2016.

The VM library procedures are being developed. During early operations, the following images will be featured:

• CentOS 6 CLI Only
• CentOS 6 GUI
• CentOS 7 CLI Only
• CentOS 7 GUI (under review as of 4/1/16)
• Ubuntu 14 Trusty Tahr CLI Only
• Ubuntu 14 Trusty Tahr GUI

The above images are also preserved in IU Scholarworks.

Support resources for new and returning users, including a quick start guide and detailed system overview, are available on the Jetstream Public Wiki.

The Request Tracker queue has been set up. Requests can be submitted to the queue by emailing either jetstream@tickets.xsede.org or help@jetstream-cloud.org.

• October - December 2015: The Jetstream PEP and test system acceptance report were both approved by NSF DGA.

Multiple presentations on Jetstream were done at the SC15 conference and were well received.

Acceptance testing on the production systems is ongoing.

LDAP access (using master at TACC) is in testing from Indiana cluster and test cluster. Initial testing is proceeding well.

XRAS allocations for Jetstream's early operations phase are complete. In this initial phase, 4,094,920 SUs were allocated (this total does not include Startup, Education, or Champion allocations).

• July - September 2015: The production systems were ordered, and arrived at IU and TACC. Installation and testing is underway.

The Jetstream test system acceptance report was finalized and sent to the NSF. All tests relevant to the test system exceeded the required performance levels. The production equipment purchase orders were placed and Statement of Work (SoW) finalized with our supplier, Dell. The Jetstream PEP was recommended to NSF DGA for approval and subsequent amending of our cooperative agreement.

The Arizona Atmosphere environment was successfully deployed on the test environment. The network performance tests were completed on the test system, allowing us to make a recommendation for the production order.

IU and UT-Austin/TACC have agreed to operate separate clouds that share a master/slave LDAP published from TACC. This is similar to how Chameleon and Wrangler are deployed. TACC's user management system, TAS, will handle intake of identity and allocation packages from XSEDE and make that information available to the production clouds. We believe we will be able to take advantage of OpenStack Kilo's trust relationships to make most of this work, but this functionality has not been tested yet. We have determined that some updates to policies and operations for the XSEDE User Portal may be needed to work with Jetstream's "try-it-out" micro-allocations.

• Initial timeline milestones:
• June 2015: Place order with Dell for production system
• May 2015: Jetstream test cluster to be configured and tested
• April 2015: Jetstream test cluster delivered and installed in Data Center

Benefits:

From Jetstream:

"In the atmosphere, a jet stream is the border between two different masses of air. The new Jetstream cloud system will operate at the border between the existing NSF-funded cyberinfrastructure and thousands of researchers and research students who will be new to use of NSF XD program resources," said Craig Stewart, PTI executive director and associate dean for research technologies at Indiana University. "Jetstream will give researchers access to cloud computing and data analysis resources interactively, when they need them."

Jetstream will be attractive to communities who have not been users of traditional HPC systems, but who would benefit from advanced computational capabilities. Among those groups are researchers in biology, atmospheric science, observational astronomy, and the social sciences.

"We expect Jetstream to be of particular interest to researchers analyzing 'born digital' data with research needs that are more suited to cloud computing than the traditional supercomputers that have been the mainstay of NSF-funded cyberinfrastructure in the past," Stewart said.

Related information: For news, press releases, and other updates, see the Jetstream website.

Project team: IU's partners in implementing and supporting Jetstream include TACC, the University of Chicago, the University of Arizona, the University of Texas at San Antonio, Johns Hopkins University, Pennsylvania State University, Cornell University, the University of Arkansas at Pine Bluff, the University of Hawaii, the National Snow and Ice Data Center, the Odum Institute at the University of North Carolina, and the National Center for Genome Analysis Support.

Jetstream is supported by a $6.6-million NSF grant (Award #ACI-1445604).

Aids achievement of the following Empowering People Actions:

• Recommendation A1
• Action 4: Cyberinfrastructure. IU should continue to advance its local cyberinfrastructure, its participation in national cyberinfrastructure, and its efforts to win federal funding for cyberinfrastructure programs that enhance IU's research capabilities.
• The creation of Jetstream will enhance the national cyberinfrastructure with a first-of-its-kind science and engineering cloud and lead to further cyberinfrastructure development within IU and on the national level.
• Action 5: Philosophy of abundance. IU should pursue strategies that approximate a philosophy of abundance, within reason, towards unmetered availability of basic IT services, support, and infrastructure for creative activity, storage, computation, communication, and other activities fundamental to the work of the university via any appropriate sourcing strategy.
• The participation in XSEDE and the creation of Jetstream creates local expertise that allows UITS to aid IU researchers in writing high-quality proposals, thus enabling a local abundance of supercomputing resources by adding availability of XSEDE supercomputers to IU systems, such as Big Red II and Karst.
• Action 6: Leveraging partnerships. IU should continue its highly successful program of relationships with hardware, software, and services vendors, and seek additional partnerships and creative exchanges that provide mutual benefits.
• Participation in XSEDE identifies IU as among a very small number of key leaders in the US cyberinfrastructure community and aids our partnerships with vendors.
• Recommendation A4
• Action 16: External funding. OVPIT should continue to lead and expand its efforts to effectively partner with academic units, campuses, administrative units, or individual investigators for external funding opportunities.

• Recommendation A7
• Action 24: World-class IT staff. IU should remain competitive with regard to compensation, benefits, facilities, workplace climate, and quality of life offerings through funding choices to attract, develop, and retain the very best technical and professional staff.
• Several of UITS' world-class staff are or will be funded by the Jetstream award.

• Recommendation B8
• Action 27: Human-centered support. IU should continue to pioneer and provision effective means of user support through advanced tools for self-service and connection to IU experts to help faculty, staff, and students effectively use IT. IU should continue its work as a support infrastructure provider for national research projects and services.
• IU's Jetstream award funds the self-service aspect of support in the form of user documentation. Additional support will be provided through the XSEDE project and sub-award.

• Recommendation C15
• Action 70: IT-enabled research. IU should purposefully select areas of great and timely promise for strategic development of IT-enabled research, scholarship, and/or creative activity.
• Action 71: IT-enabled research resources. IU should identify a base of resources to provide both initial and sustained investments in selected areas for IT-enabled research, scholarship, and/or creative activity. This may include reallocating current resources and developing new ones, including endowments, grants, and/or additional fees.

Project team: All Research Technologies staff are involved directly or indirectly in the XSEDE project.

Governance:

• Craig Stewart (Associate Dean of Research Technologies, Executive Director of the Pervasive Technology Institute) is the PI on this award.
• Therese Miller (Manager, Collaboration and Engagement Support, UITS Research Technologies) is the project manager for IU's participation in Jetstream.