Project: One Degree Imager - Portal, Pipeline, and Archive (ODI-PPA) (EP Action Item 6)

Primary UITS contact: Arvind Gopu

Last update: July 4, 2016

Description: Since 2009, the Pervasive Technology Institute (PTI) in partnership with the Research Technologies (RT) division of UITS has been collaborating on the One Degree Imager - Portal, Pipeline, and Archive (ODI-PPA) Science Gateway project with the Wisconsin, Indiana, Yale, and National Optical Astronomy Observatory (WIYN) Consortium (WIYN, Inc.), an astronomy consortium based in Tucson, AZ.

The ODI-PPA Science Gateway provides a uniform web portal interface for astronomers to view their data, search and filter for data and metadata, analyze existing data products, and submit/monitor data processing jobs. It also provides a modular framework to incorporate advanced data reduction and analysis tools, and uses IU's computing, mass storage, and networking resources.

The ODI instrument uses a new detector technology called orthogonal transfer arrays (OTAs). This technology enables real-time corrections for blurring due to the Earth's atmosphere and telescope motion, and also provides up to a 25% improvement in resolution compared to a static imager. The ODI instrument's original design featured an image sensor with 64 OTA detectors, and more than one billion pixels, providing imaging over an extremely large field of view (one degree across, an area four times that of the full moon) at very high spatial resolution. A partially populated focal plane ODI with 13 OTA detectors, designated pODI, was installed and commissioned by WIYN in fall 2012, and has been in science operations since March 2013. This was expanded to 30 OTA detectors in spring/summer 2015.

Outcomes: The ODI-PPA system has been operational at a production level since March 2013. The ODI-PPA team has been actively developing the product to its full feature set since April 2012. See this page for a list of papers and posters about this project.

We have made several production releases, initially to aid commissioning of the instrument, and subsequently to enable observers to view, download, and analyze their data. In the early stages of development, several user testing cycles were executed in July, August, and September 2012, and January, February, April, and summer 2013. Deployment and operation of the calibration pipeline and the QuickReduce pipeline has been ongoing since summer 2013. In February 2012, our Science Gateway design and plan for development was reviewed favorably by a team of external reviewers. Prior to that, in October 2010, a proposal we had submitted was reviewed favorably by almost the same team of external reviewers convened by WIYN.

Milestones and status:

  • Proposal submitted for review by external committee convened by WIYN Completed August 2010
  • Proposal Review Completed October 2010
  • User engagement workshops Completed April-June 2011
  • Critical Design Review Completed February 2012
  • End-user Test Release 1, 2, and 3 Completed July, August, and September 2012, respectively
  • PPA Commissioning Release Completed October 2012
  • ODI-PPA paper about "FRamework for web-based Interactive Astronomy Analysis" (FRIAA) presented in Astronomical Data Analysis Software and Systems (ADASS XXII) held in Urbana, IL Completed November 2012
  • End-user Test Release 4 Completed January 2013
  • Second Commissioning Release Completed January 2013
  • Pre-Operational End-user Test Release Completed mid-February 2013
  • First Major Operational Production Release Completed mid-March 2013
  • Second Major Operational Production Release with fully integrated Calibration Pipeline Completed May 2013
  • ODI-PPA booth in American Astronomical Society conference (AAS 222) held in Indianapolis Completed June 2013
  • Tier 2 functionality and user workspaces: Initial integration of QuickReduce pipeline provided for user-driven execution Completed September 2013
  • ODI-PPA papers about Image Explorer and Source Explorer presented in Astronomical Data Analysis Software and Systems (ADASS XXIII) held in Waikoloa, HI Completed early October 2013
  • Tier 2 functionality and user workspaces: Deployment of source overlay feature to production Completed November 2013
  • Tier 2 functionality: Integrate QuickReduce pipeline on Big Red II supercomputer Completed January 2014
  • Tier 2 functionality: Advanced features for previously integrated QuickReduce pipeline:
    • For user-driven execution Completed January 2014
    • For operator-driven execution Completed February 2014
  • Tools to analyze:
    • Long-term trends of the instrument (operator) Completed May 2014
    • Collection of images (user) Completed early May 2014
  • Tier 2 functionality: Integration of SWARP stacking code:
    • For user-driven execution Completed March 2014
    • For user- or operator-driven execution to produce multiple stacks via a single job (initial iteration) Completed June 2014
    • For operator-driven execution Summer 2015 (delayed)
  • Paper summarizing overall ODI-PPA architecture and poster about QuickReduce integration presented at SPIE 2014 conference in Montreal, Canada Completed June 22-25, 2014
  • ADASS 2014 conference:
    • Submitted three abstracts Completed July 2014
    • Submitted and presented two papers (1, 2) and one poster at ADASS 2014 conference in Calgary October 2014
  • Draft and sign short-term MOU for February through September 2015 Completed January 2015
  • Open source (license) and release PPA code February 2015
  • Present ODI-PPA's larger umbrella project at "Big Data in Astronomy" workshop in Tucson, AZ Completed March 2015
  • Adjust and confirm scaling of ODI-PPA to 5x6 focal plane data, including ability to ingest incoming raw 5x6 images, produce master calibrations, and execute QuickReduce pipelines to produce reduced, science-quality 5x6 images Completed April 2015
  • Image Explorer improvements:
    • Enhanced encoding/decoding algorithms Completed March 2015
    • New client-side Image Statistics and Cross-cut tools to replace legacy server-side, code-based features Completed June 2015
  • Image Explorer Completed June 2015
    • Enhanced encoding/decoding algorithms Completed March 2015
    • Image Statistics Completed June 2015
    • Cross-cut Completed June 2015
    • ImExamine function Spring/summer 2016
  • Put together Docker container for QuickReduce pipeline to enable complete re-run capability of older version of pipeline
    • Prototyping Completed early August 2015
  • Guide Stream video-related tasks
    • Ingest Guide Stream data for ODI 5x6 Completed September 2015
    • Produce and display additional Quality Assurance plots Summer 2016
  • Draft operational service agreement for October 2015 and beyond Completed August 2015
    • Coordinate signing of operational service agreement for October 2015-September 2018 Completed September 2015
  • Comprehensive ingestion of NOAO pipeline (AuCaP 1.0) based 3x3 pODI data products Completed October 2015
  • Updates to integrated QuickReduce pipeline workflow to allow twilight flats, and other recent enhancements and minor usability improvements Completed November 2015
  • Improved search menu options for novice users (showing, for example, "Best Available Data") Completed November 2015
  • Submit abstracts to SPIE2016 conference Completed December 2015
  • Fully refactored QuickReduce pipeline workflow to allow advanced customizations and improved data provenance associations - Summer 2016
  • Image Explorer v2 Summer 2016 (making steady progress)
  • Tier 2 functionality: User execution of QuickReduce pipeline and SWARP together to produce a dither stack right after a dither sequence is taken on the instrument Summer 2016
  • Implement features not already in place, and write/design three papers and one poster at the SPIE 2016 conference Completed June 2016
  • User outreach workshop to be held at IU - August 2016
  • Fully refactored QuickReduce pipeline workflow to allow advanced customizations and improved data provenance associations - Summer 2016
  • Image Explorer v2 - Summer 2016 (Making steady progress)
  • Tier 2 functionality: User execution of QuickReduce pipeline and SWarp together to produce a dither stack right after a dither sequence is taken on the instrument - Summer 2016

Comment process: Email general comments to Arvind Gopu.

Software used: Zend+PHP, BootStrap, Highcharts, AngularJS, and jQuery JavaScript libraries, Python, GraphicsMagick; integrated applications: IU PTI-based Image Explorer and Source Explorer, University of Wisconsin-based QuickReduce pipeline

Client impact: In the past, imaging of very faint objects, better discrimination between stars and distant galaxies, and delineation of detailed features of objects such as star-forming regions and nebulae have only been possible with space-based telescopes. Starting with ODI and similar instruments, such capabilities will be available to ground-based telescopes as well.

The ODI-PPA Science Gateway provides an advanced web portal integrated with a massive disk cache, tape archive, and grid/cluster computing resources, as well as a modular framework to incorporate advanced data reduction and analysis tools. ODI-PPA will radically transform the way ground-based telescopes are used, and will help ODI achieve its full potential.

Aids achievement of the following Empowering People actions:

  • Recommendation A1
    • Action 4: Cyberinfrastructure. IU should continue to advance its local cyberinfrastructure, participation in national cyberinfrastructure, and its efforts to win federal funding of cyberinfrastructure programs that enhance IU's research capabilities.
    • Action 6: Leveraging partnerships. IU should continue its highly successful program of relationships with hardware, software, and services vendors, and seek additional partnerships and creative exchanges that provide mutual benefits.
  • Recommendation A4
    • Action 16: External funding. OVPIT should continue to lead and expand its efforts to effectively partner with academic units, campuses, administrative units, or individual investigators for external funding opportunities.
  • Recommendation B9
    • Action 33: Research data utility. IU should provision a data utility service for research data that affords abundant near- and long-term storage, ease of use, and preservation capabilities. (RT leads)
  • Recommendation C15
    • Action 70: IT-enabled research. IU should purposefully select areas of great and timely promise for strategic development of IT-enabled research, scholarship, and/or creative activity. (PTI leads)
    • Action 71: IT-enabled research resources. IU should identify a base of resources to provide both initial and sustained investments in selected areas for IT-enabled research, scholarship, and/or creative activity. This may include reallocating current resources and developing new ones, including endowments, grants, and/or additional fees. (RT leads)

UITS project team:

  • Arvind Gopu, Project Manager
  • Soichi Hayashi, Developer
  • Michael Young, Developer
  • Ralf Kotulla, Collaborator/Affiliate Developer
  • Thomas Lee, Hardware Maintenance Consultant

Former team members:

  • Jeffrey Cox, Former Developer
  • Raminder Singh, Former Developer
  • Marlon Pierce, Former Workflow Design Consultant
  • Suresh Marru, Former Developer
  • Yiming Sun, Former Developer

Governance:

  • UITS:
    • Craig Stewart, Associate Dean of Research Technologies, Executive Director of the Pervasive Technology Institute (PTI)
    • Matthew Link, Director, Systems (Research Technologies)
    • Robert Henschel, Manager, Scientific Applications and Performance Tuning (Research Technologies)
  • IU Astronomy:
    • John Salzer, Department Chair
    • Catherine Pilachowski, Daniel Kirkwood Chair
    • Katherine Rhode, Associate Professor
    • Eileen Friel, Professor

This is document bblr in the Knowledge Base.
Last modified on 2017-05-11 15:21:23.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.