ARCHIVED: Completed project: Electron Microscopy Center - Portal, Pipeline, and Archive (EMCenter-PPA) (EP Action Item 6)

This content has been archived, and is no longer maintained by Indiana University. Information here may no longer be accurate, and links may no longer be available or reliable.

Primary UITS contact: Arvind Gopu

Completed: September 3, 2014

Description: The EMCenter facility at Indiana University's Biology department enables electron microscopy of frozen, hydrated specimens at very high resolution. The imaging data is collected and stored temporarily on the Data Capacitor wide-area network (DC-WAN), and then archived to the Scholarly Data Archive (SDA), once per night. The raw microscope data is usually run through a sequence of initial processing steps to identify objects of interest and produce a data cube. Data cubes are then run through data processing software, producing output that can be used in scientific analysis. Both of these processing steps are typically executed on IU's Quarry research computing cluster. The data acquisition and processing are led by David Gene Morgan (IU Biology department) and his team. EMCenter users also use a few standard desktop visualization applications to interactively analyze their data.

The computing expertise of the scientists who use EMCenter's microscope spans a wide spectrum, from those who are comfortable with supercomputers and batch schedulers to more novice users. Currently, all users who take data on EMCenter's microscopes are required to get a Quarry (and, in some cases, SDA) account to access or stage data, or to perform subsequent processing steps. Downloading, staging, and loading a large amount of data for processing and analysis can be difficult. Even though previously observed/processed data are safely stored on the SDA, access is a challenge, especially for those outside of David's team. There is only minimal search capability for archived data or metadata, and David is responsible for retrieving previously archived data using a manual process.

The Scientific Applications and Performances Tuning (SciAPT) team within the Research Technologies division of UITS adapted the framework designed and developed for the One Degree Imager - Portal, Pipeline, and Archive (ODI-PPA) project, and deployed a system similar to ODI-PPA to serve as a single point of access for EMCenter data including search, browsing, download, data processing on IU clusters, and interactive visualization. This enables more seamless and robust access to IU cyberinfrastructure for EMCenter user. The SciAPT team proceeded in two phases:

  • Phase I: Initially adapt the data ingestion and most basic portal functions, allowing EMCenter users to search and download their data and perform basic Image Explorer visualization and analysis
  • Phase II: Further adapt the data processing and more advanced visualization frameworks, depending on the results of previous work

Outcomes: After initial discussions with David Morgan's group, the SciAPT team developed a prototype solution that was demonstrated at a conference booth in June 2013. An active development phase began shortly thereafter and led to testing cycles in fall 2013, followed by operational deployment.

Milestones and status:

  • Phase I:
    • Prototype (presented demo at AAS conference) Completed June 2013
    • RT standard operating procedure tasks Completed July-August 2013
      • Proposal to RT leadership
      • Added separate line item for effort reporting
      • Added project to UITS projects page
    • Active development September-October 2013 (delayed)
    • Testing mid-November 2013-mid-February 2014 (delayed but completed February 2014)
    • Operations (if testing is successful) Completed June 2014* (deployment previously delayed)
      • Retroactive data uploads Ongoing summer 2014
    • Conference abstract by David Morgan, et al. Accepted June 2014; presentation September 2014
  • Phase II:
    • Requirements gathering Completed early June 2014 (previously delayed)
    • Active development Suspended until additional resources are identified
    • Testing Suspended until additional resources are identified
    • Operations (if successful) Suspended until additional resources are identified

* Scholarly highlight in fall 2013, and once every year after that.

Comment process: Email general comments to Arvind Gopu.

Software used: Z+PHP, BootStrap and jQuery JavaScript libraries, Java, Python, GraphicsMagick

Transition from pilot to production: The SciAPT team received periodic feedback from David Morgan et al., starting with the prototype and later with the test release (in Phase I). The service stack was deployed on Intelligent Infrastructure (II) VMs, while a separate test and development platform service stack for new features, bug fixes, and OS updates continued to be deployed on SciAPT VMs (to keep costs low). The SciAPT team worked on a service level agreement (SLA) with EMCenter over the course of the past year. Bug fixes or code updates requiring the SciAPT team's participation are tracked using the Request Tracker ticketing system. EMCenter still plans to explore the possibility of getting feedback from its Research Advisory Committee and/or via user surveys. System and environmental updates, including operating system/kernel patches verified on the test platform before deployment to the production platform, are still handled by Arvind Gopu, but are expected to be handled by David Morgan or a designated representative within EMCenter in the future (this will be worked out as the SLA document evolves, outside of this project page). The SciAPT team will provide EMCenter with adequate documentation describing the operation of the service stack.

Client impact: The EMCenter-PPA Science Gateway should provide EMCenter users with seamless and robust access to IU cyberinfrastructure, including an advanced web portal transparently integrated with a massive disk cache, tape archive, and high-performance computing resources, and a modular framework to incorporate advanced data reduction and analysis tools. By radically transforming the way the electron microscopes are used, it should help the EMCenter achieve its full potential.

Aids achievement of the following Empowering People actions:

  • Recommendation A1
    • Action 4: Cyberinfrastructure. IU should continue to advance its local cyberinfrastructure, participation in national cyberinfrastructure, and its efforts to win federal funding of cyberinfrastructure programs that enhance IU's research capabilities.
    • Action 6: Leveraging partnerships. IU should continue its highly successful program of relationships with hardware, software, and services vendors, and seek additional partnerships and creative exchanges that provide mutual benefits.
  • Recommendation A4
    • Action 16: External funding. OVPIT should continue to lead and expand its efforts to effectively partner with academic units, campuses, administrative units, or individual investigators for external funding opportunities.
  • Recommendation A5
    • Action 19: Administrative data storage and retention. IU should provision data storage that provides appropriate physical and electronic protection. Sensitive, non-public, and/or important university information should be rigorously governed by policies and processes that ensure appropriate maintenance and retention. (IIA leads; RT supporting)
  • Recommendation A7
    • Action 25: Research into IT. IU should support and pursue research into information technology itself. IT professionals and faculty should seek partnership opportunities for scholarly publication and invention disclosure that document meritorious research and discovery. (PTI leads; RT supporting)
  • Recommendation B9
    • Action 33: Research data utility. IU should provision a data utility service for research data that affords abundant near- and long-term storage, ease of use, and preservation capabilities. (RT leads)
  • Recommendation B11
    • Action 50: K-12 outreach. IU should use its distinct capabilities in education and technology to reach out to K-12 teachers, administrators, and students in ways that further an effective primary through post-secondary (P-20) approach to improve Hoosier education. (LT leads; RT supporting)
  • Recommendation C15
    • Action 70: IT-enabled research. IU should purposefully select areas of great and timely promise for strategic development of IT-enabled research, scholarship, and/or creative activity. (PTI leads)
    • Action 71: IT-enabled research resources. IU should identify a base of resources to provide both initial and sustained investments in selected areas for IT-enabled research, scholarship, and/or creative activity. This may include reallocating current resources and developing new ones, including endowments, grants, and/or additional fees. (RT leads)
    • Action 72: IT research hiring. IU should carefully assess new skills that are necessary to advance promising opportunities as research becomes more IT-intensive. Campus, school, and departmental leaders should help to target some strategic hiring to supply or augment expertise for advanced, IT-enabled research and creative activity. (RT leads)

UITS project team:

  • Arvind Gopu (Project Manager)
  • Soichi Hayashi (Developer)
  • Michael Young (Developer)
  • Thomas Lee (Hardware Maintenance Consultant)


  • UITS: Craig Stewart, Matt Link, Robert Henschel
  • IU EMCenter: David Gene Morgan, Roger Innes

This is document bdmz in the Knowledge Base.
Last modified on 2018-01-18 17:28:56.