ARCHIVED: Information about the 2012 upgrade to Quarry at IU

The High Performance Systems group recently finished upgrading Indiana University's Quarry cluster. Work included a hardware refresh of Quarry's critical management components and an upgrade of its operating system to Red Hat Enterprise Linux version 6 (RHEL 6). Following is a brief outline of the changes. If you encounter problems on Quarry after the upgrade, or have questions, email the High Performance Systems group.

Operating system upgraded to RHEL 6

The operating system and associated packages installed on Quarry was upgraded to RHEL version 6.

TORQUE and Moab upgraded to latest versions

TORQUE and Moab were upgraded to the latest stable versions. Quarry and Mason now run the same versions of TORQUE and Moab to facilitate the running of identical jobs on both clusters.

Kerberos tickets no longer automatically generated

As a result of upgrading to RHEL 6, Kerberos tickets are longer automatically generated at login. To mount the Research File System (RFS), will you need to execute the kinit command before executing the klog command. This behavior is consistent with the behavior on Mason.

SoftEnv replace by Modules

Quarry now uses the Modules package for manipulating user environments. Although Modules is similar in functionality to SoftEnv, one important difference to note is that the soft command is longer available. Mason and the Extreme Science and Engineering Discovery Environment (XSEDE) digital services also use Modules, as will the Big Red II system.

For more, see Use modules to manage your software environment on IU research supercomputers If you encounter a problem using Modules, or need help, email the Scientific Applications and Performance Tuning group.

TORQUE/PBS queue structure changed

Because the entire cluster now runs the same version of RHEL, a separate PG queue is no longer needed. The nodes previously in the PG queue have been folded into the Serial, Normal, Long, and Himem queues. Similarly, the Himem queue is no longer a stand-alone queue. Jobs automatically route to the proper queue based on the number of cores and amount of memory requested. For more, see ARCHIVED: Use TORQUE to submit and manage jobs on high performance computing systems

Condo queue available for condominium computing

A separate Condo queue is available for users who purchase condominium computing nodes on the Quarry cluster. The Condo queue maps to Quarry's Intel E5645 (Westmere/Nehalem) architecture. The regular batch queue maps to Quarry's Intel E5335 (Clovertown) and Intel E5410 (Harpertown) architectures.

Condominium computing users may submit jobs to the regular and Condo queues. For those users, Condo queue fairshare calculations will be separate from non-condominium usage. Jobs that run in the Condo queue will not affect fairshare calculations for jobs submitted to the regular batch queue, and vice versa.

Note: Non-condominium users must submit jobs to the regular batch queue, but system administrators are investigating the possibility of creating a preempt queue to let non-condominium users test their jobs on the higher performance nodes. However, such jobs would be killed whenever there are not enough available condominium nodes to accommodate existing condominium jobs.

Separate login node for condominium computing

A separate condominium computing login node is now available, allowing condominium users to compile programs for the Westmere architecture. Codes compiled for Mason's Westmere architecture are already compatible with Quarry's condominium nodes, but are not backward-compatible with Quarry's Clovertown/Harpertown architecture. However, codes compiled for the Clovertown/Harpertown architectures are forward-compatible with the new Westmere architecture.