Quarry usage policies

Username:
Password:

On this page:

Accounts

Quarry serves multiple communities as a research cluster and as a general purpose Linux environment. Accounts are open to all Indiana University students, faculty, and staff, as well as affiliated researchers.

Home directories

Quarry home directory disk space is allocated on the IBM N5500 NAS storage device. New accounts have a 10 GB disk quota, which is shared with Big Red and the Research Database Complex (RDC), if you have accounts on those systems; see At IU, how much disk space is available to me on the research systems? If you need additional permanent storage space, apply for an account on IU's Scholarly Data Archive (SDA).

Computational resources (queues)

Each queue has properties defined in order to maximize job throughput and minimize time spent waiting in queue. Quarry uses a TORQUE routing queue to funnel jobs from the default BATCH queue into the SERIAL, NORMAL, and LONG queues. This sorting is done according to the size of the job's resource requests as documented below. By default, your jobs will run in a work queue chosen by the batch queue based on your resource specification unless you specifically submit your jobs to one of the special queues (DEBUG, PG, or OSG).

Note: Cluster-wide, the maximum number of tasks is 2,768 (346 compute nodes available [342 in queues, 4 in user-selectable debug] X 8 tasks per node).

SERIAL queue properties

Nodes: 5 serial (q029-q033) + 33 normal (q034-q066) + 46 long (q067-q112) + 28 himem (q113-q140) = 112 total
Maximum walltime: 12 hours
Maximum nodes per job: 1 node
Maximum cores per job: 8 cores
Maximum number of jobs per queue: 2,000
Maximum number of jobs per user: 500
Direct submission: No

NORMAL queue properties

Nodes: 33 normal (q034-q066) + 46 long (q067-q112) = 79 total
Maximum walltime: 7 days
Maximum nodes per job: 6 nodes
Maximum cores per job: 48 cores
Maximum number of jobs per queue: 1,500
Maximum number of jobs per user: 500
Direct submission: No

LONG queue properties

Nodes: 46 long (q067-q112) = 46 total
Maximum walltime: 14 days
Maximum nodes per job: 42 nodes
Maximum cores per job: 336 cores
Maximum number of jobs per queue: 596
Maximum number of jobs per user: 50
Direct submission: No

DEBUG queue properties

Nodes: 4 blades dedicated (q025-q028)
Maximum walltime: 30 minutes
Maximum nodes per job: 4 nodes
Maximum cores per job: 32 cores
Maximum number of jobs per queue: None
Maximum number of jobs per user: 2
Direct submission: Yes (qsub -q debug)

Note: The DEBUG queue is intended for short, quick-turnaround test jobs requiring less than 30 minutes of wall clock time.

Once code has been debugged, you may submit it to the other work queues via the normal qsub mechanism. You may allow the batch queue to dispatch your jobs to an appropriate queue, or select one of the user-selectable queues below.

HIMEM queue properties

Nodes: 28 himem (q113-q140) = 28 total
Maximum walltime: 14 days
Maximum nodes per job: 28 (= maximum in queue)
Maximum cores per job: 224 (= maximum in queue)
Maximum jobs per queue: 500
Maximum jobs per user: 50
Direct submission: Yes (qsub -q himem)

PG queue properties

Nodes: 230 iDataPlex nodes dedicated (pg1-pg230)
Maximum walltime: 14 days
Maximum nodes per job: 230 (= maximum in queue)
Maximum cores per job: 1,840 (= maximum in queue)
Maximum number of jobs per queue: 500
Maximum number of jobs per user: 250
Direct submission: Yes (qsub -q pg)

Note: The PG queue nodes are running RedHat Enterprise Linux 5.6, in contrast to other Quarry nodes, which are running RedHat Enterprise Linux 4.8. If you elect to submit jobs to these nodes, you should start with a single, small job, if possible, to determine if there will be problems running under a new OS release. Also, although the PG nodes are available for general use, priority is given to jobs from the Polar Grid project.

OSG queue properties (group restricted access)

Nodes: 16 blades dedicated (q009-q024)
Maximum walltime: 14 days
Maximum nodes per job: 16 (= maximum in queue)
Maximum cores per job: 128 (= maximum in queue)
Maximum number of jobs per queue: None
Maximum number of jobs per user: None
Direct submission: Not applicable

Note: Access to the OSG queue is restricted, and jobs are routed to this queue via Globus. They are not part of the normal queue system on Quarry.

CPU limits and batch jobs

User processes on the login nodes are limited to 20 minutes of CPU time. Processes exceeding this limit are automatically terminated with no warning. You may run interactive jobs requiring more than 20 minutes of CPU time on nodes b005, b006, b007, and b008. There is a 24-hour CPU time limit on all user processes running on these nodes. If you require more than 24 hours of CPU time, submit a batch job to the TORQUE batch queuing system (also called PBS) with the qsub command.

Mail usage

UITS does not provide a production mail service on the Quarry cluster; however, TORQUE communicates via email. Quarry will send email about your jobs to the address specified in the ~/.forward file in your home directory. (Note the period [ . ] preceding the filename.) By default, this is the email address you provided when you requested your account.

If you'd like to change this email address, enter a command similar to the following, replacing username@host.com with your email address:

jdoe@Quarry:~> echo "username@host.com" > ~/.forward

Be sure to use a valid email address; if you do not, you will not be notified about the status of your jobs. For help, see How do I forward my mail from a Unix account?

Scheduled downtime

The Quarry maintenance window is scheduled for the first Tuesday of each month. Notice of any emergency downtime will be posted on the IT Notices web page.

Questions and comments

The policies described in this document were established with the goal of providing a stable, powerful, and efficient research computing environment for IU faculty, graduate students, and staff. If you have any questions or comments about these policies, UITS welcomes your input; email High Performance Systems.

This is document avpl in domain all.
Last modified on April 28, 2011.