Quarry usage policies
On this page:
- Accounts
- Home directories
- Computational resources (queues)
- CPU limits and batch jobs
- Mail usage
- Scheduled downtime
- Questions and comments
Accounts
Quarry serves multiple communities as a research cluster and as a general purpose Linux environment. Accounts are open to all Indiana University students, faculty, and staff, as well as affiliated researchers.
Home directories
Quarry home directory disk space is allocated on the IBM N5500 NAS storage device. New accounts have a 10 GB disk quota, which is shared with Big Red and the Research Database Complex (RDC), if you have accounts on those systems; see At IU, how much disk space is available to me on the research systems? If you need additional permanent storage space, apply for an account on IU's Scholarly Data Archive (SDA).
Computational resources (queues)
Each queue has properties defined in order to maximize job throughput and minimize time spent waiting in queue. Quarry uses a TORQUE routing queue to funnel jobs from the default BATCH queue into the SERIAL, NORMAL, and LONG queues. This sorting is done according to the size of the job's resource requests as documented below. By default, your jobs will run in a work queue chosen by the batch queue based on your resource specification unless you specifically submit your jobs to one of the special queues (DEBUG, PG, or OSG).
Note: Cluster-wide, the maximum number of tasks is 2,768 (346 compute nodes available [342 in queues, 4 in user-selectable debug] X 8 tasks per node).
SERIAL queue properties
-
Nodes: 5 serial
(
q029-q033) + 33 normal (q034-q066) + 46 long (q067-q112) + 28 himem (q113-q140) = 112 total - Maximum walltime: 12 hours
- Maximum nodes per job: 1 node
- Maximum cores per job: 8 cores
- Maximum number of jobs per queue: 2,000
- Maximum number of jobs per user: 500
- Direct submission: No
NORMAL queue properties
-
Nodes: 33 normal
(
q034-q066) + 46 long (q067-q112) = 79 total - Maximum walltime: 7 days
- Maximum nodes per job: 6 nodes
- Maximum cores per job: 48 cores
- Maximum number of jobs per queue: 1,500
- Maximum number of jobs per user: 500
- Direct submission: No
LONG queue properties
-
Nodes: 46 long
(
q067-q112) = 46 total - Maximum walltime: 14 days
- Maximum nodes per job: 42 nodes
- Maximum cores per job: 336 cores
- Maximum number of jobs per queue: 596
- Maximum number of jobs per user: 50
- Direct submission: No
DEBUG queue properties
- Nodes: 4 blades dedicated (q025-q028)
- Maximum walltime: 30 minutes
- Maximum nodes per job: 4 nodes
- Maximum cores per job: 32 cores
- Maximum number of jobs per queue: None
- Maximum number of jobs per user: 2
-
Direct submission: Yes (
qsub -q debug)
Note: The DEBUG queue is intended for short, quick-turnaround test jobs requiring less than 30 minutes of wall clock time.
Once code has been debugged, you may submit it to the other work
queues via the normal qsub mechanism. You may allow the
batch queue to dispatch your jobs to an appropriate queue, or select
one of the user-selectable queues below.
HIMEM queue properties
-
Nodes: 28 himem
(
q113-q140) = 28 total - Maximum walltime: 14 days
- Maximum nodes per job: 28 (= maximum in queue)
- Maximum cores per job: 224 (= maximum in queue)
- Maximum jobs per queue: 500
- Maximum jobs per user: 50
-
Direct submission: Yes (
qsub -q himem)
PG queue properties
- Nodes: 230 iDataPlex nodes dedicated (pg1-pg230)
- Maximum walltime: 14 days
- Maximum nodes per job: 230 (= maximum in queue)
- Maximum cores per job: 1,840 (= maximum in queue)
- Maximum number of jobs per queue: 500
- Maximum number of jobs per user: 250
-
Direct submission: Yes (
qsub -q pg)
Note: The PG queue nodes are running RedHat Enterprise Linux 5.6, in contrast to other Quarry nodes, which are running RedHat Enterprise Linux 4.8. If you elect to submit jobs to these nodes, you should start with a single, small job, if possible, to determine if there will be problems running under a new OS release. Also, although the PG nodes are available for general use, priority is given to jobs from the Polar Grid project.
OSG queue properties (group restricted access)
- Nodes: 16 blades dedicated (q009-q024)
- Maximum walltime: 14 days
- Maximum nodes per job: 16 (= maximum in queue)
- Maximum cores per job: 128 (= maximum in queue)
- Maximum number of jobs per queue: None
- Maximum number of jobs per user: None
- Direct submission: Not applicable
Note: Access to the OSG queue is restricted, and jobs are routed to this queue via Globus. They are not part of the normal queue system on Quarry.
CPU limits and batch jobs
User processes on the login nodes are limited to 20 minutes of CPU
time. Processes exceeding this limit are automatically terminated with
no warning. You may run interactive jobs requiring more than 20
minutes of CPU time on nodes b005, b006,
b007, and b008. There is a 24-hour CPU time
limit on all user processes running on these nodes. If you require
more than 24 hours of CPU time, submit a batch job to the TORQUE batch
queuing system (also called PBS) with the qsub
command.
Mail usage
UITS does not provide a production mail service on the
Quarry cluster; however, TORQUE communicates via email. Quarry will
send email about your jobs to the address specified in the
~/.forward file in your home directory. (Note the period
[ . ] preceding the filename.) By default, this
is the email address you provided when you requested your account.
If you'd like to change this email address, enter a command similar
to the following, replacing username@host.com with your
email address:
Be sure to use a valid email address; if you do not, you will not be notified about the status of your jobs. For help, see How do I forward my mail from a Unix account?
Scheduled downtime
The Quarry maintenance window is scheduled for the first Tuesday of each month. Notice of any emergency downtime will be posted on the IT Notices web page.
Questions and comments
The policies described in this document were established with the goal of providing a stable, powerful, and efficient research computing environment for IU faculty, graduate students, and staff. If you have any questions or comments about these policies, UITS welcomes your input; email High Performance Systems.
Last modified on April 28, 2011.







