What is Moab?
On this page:
- Moab at IU
- Moab on XSEDE
- Fairshare scheduling
- Estimating when your job will start
- Displaying jobs eligible for scheduling
Moab is a job scheduling and management system for use on clusters, supercomputers, and grids. Highly optimized and configurable, Moab can support a large array of scheduling and fairness policies, dynamic priorities, and extensive reservations. Patented by Adaptive Computing, Inc., Moab is widely used to optimize the manageability and efficiency of systems ranging in complexity from clusters of a few processors to multi-teraFLOPS supercomputers. For more, see the Adaptive Computing Moab Workload Manager Administrator Guide.
Following are some common, useful Moab commands:
||Display the jobs in the Moab job queue.
(Jobs may be in a number of states; "running" and "idle" are the most common.)
||Check the status of a job (
||Show an estimate of when your job (
||Show fairshare information.|
||Check the status of a node (
||Show current reservations.|
||Show intervals and node counts presently available for backfill jobs.|
Note: In older versions of Moab, use
diagnose -f instead of
mdiag -f to view
For more Moab commands and their descriptions, see the Adaptive Computing Scheduler Commands page.
Moab at IU
On Indiana University's Big Red II, Quarry, and Mason research systems, Moab serves as the job scheduler for the TORQUE resource manager (based on OpenPBS).
Once a job has been submitted to one of the TORQUE/Loadleveler queues, it may become eligible for dispatch by Moab.
Note: Following a system-wide upgrade in December 2012, Quarry now runs Red Hat Enterprise Linux version 6 (RHEL 6) and uses the Modules package (instead of SoftEnv) for manipulating user environments. For more, see Information about the 2012 upgrade to Quarry at IU. If you encounter any problems or have questions, email the High Performance Systems group.
Moab on XSEDE
To see which Extreme Science and Engineering Discovery Environment (XSEDE) digital services use Moab for scheduling jobs, see On XSEDE compute systems, what applications are used for scheduling jobs?
Fairshare scheduling allows historical resource usage to affect job priority decisions. Administrators can set target usage goals for each user, group, class, or service group. Moab tracks class usage and gives priority to other job classes when one job class exceeds its usage goals.
At IU, Moab's fairshare mechanism records daily usage on Big Red II, Quarry, and Mason, and then uses data from the previous seven days to adjust job priorities. A .80 decay factor is applied to reduce the impact of previous data by 20% each day. Each usage class (usually a username) is assigned a goal of 5% usage. When a user exceeds the usage goal, that user's jobs are then given lower scheduling priority.
To display Moab's fairshare scheduling usage table, use the
mdiag -f command.
The following example shows that two users on IU's Quarry system
dsheen) have exceeded their
fairshare goals. Consequently, their jobs will be given lower priority
over the following week:
Estimating when your job will start
Moab uses fairshare tables to determine which job gets assigned to
the next open processors. To view the states of submitted jobs, use
Following is sample output from IU's Quarry system:[root@Quarry]# showq active jobs-------------------- JOBID USERNAME STATE PROCS REMAINING STARTTIME 17199 heap Running 1 2:53:12 Wed Sep 17 11:20:45 17200 heap Running 1 2:53:52 Wed Sep 17 11:21:25 17201 heap Running 1 2:54:32 Wed Sep 17 11:22:05 17202 heap Running 1 2:55:13 Wed Sep 17 11:22:46 17203 heap Running 1 2:55:53 Wed Sep 17 11:23:26 17204 heap Running 1 2:56:33 Wed Sep 17 11:24:06 17205 heap Running 1 2:57:13 Wed Sep 17 11:24:46 . . . 6 active jobs eligible jobs---------------------- JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME 16672 ejolson Idle 1 8:08:00:00 Tue Sep 16 23:27:05 16673 ejolson Idle 1 8:08:00:00 Tue Sep 16 23:27:06 16674 ejolson Idle 1 16:16:00:00 Tue Sep 16 23:27:06 16675 ejolson Idle 1 16:16:00:00 Tue Sep 16 23:27:06 16676 ejolson Idle 1 8:08:00:00 Tue Sep 16 23:27:06 16677 ejolson Idle 1 8:08:00:00 Tue Sep 16 23:27:06 6 eligible jobs blocked jobs---------------- JOBID USERNAME STATE PROCS WCLIMIT QUEUETIME 0 blocked jobs Total Jobs: 116 Active Jobs: 104 Eligible Jobs: 6 Blocked Jobs: 0
In the above example, jobs atop the "eligible jobs" list will run
next if resources are available. These jobs can be delayed if the
resources they need are blocked off by an existing reservation. To
view a list of existing reservations, use the
To view the estimated start time of a job (
The following sample shows the estimated start time of a particular
16672) on IU's Quarry system:
If a job already has a node or nodes reserved,
showstart will display the reservation start time. In all
showstart will display the earliest
possible start time, assuming the job in question has
the highest priority. In most cases, your job will not have the
highest priority, so
showstart will provide only an
estimate of its start time.
Displaying jobs eligible for scheduling
To view the priority of all jobs eligible for scheduling, use:showq -i
In the following sample output on IU's Quarry system, jobids ending
with an asterisk (
* ) already have
reservations, so the
showstart data for these jobs will
be reasonably accurate:
This document was developed with support from National Science Foundation (NSF) grant OCI-1053575. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.
Last modified on September 30, 2013.