What is Moab?

Moab is a job scheduling and management system for use on clusters, supercomputers, and grids. Highly optimized and configurable, Moab can support a large array of scheduling and fairness policies, dynamic priorities, and extensive reservations. Patented by Adaptive Computing, Inc., Moab is widely used to optimize the manageability and efficiency of systems ranging in complexity from clusters of a few processors to multi-teraFLOPS supercomputers. For more, see the Adaptive Computing Moab Workload Manager Administrator Guide.

Following are some common, useful Moab commands:

Moab command Function
showq Display the jobs in the Moab job queue.
(Jobs may be in a number of states; "running" and "idle" are the most common.)
checkjob jobid Check the status of a job (jobid). For verbose mode, add -v (e.g., checkjob -v jobid).
showstart jobid Show an estimate of when your job (jobid) might start.
mdiag -f Show fairshare information.
checknode node_name Check the status of a node (node_name).
showres Show current reservations.
showbf Show intervals and node counts presently available for backfill jobs.
Note:
In older versions of Moab, use diagnose -f instead of mdiag -f to view fairshare information.

For more Moab commands and their descriptions, see the Adaptive Computing Scheduler Commands page.

Back to top

Moab at IU

On Indiana University's Big Red II, Carbonate, Karst, and Mason research systems, Moab serves as the job scheduler for the TORQUE resource manager (based on OpenPBS).

Once a job has been submitted to one of the TORQUE/Loadleveler queues, it may become eligible for dispatch by Moab.

Note:
Mason, Indiana University's large memory computer cluster, will be retired on January 1, 2018. For more, see About the Mason retirement.

Back to top

Moab on XSEDE

To see which Extreme Science and Engineering Discovery Environment (XSEDE) digital services use Moab for scheduling jobs, see On XSEDE compute systems, what applications are used for scheduling jobs?

Back to top

Fairshare scheduling

Fairshare scheduling allows historical resource usage to affect job priority decisions. Administrators can set target usage goals for each user, group, class, or service group. Moab tracks class usage and gives priority to other job classes when one job class exceeds its usage goals.

At IU, Moab's fairshare mechanism records daily usage on Big Red II, Carbonate, Karst, and Mason, and then uses data from the previous seven days to adjust job priorities. A .80 decay factor is applied to reduce the impact of previous data by 20% each day. Each usage class (usually a username) is assigned a goal of 5% usage. When a user exceeds the usage goal, that user's jobs are then given lower scheduling priority.

To display Moab's fairshare scheduling usage table, use the mdiag -f command.

The following example shows that two users on IU's Karst system (vngunray and spalpati) have exceeded their fairshare goals. Consequently, their jobs will be given lower priority over the following week:

  [pamidala@h4]$ mdiag -f
  FairShare Information

  Depth: 7 intervals   Interval Length: 1:00:00:00   Decay Rate: 0.80

  FS Policy: DEDICATEDPS
  System FS Settings:  Target Usage: 0.00    Flags: 0

  FSInterval        %     Target       0       1       2       3       4       5       6
  FSWeight       ------- -------  1.0000  0.8000  0.6400  0.5120  0.4096  0.3277  0.2621
  TotalUsage      100.00 -------  1872.2  1605.8   631.7  1868.0  3222.6  1857.5  1439.1

  USER
  -------------
  quigjinn*         0.00   5.00  ------- ------- ------- ------- ------- ------- -------
  vngunray*        45.91   5.00    81.11   45.57   79.98   49.70    4.88   20.49   10.79
  owkenobi*         0.00   5.00  ------- ------- ------- ------- ------- ------- -------
  jarbinks*         0.00   5.00  ------- ------- ------- ------- ------- ------- -------
  macwindu*         0.25   5.00  ------- ------- -------    0.23    0.74    0.78 -------
  fvalorum*         0.00   5.00  ------- ------- ------- ------- ------- ------- -------
  bossnass*         0.00   5.00  ------- ------- ------- ------- ------- ------- -------
  yoda1999*         0.00   5.00  ------- ------- ------- ------- ------- ------- -------
  spalpati*        39.26   5.00    14.97   43.03 -------   33.74   86.48   62.68 -------

Back to top

Estimating when your job will start

Moab uses fairshare tables to determine which job gets assigned to the next open processors. To view the states of submitted jobs, use the showq command.

Following is sample output from IU's Karst system:

  [plorgana@h4]$ showq
  active jobs--------------------
  JOBID              USERNAME      STATE  PROCS   REMAINING            STARTTIME

  17199              lcalriss    Running     1     2:53:12  Wed Sep 17 11:20:45
  17200              lcalriss    Running     1     2:53:52  Wed Sep 17 11:21:25
  17201              lcalriss    Running     1     2:54:32  Wed Sep 17 11:22:05
  17202              lcalriss    Running     1     2:55:13  Wed Sep 17 11:22:46
  17203              lcalriss    Running     1     2:55:53  Wed Sep 17 11:23:26
  17204              lcalriss    Running     1     2:56:33  Wed Sep 17 11:24:06
  17205              lcalriss    Running     1     2:57:13  Wed Sep 17 11:24:46
  .
  .
  .

  6 active jobs

  eligible jobs----------------------
  JOBID              USERNAME      STATE  PROCS     WCLIMIT            QUEUETIME

  16672              yoda1980       Idle     1  8:08:00:00  Tue Sep 16 23:27:05
  16673              yoda1980       Idle     1  8:08:00:00  Tue Sep 16 23:27:06
  16674              yoda1980       Idle     1 16:16:00:00  Tue Sep 16 23:27:06
  16675              yoda1980       Idle     1 16:16:00:00  Tue Sep 16 23:27:06
  16676              yoda1980       Idle     1  8:08:00:00  Tue Sep 16 23:27:06
  16677              yoda1980       Idle     1  8:08:00:00  Tue Sep 16 23:27:06

  6 eligible jobs

  blocked jobs----------------
  JOBID              USERNAME      STATE  PROCS     WCLIMIT            QUEUETIME

  0 blocked jobs

  Total Jobs: 116 Active Jobs: 104 Eligible Jobs: 6 Blocked Jobs: 0

In the above example, jobs atop the "eligible jobs" list will run next if resources are available. These jobs can be delayed if the resources they need are blocked off by an existing reservation. To view a list of existing reservations, use the showres command.

To view the estimated start time of a job (jobid), use:

  showstart jobid

The following sample shows the estimated start time of a particular job (16672) on IU's Karst system:

  [darvader@h4]$ showstart 16672
  job 16672 requires 1 proc for 8:08:00:00
  Earliest start in       5:03:54:32 on Mon Sep 22 17:00:00
  Earliest completion in 13:11:54:32 on Wed Oct  1 01:00:00
  Best Partition: torque

If a job already has a node or nodes reserved, showstart will display the reservation start time. In all other cases, showstart will display the earliest possible start time, assuming the job in question has the highest priority. In most cases, your job will not have the highest priority, so showstart will provide only an estimate of its start time.

Back to top

Displaying jobs eligible for scheduling

To view the priority of all jobs eligible for scheduling, use:

  showq -i

In the following sample output on IU's Karst system, jobids ending with an asterisk (*) already have reservations, so the showstart data for these jobs will be reasonably accurate:

  [askywalk@h4 ~]$ showq -i

  eligible jobs----------------------
  JOBID              USERNAME      STATE PROCS     WCLIMIT            QUEUETIME
  
  116116             dartmaul       Idle   160     1:00:00  Sat Feb  7 09:57:58
  116117             dsidious       Idle   160     1:00:00  Sat Feb  7 10:03:17
  118873*            pamidala       Idle    32  1:00:00:00  Mon Feb  9 13:18:56
  118874*            quigjinn       Idle    32  1:00:00:00  Mon Feb  9 13:18:59
  118872             dtyranus       Idle    32  1:00:00:00  Mon Feb  9 13:18:56
  118871             vngunray       Idle    32  1:00:00:00  Mon Feb  9 13:18:53
  118867             bobafett       Idle    32  1:00:00:00  Mon Feb  9 13:18:29
  118878             jangfett       Idle    32    12:00:00  Mon Feb  9 13:20:00
  1188758            captypho       Idle    32  1:00:00:00  Mon Feb  9 13:19:38
  .....

  103 eligible jobs
  .....

Back to top

This document was developed with support from National Science Foundation (NSF) grants 1053575 and 1548562. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.

This is document avmu in the Knowledge Base.
Last modified on 2017-07-19 08:44:49.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.