About Karst at Indiana University

Karst will be retired from service in December of 2020. As of October 12, 2020, no new Karst accounts are being created. You will not be able to log into Karst after December 18, 2020, but the data in your Karst home directory will remain accessible from the other IU research supercomputers until December 31, 2021. For more, including information about Quartz, Karst's replacement system, see About the Karst retirement.

On this page:

System overview

Karst at Indiana University is a high-throughput computing cluster designed to deliver large amounts of processing capacity over long periods of time. Karst provides batch processing and node-level co-location services that make it well suited for running high-throughput and data-intensive parallel computing jobs.

Karst's system architecture provides the advanced performance needed to accommodate high-end, data-intensive applications critical to scientific discovery and innovation. Compute nodes have 32 GB of RAM and 250 GB of local disk storage. Data nodes have 64 GB of RAM and 24 TB of local storage. All nodes are housed in the IU Bloomington Data Center, run Red Hat Enterprise Linux (RHEL) 6.x, and are connected via 10-gigabit Ethernet to the IU Science DMZ.

The colocation service previously offered on Karst is now offered on Carbonate; for more, see About Carbonate at Indiana University.

System access

Access is available to IU students, faculty, staff, and sponsored affiliates. For details, see the Research system accounts (all campuses) section of Computing accounts at IU.

Once your account is created, you can use any SSH2 client to access karst.uits.iu.edu. Log in with your IU username and passphrase.


HPC software

The Research Applications and Deep Learning (RADL) group, within the Research Technologies division of UITS, maintains and supports the high performance computing (HPC) software on IU's research supercomputers. To see which applications are available on a particular system, log into the system, and then, on the command line, enter module avail.

For information about adding packages to your user environment, see Use Modules to manage your software environment on IU's research supercomputers.

To request software, submit the HPC Software Request form.

Set up your user environment

On the research supercomputers at Indiana University, the Modules environment management system provides a convenient method for dynamically customizing your software environment.

For more about using Modules to configure your user environment, see Use Modules to manage your software environment on IU's research supercomputers.

File storage options

Before storing data on this system, make sure you understand the information in the Work with data containing PHI section (below).

Work with data containing PHI

The Health Insurance Portability and Accountability Act of 1996 (HIPAA) established rules protecting the privacy and security of individually identifiable health information. The HIPAA Privacy Rule and Security Rule set national standards requiring organizations and individuals to implement certain administrative, physical, and technical safeguards to maintain the confidentiality, integrity, and availability of protected health information (PHI).

This UITS system or service meets certain requirements established in the HIPAA Security Rule thereby enabling its use for work involving data that contain protected health information (PHI). However, using this system or service does not fulfill your legal responsibilities for protecting the privacy and security of data that contain PHI. You may use this system or service for work involving data that contain PHI only if you institute additional administrative, physical, and technical safeguards that complement those UITS already has in place.

For more, see Your legal responsibilities for protecting data containing protected health information (PHI) when using UITS Research Technologies systems and services.

Although PHI is classified as Critical data, other types of institutional data classified as Critical are not permitted on Research Technologies systems. For help determining which institutional data elements classified as Critical are considered PHI, see About protected health information (PHI) data elements in the classifications of institutional data.

If you have questions about securing HIPAA-regulated research data at IU, email securemyresearch@iu.edu. SecureMyResearch provides self-service resources and one-on-one consulting to help IU researchers, faculty, and staff meet cybersecurity and compliance requirements for processing, storing, and sharing regulated and unregulated research data; for more, see About SecureMyResearch. To learn more about properly ensuring the safe handling of PHI on UITS systems, see the UITS IT Training video Securing HIPAA Workflows on UITS Systems. To learn about division of responsibilities for securing PHI, see Shared responsibility model for securing PHI on UITS systems.

Run jobs on Karst

Karst uses the TORQUE resource manager (based on OpenPBS) and the Moab Workload Manager to manage and schedule batch jobs. Moab uses fairshare scheduling to track usage and prioritize jobs.

The default wall time for jobs running on Karst compute nodes is 30 minutes; the default virtual memory per job is 8 MB.

User processes on the login nodes are limited to 20 minutes of CPU time. Processes on the login nodes that run longer than 20 minutes are terminated automatically (without warning). If your application requires more than 20 minutes of CPU time, submit a batch job or an interactive session using the TORQUE qsub command.

Because of this limit:

  • When running Java programs, add the -Xmx parameter (values must be multiples of 1,024 greater than 2 MB) on the command line to specify the Java Virtual Machine (JVM) maximum heap size. For example, to run a Java program (for example, Hello_DeathStar) with a maximum heap size of 640 MB , on the command line, enter:
      java -Xmx640m Hello_DeathStar
  • You should perform any debugging or testing by running interactive jobs on the compute nodes. Specify the DEBUG queue for short, quick-turnaround test jobs (maximum wall time: 30 minutes) or the INTERACTIVE queue for longer jobs (maximum wall time: 8 hours).

Submit jobs

Use the TORQUE qsub command to submit non-interactive or interactive batch jobs for execution on Karst's compute nodes:

  • Non-interactive jobs: To run a job in batch mode on Karst, first prepare a TORQUE job script that specifies the application you want to run and the resources required to run it, and then submit it to TORQUE with the qsub command.

    Do not specify a destination queue in your job script or your qsub command. TORQUE passes your job to the system's default routing queue (BATCH) for placement, based on its resource requirements, in the SERIAL, NORMAL, or LONG execution queue. From there, your job dispatches whenever the required resources become available.

    If your job has resource requirements that are different from the defaults (but not exceeding the maximums allowed), specify them either with TORQUE directives in your job script, or with the -l (a lower-case "L"; short for resource_list) option in your qsub command.

    Command-line options override PBS directives in your job script.

    On the command line, you can specify multiple attributes with either one -l switch followed by multiple comma-separated attributes, or multiple -l switches, one for each attribute. For example, to submit a job (e.g., death_star.script) that requires 24 hours of wall time (instead of the default 30 minutes) and 10 GB of virtual memory to run on four cores on one node, you may enter either of the following commands (they are equivalent):

      qsub -l nodes=1:ppn=4,vmem=10gb,walltime=24:00:00 death_star.script
      qsub -l nodes=1:ppn=4 -l vmem=10gb -l walltime=24:00:00 death_star.script
    Each Karst compute node has 16 CPU cores available, allowing you to "stack" up to 16 jobs on one compute node (provided none of the jobs has unique resource constraints, such as special memory requirements). To run up to 16 jobs "stacked" on a single compute node, specify -l nodes=1:ppn=1 in the script or qsub command for each job.
  • Interactive jobs: To submit an interactive job, use qsub with the -I (to specify an interactive job) and -q interactive (to specify submission to the INTERACTIVE batch queue) switches; for example:
      qsub -I -q interactive -l nodes=1:ppn=4,vmem=10gb,walltime=4:00:00

    Submitting your job to the INTERACTIVE queue directs it to a specific set of nodes that are configured for shared access (versus single-user in the general batch queues). Consequently, your interactive job most likely will dispatch faster in the INTERACTIVE queue than in the general execution queues.

Useful qsub options include:

Option Function
-a <date_time> Execute the job only after specified date and time (<date_time>).
-I Run the job interactively. (Interactive jobs are forced to be not re-runnable.)
-m e Mail a job summary report when the job terminates.
-q <queue_name> Specify the destination queue (<queue_name>) for the job. (On Karst, use this only for Submit jobs to the DEBUG, INTERACTIVE, or PREEMPT queues.)
-r [y|n] Declare whether the job is re-runnable. Use the argument n if the job is not re-runnable. The default value is y (re-runnable).
-V Export all environment variables in your current environment to the job.

For more, see the qsub manual page.

Monitor jobs

To monitor the status of a queued or running job, use the TORQUE qstat command.

Useful qstat options include:

Option Action
-a Display all jobs.
-f Write a full status display to standard output.
-n List the nodes allocated to a job.
-r Display jobs that are running.
-u user1,user2 Display jobs owned by specified users.

For more, see the qstat manual page.

Delete jobs

To delete a queued or running job, use the qdel command.

Occasionally, a node will become unresponsive and unable to respond to the TORQUE server's requests to kill a job. In such cases, try using qdel -W <delay> to override the delay between SIGTERM and SIGKILL signals; for <delay>, specify a value in seconds.

For more, see the qdel manual page.

Queue information

Karst employs a default routing queue that funnels jobs, according to their resource requirements, into three execution queues configured to maximize job throughput and minimize wait times (the amount of time a job remains queued, waiting for required resources to become available). Depending on the resource requirements specified in either your batch job script or your qsub command, the routing queue (BATCH) automatically places your job into the SERIAL, NORMAL, or LONG queue.

The maximum wall time allowed for jobs running on Karst is 14 days. If your job requires more than 14 days of wall time, email the High Performance Systems group for assistance.

You do not have to specify a queue in your job script or in your qsub command to submit your job to one of the three batch execution queues; your job will run in the SERIAL, NORMAL, or LONG queue unless you specifically submit it to the DEBUG, PREEMPT, or INTERACTIVE queue, the properties of which are as follows:

  • DEBUG: The DEBUG queue is intended for short, quick-turnaround test jobs requiring less than 1 hour of wall time.
    Maximum wall time: 1 hour
    Maximum nodes per job: 4
    Maximum cores per job: 64
    Maximum number of jobs per queue: None
    Maximum number of jobs per user: 2
    Direct submission: Yes

    To submit a batch job to the DEBUG queue, either add the #PBS -q debug directive to your job script, or enter qsub -q debug on the command line.

    For longer debugging or testing sessions, submit an interactive job to the INTERACTIVE queue instead.
  • INTERACTIVE: Interactive jobs submitted to the INTERACTIVE queue should experience less wait time (start sooner) than interactive jobs submitted to the batch execution queues.
    Maximum wall time: 8 hours
    Maximum nodes per job: None
    Maximum cores per job: 8
    Maximum number of jobs per queue: 128
    Maximum number of jobs per user: 16
    Direct submission: Yes

    To submit an interactive job to the INTERACTIVE queue, on the command line, enter qsub with the -I and -q interactive options added; for example:

      qsub -I -q interactive -l nodes=1:ppn=1,walltime=4:00:00
    If you enter qsub without the -q interactive option, your interactive job will be placed in the routing queue for submission to the SERIAL, NORMAL, or LONG batch execution queue, which most likely will entail a longer wait time for your job.
  • PREEMPT: Jobs submitted to the PREEMPT queue run on "condominium nodes" owned by members of the "condominium computing" service; however, when a job submitted by a "condominium node" owner is ready to dispatch (and no other nodes are available), the non-condominium job with the lowest accrued wall time will be preempted. Consequently, non-condominium jobs in the PREEMPT queue may dispatch multiple times before running to completion.
    Maximum wall time: 14 days
    Maximum nodes per job: None
    Maximum cores per job: None
    Maximum number of jobs per queue: 1,800
    Maximum number of jobs per user: 200
    Direct submission: Yes

    To submit a job to the PREEMPT queue, add the #PBS -q preempt directive to your job script, or enter qsub -q preempt on the command line.

To see current status information for the work queues on Karst, on the command line, enter:

  qstat -q
To best meet the needs of all research projects affiliated with Indiana University, UITS Research Technologies administers the batch job queues on IU's research supercomputers using resource management and job scheduling policies that optimize the overall efficiency and performance of workloads on those systems. If the structure or configuration of the batch queues on any of IU's research supercomputers does not meet the needs of your research project, contact UITS Research Technologies.

Acknowledge grant support

The Indiana University cyberinfrastructure, managed by the Research Technologies division of UITS, is supported by funding from several grants, each of which requires you to acknowledge its support in all presentations and published works stemming from research it has helped to fund. Conscientious acknowledgment of support from past grants also enhances the chances of IU's research community securing funding from grants in the future. For the acknowledgment statement(s) required for scholarly printed works, web pages, talks, online publications, and other presentations that make use of this and/or other grant-funded systems at IU, see Sources of funding to acknowledge in published work if you use IU's research cyberinfrastructure

Get help

For an overview of Karst documentation, see Get started on Karst.

Support for IU research supercomputers, software, and services is provided by various teams within the Research Technologies division of UITS.

For general questions about research computing at IU, contact UITS Research Technologies.

For more options, see Research computing support at IU.

This is document bezu in the Knowledge Base.
Last modified on 2020-08-10 09:32:25.

Contact us

For help or to comment, email the UITS Support Center.