Forward resource limits from your shell to your batch job on Big Red II at IU

Note:

Big Red II will be retired from service on December 15, 2019. After that date, you will no longer be able to log into Big Red II; however, the data in your Big Red II home directory will remain accessible from your home directory on any of the other IU research supercomputers. Requests for new software installations on Big Red II are currently being redirected to Big Red 3 unless the requested software has requirements specific to Big Red II.

IU graduate students, faculty, and staff can create Big Red 3 accounts using the instructions in Get additional IU computing accounts. Undergraduate students and affiliates can get Big Red 3 accounts if they are sponsored by full-time IU faculty or staff members; see About Big Red 3, Big Red II, RDC, and SDA accounts for undergraduate students and sponsored affiliates at IU. Grand Challenges users who create Big Red 3 accounts can submit the Request Access to Specialized HPC Resources form to request exclusive access to a portion of the system for running jobs.

On Big Red II at Indiana University, user resource limits set in the shell (with ulimit in bash, or limit in tcsh) are not automatically forwarded to batch jobs on the Extreme Scalability Mode (ESM) or Cluster Compatibility Mode (CCM) compute nodes.

To enable the forwarding of user resource limits, set the APRUN_XFER_LIMITS runtime environment variable to 1 (one) by adding one of the following lines (depending on the shell) to your job script:

  • In bash, add:
    export APRUN_XFER_LIMITS=1
    
  • In tcsh, add:
    setenv APRUN_XFER_LIMITS 1
    

For example:

  • ESM job (bash shell):
    #!/bin/bash
    #PBS -l nodes=1:ppn=32
    #PBS -l walltime=00:30:00
    #PBS -q cpu
    #PBS -j oe
      
    export APRUN_XFER_LIMITS=1  
    aprun -n 32 ~/my_code
    
  • CCM job (tcsh shell):
    #!/bin/tcsh
    #PBS -l nodes=1:ppn=16
    #PBS -l walltime=00:30:00
    #PBS -l gres=ccm
    #PBS -j oe  
    
    setenv APRUN_XFER_LIMITS 1
    module load ccm
    ccmrun ~/my_code
    

For more about the APRUN_XFER_LIMITS runtime environment variable, see its entries in the aprun and ccmrun manual pages:

man aprun | less -p"APRUN_XFER_LIMITS"
man ccmrum | less -p"APRUN_XFER_LIMITS"

For more on running batch jobs on Big Red II, see:

Support for IU research computing systems, software, and services is provided by various teams within the Research Technologies division of UITS.

For general questions about research computing at IU, contact UITS Research Technologies.

For more options, see Research computing support at IU.

This is document bdvg in the Knowledge Base.
Last modified on 2019-08-27 09:02:45.

Contact us

For help or to comment, email the UITS Support Center.