Getting started on Big Red

Username:
Password:

On this page:

Introduction

Big Red is one of the most powerful university-owned computers in the US, and one of the 50 fastest supercomputers in the world. Part of a comprehensive strategy to build an advanced cyberinfrastructure to support research at Indiana University, Big Red has a theoretical peak performance of more than 30 teraflops, and has achieved more than 21 teraflops on numerical computations.

Click to see a schematic drawing of Big Red's components and layout.

To manage the multiple users, processors, and jobs running on the system, Big Red uses LoadLeveler to submit and monitor jobs. LoadLeveler relies on the Moab scheduler software for job scheduling, incorporating a fair share mechanism based on research system time used by each user trying to run a job.

The fair share mechanism does not allow users to run jobs on the login node or on the compute node outside of the LoadLeveler job submission system. If you submit a job outside of LoadLeveler and it uses more than 20 minutes of CPU time, it will be terminated. This applies to Globus jobs that use the jobmanager/fork.

Requesting an account or software

TeraGrid allocations: UITS encourages non-IU users to request time on Big Red through the TeraGrid allocation process. See How do I apply for a new TeraGrid allocation?
IU researchers: To create an account on Big red, visit the Account Management Service at: https://itaccounts.iu.edu/
After you log in (using your Network ID username and passphrase), click create more accounts. Follow the instructions to create your Big Red account.
If you are at IU and have an account on one of IU's research systems, you can request software using the High Performance Systems Software Request form.

Connecting and logging in

If you are at IU, you can log into Big Red using your Network ID passphrase. The following login methods are also available for both IU and non-IU researchers:

MyProxy: UITS recommends this option. Use a TeraGrid resource that allows password-based login (e.g., NCSA or SDSC) as a bridge, and then use MyProxy to get a proxy and log into Big Red.
SSH keys: Use SSH keys to log in from your own workstation; see How do I set up authentication with SSH keys on a TeraGrid resource?
Globus certificate: UITS does not recommend this method for new users. Use a default NCSA certificate or another TeraGrid-approved Globus certificate to log in. You will need to have your own installation of Globus Toolkit to use this method.

For more, see What hostname should I use to access Big Red?

The default shell is bash. To change your shell permanently, use the changeshell command:

jdoe@BigRed:~> changeshell This program will assist you in changing your login shell on all nodes of the Big Red cluster. . . . 1) bash 2) tcsh . . . 5) quit Select 1-5: 2 Changing login shell for jdoe Password: Shell changed. Your shell has been changed to the Cornell tcsh shell This will take effect on all nodes within 15 minutes

Note: For TeraGrid users, changeshell will not work. To change your shell on Big Red, email High Performance Systems at Indiana University.

Notes

Windows users: If you use an SSH client in Windows, you cannot open tools that need a graphical user interface (GUI), like the TotalView debugger and the Vampir-NG profiler. You'll need X Window emulation software such as Cygwin. UITS recommends using XLiveCD, which was created by the Research Technologies division of UITS. See How can I run X Window System applications on Unix systems from a Microsoft Windows workstation?
Intra-cluster logins: When you log into your Big Red account for the first time, passphrase-less SSH keys will be automatically created in your home directory. Those keys should enable you to log into compute nodes that you have gained access to through LoadLeveler without entering a password or a passphrase. In other words, parallel jobs should run seamlessly on multiple compute nodes without any manual intervention.
However, you may see the following error message when you try to access LoadLeveler-assigned compute nodes:
Permission denied (publickey,password,keyboard-interactive) This indicates that the intra-cluster RSA key pair in your home directory is either not present or corrupted. If this happens, enter gensshkeys ; it will generate a passphrase-less key pair for you, allowing you seamless intra-cluster logins between any nodes in the cluster assigned for your use by LoadLeveler.
Forwarding email address for job-related messages: Big Red will send email about your jobs to the address specified in the ~/.forward file in your home directory. (Note the period [ . ] preceding the filename.) By default, this is the email address you provided when you requested your account.
If you'd like to change this email address, enter a command similar to the following, replacing username@host.com with your email address:
hpctrn01@BigRed:~> echo "username@host.com" > ~/.forward
Be sure to use a valid email address; if you do not, you will not be notified about the status of your jobs.

Using SoftEnv to set up your software environment

SoftEnv, an environment management system, lets you customize your environment (i.e., specify the software packages you plan to use) using symbolic keywords. For information about using SoftEnv on Big Red, see On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment?

File storage options

You can store files on your home directory or in scratch space, or use a GPFS file system. For more, see At IU, how much disk space is available to me on the research systems?

Compiling and running programs

For information about available compilers and how to use them, see Compiling programs on Big Red at IU.

Parallel applications and message-passing libraries

Big Red is a parallel machine; it is not structured for serial codes. Serial codes waste 75% of the processor cores and the interconnect switch (which account for 1/3 the cost of the machine).

Using message passing

Select message-passing packages using SoftEnv. See On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment?
Note: TeraGrid users get a default MPICH library; no MPI libraries exist in the default environment for IU users.
You must link the library that's consistent with the address precision (32-bit or 64-bit) you chose for the compile.
Once an MPI library is added, compiles are made through a wrapper to the IBM/Gnu compiler that built the library. For instance, mpif90 is actually just a wrapper to xlf90_r. The same switches used by xlf90 are available to mpif90_r.

MPICH

Argonne original
MPICH 1 is available; MPICH 2 could be
Uses mpirun
Limited runtime environment

OpenMPI

Replaced LAM
No more lamboot
Looks like MPICH to the user
Improving with each new release

Submitting batch jobs to LoadLeveler

Writing the script

Job scripts are divided into a keyword stanza and an execution section. If any lines exist in the script other than keywords (including just #!/bin/bash), the script is executed. Otherwise, it is sourced.
It's best to write your job in its own script file and tell LoadLeveler to execute that.
Preface all keywords with #@ .
Keywords used in scripts include: output, error, executable, notification, node_usage, node, job_type, checkpoint, queue

Following is an annotated example of a typical script that you may use as a template:

# Pretty typical file designator. Will take absolute paths or try to write # to your initial directory.; $(Cluster) is unique job id #@ output = test_namd.$(Cluster).out #@ error = test_namd.$(Cluster).err # When do you want the job to send an email? Must put address in your # .forward file. #@ notification = complete # How long to run? The default is hh:mm:ss or you could specify in seconds. #@ wall_clock_limit = 1:00:00 # Lots of choices. "COPY_ALL" sources your current environment for the # job's environment. #@ environment = COPY_ALL # What queue to run the job in? #@ class = NORMAL # Where to start running the job #@ initialdir = /N/gpfsbr/namd_example # ## TeraGrid Account number goes in the line below. "NONE" is the account # number for IU users. #@ account_no = NONE # The name of the executable to be run. #@ executable = My_execution_script.ksh # Whatever else the executable wants put on its command line. #@ arguments = fee fi foe fum # Job control keywords. #@ node = 4 #@ tasks_per_node = 2 #@ job_type = MPICH #@ node_usage = not_shared # What to do if the job halts. Checkpoint is not totally supported yet. #@ checkpoint = no #@ restart = no # The last keyword in the list. Sort of a "stop" key. Anything below this # keyword will be executed. #@ queue

Following is an example of a script that runs an NAMD job:

#@ output = test_namd.$(Cluster).out #@ error = test_namd.$(Cluster).err #@ notification = complete #@ wall_clock_limit = 1:00:00 #@ environment = COPY_ALL # ## FAST is small debug queue, NORMAL is 2 weeks, BIG is large jobs for 48 hour max. ## Type llclass on command line for up to date information #@ class = NORMAL # ## Please change this line to your work directory. #@ initialdir = /N/gpfsbr/namd_example # #@ executable = mpich_namd.bash # # TeraGrid Account number goes in the line below. I used none for test. #@ account_no = NONE # #@ node_usage = not_shared #@ node = 4 #@ tasks_per_node = 2 #@ job_type = MPICH #@ checkpoint = no #@ queue

The above script used the following mpich_namd.bash script:

#------------------------------------------------------------------ ## This should point to your work directory if you use it. cd /N/gpfsbr/namd_example # export NAMD2=`which namd2` # ## $LOADL_TOTAL_TASKS and $LOADL_HOSTFILE are defined by LoadLeveler ## for all jobs of type MPICH mpirun -np $LOADL_TOTAL_TASKS -machinefile $LOADL_HOSTFILE $NAMD2 apoa1.namd

Submitting and monitoring the job

To submit your batch job to LoadLeveler, enter:

llsubmit scriptname

For information about tracking your job, see Monitoring LoadLeveler jobs on Big Red at IU.

Output and error files

If your job runs to completion, the messages it tried to print on the console will be directed to the output file you specified in your LoadLeveler script. For example, if you used ${Executable}.${Cluster}.out/err in your script, the output file would be named test_namd.job-id.out and the error file would be named test_namd.job-id.err.

Email notification from your jobs

If you plan to run a large number of jobs through Big Red in a relatively short period of time and will be forwarding notifications to an external mail server, make sure the external mail server can handle the load. If you are not absolutely certain that the external email server can handle a large volume of mail, include this line in your jobs:

# @notification = never

Running interactive jobs

Four compute Blades, b509, b510, b511, and b512, are reserved for interactive use on Big Red. Access is available via the Big Red login nodes, and all users may log into these nodes at any time. They are intended specifically for long-running (more than 20 minutes of CPU time) interactive jobs, particularly interactive debugging of parallel jobs run over the Myrinet interconnect.

To use the interactive nodes on Big Red, log into the cluster as you normally would, and then connect (via SSH) to one of these nodes:

b509 b510 b511 b512

These nodes are not running LoadLeveler, and are open for any and all users on the cluster. To run MPI jobs on them, you'll need to create a machine file containing the names of the nodes on which you want your job to run. For example, if you want to run an 8-processor job across two of the nodes, you could use a file containing these lines:

b511 b511 b511 b511 b512 b512 b512 b512

Then, log into any of the nodes (b511 or b512 would probably make the most sense in this example) and run your job:

$ mpirun -np 8 -machinefile your file your MPI-linked binary

User activity is not restricted on these nodes, so you may run into trouble if you request four tasks on a single node (e.g., another user may have one or more processes running on one or more of the nodes you've requested, using MX ports on the Myrinet adapter). You can see the status of the MX ports on a node with the mt_endpoint_info command:

janedoe@BigRed:~> # mx_endpoint_info 1 Myrinet board installed. The MX driver is configured to support a maximum of: 8 endpoints per NIC, 1152 NICs on the network, 32 NICs per host =================================================================== Board #0: Endpoint PID Command Info ether none none raw 2804 fma 1 26506 su3imp.mpiP There are currently 1 regular endpoint open

In this example, one process (26506) is using one endpoint on the MX adapter (fma process 2804 is a Myrinet mapping daemon; it's often listed and can be safely ignored). Big Red supports up to eight endpoints on the MX adapters, but since there is usually one process associated with each endpoint, you may see some blocking on the node once you have more than four endpoints open (Big Red has four processors per node).

To determine the state of MX endpoints, run the following command from the Big Red login nodes or the interactive nodes:

$ psh interactive '/opt/mx/bin/mx_endpoint_info | grep "open"'

Running bioinformatics software

UITS provides scripts that you can use to run jobs that use BLAST and other bioinformatics software on Big Red. For more, see the Bioinformatics Support page.

File transfers using GridFTP

For information about using GridFTP (tgcp) to transfer files to and from Big Red, see On Big Red, how do I transfer files using GridFTP (globus-url-copy)?

This document was developed with support from the National Science Foundation (NSF) under Grant No. 0503697 to the University of Chicago and subcontracted to Indiana University. Additional support was provided by IU through its participation in the TeraGrid, which is supported by the NSF under Grants No. 0833618, SCI451237, SCI535258, and SCI504075. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.

This is document avjx in domains all and tgrid-all.
Last modified on July 31, 2009.