Getting started on Big Red

Username:
Password:

On this page:

Introduction

When commissioned in 2006, Big Red was one of the most powerful university-owned computers in the US, and one of the 50 fastest supercomputers in the world. Part of a comprehensive strategy to build an advanced cyberinfrastructure to support research at Indiana University, Big Red has a theoretical peak performance of more than 40 teraflops, and has achieved more than 28 teraflops on numerical computations.

To manage the multiple users, processors, and jobs running on the system, Big Red uses LoadLeveler to submit and monitor jobs. LoadLeveler relies on the Moab scheduler software for job scheduling, incorporating a fair share mechanism based on research system time used by each user trying to run a job.

The fair share mechanism does not allow users to run jobs on the login node or on the compute node outside of the LoadLeveler job submission system. If you submit a job outside of LoadLeveler and it uses more than 20 minutes of CPU time, it will be terminated. This applies to Globus jobs that use the jobmanager/fork.

Requesting an account or software

Requesting an account

IU researchers: To create an account on Big red, visit the Account Management Service at: https://itaccounts.iu.edu/
After you log in (using your Network ID username and passphrase), click create more accounts. Follow the instructions to create your Big Red account.
Non-IU researchers: Affiliate accounts on the research systems are available for non-IU researchers. For more, see About sponsored computing accounts for IU affiliates.

Note: Accounts remain valid only while the account holder is a registered IU student, or an IU faculty or staff member. On Big Red, Quarry, and the Research Database Complex, accounts are disabled during the semester following the account holder's departure from IU, and then are purged within six months. To exempt a research systems account from disabling, email a request to the Support Center's Academic Accounts team ( valid@indiana.edu ). If the request is approved, the account will remain activated for one calendar year beyond the user's departure from IU. Then, at the end of the year, the account will be purged. Extensions beyond one year for research accounts are granted only for accounts involved in funded research and having an IU faculty sponsor, or with approval of the Dean or Director of Research and Academic Computing.

Requesting software

Request software using the Research Systems Software Request form.

Logging in

Log into Big Red using your Network ID passphrase. The default shell is bash. To change your shell permanently, use the changeshell command:

jdoe@BigRed:~> changeshell This program will assist you in changing your login shell on all nodes of the Big Red cluster. . . . 1) bash 2) tcsh . . . 5) quit Select 1-5: 2 Changing login shell for jdoe Password: Shell changed. Your shell has been changed to the Cornell tcsh shell This will take effect on all nodes within 15 minutes

Notes

Windows users: If you use an SSH client in Windows, you cannot open tools that need a graphical user interface (GUI), such as the TotalView debugger and the Vampir-NG profiler. You'll need X Window emulation software, such as Cygwin. UITS recommends using XLiveCD, created by the Research Technologies division of UITS.
Intra-cluster logins: When you log into your Big Red account for the first time, passphrase-less SSH keys will be automatically created in your home directory. Those keys should enable you to log into compute nodes that you have gained access to through LoadLeveler without entering a password or a passphrase. In other words, parallel jobs should run seamlessly on multiple compute nodes without any manual intervention.
However, you may see the following error message when you try to access LoadLeveler-assigned compute nodes:
Permission denied (publickey,password,keyboard-interactive)
This indicates that the intra-cluster RSA key pair in your home directory is either not present or corrupted. If this happens, enter gensshkeys ; it will generate a passphrase-less key pair for you, allowing you seamless intra-cluster logins between any nodes in the cluster assigned for your use by LoadLeveler.
Forwarding email address for job-related messages: Big Red will send email about your jobs to the address specified in the ~/.forward file in your home directory. (Note the period [ . ] preceding the filename.) By default, this is the email address you provided when you requested your account.
If you'd like to change this email address, enter a command similar to the following, replacing username@host.com with your email address:
hpctrn01@BigRed:~> echo "username@host.com" > ~/.forward
Be sure to use a valid email address; if you do not, you will not be notified about the status of your jobs.

Using SoftEnv to set up your software environment

SoftEnv, an environment management system, lets you customize your environment (i.e., specify the software packages you plan to use) using symbolic keywords. For information about using SoftEnv on Big Red, see On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment?

File storage options

You can store files on your home directory or in scratch space. For more, see At IU, how much disk space is available to me on the research systems?

Compiling and running programs

For information about available compilers and how to use them, see Compiling programs on Big Red at IU.

Parallel applications and message-passing libraries

Big Red is a parallel machine; it is not structured for serial codes. Serial codes waste 75 percent of the processor cores and the interconnect switch (which account for one third the cost of the machine).

Using message passing

Select message-passing packages using SoftEnv. See On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment?
You must link the library that's consistent with the address precision (32-bit or 64-bit) you chose for the compile.
Once an MPI library is added, compiles are made through a wrapper to the IBM/Gnu compiler that built the library. For instance, mpif90 is actually just a wrapper to xlf90_r. The same switches used by xlf90 are available to mpif90_r.

MPICH

Argonne original
MPICH 1 is available; MPICH 2 could be
Uses mpirun
Limited runtime environment

OpenMPI

Replaced LAM
No more lamboot
Looks like MPICH to the user
Improving with each new release

Submitting batch jobs to LoadLeveler

Writing the script

Job scripts are divided into a keyword stanza and an execution section. If any lines exist in the script other than keywords (including just #!/bin/bash), the script is executed. Otherwise, it is sourced.
It's best to write your job in its own script file, and tell LoadLeveler to execute that.
Preface all keywords with "# @".
Keywords used in scripts include output, error, executable, notification, node_usage, node, job_type, checkpoint, and queue.

Following is an example of a script that runs an NAMD job:

# @ output = test_namd.$(Cluster).out # @ environment = COPY_ALL # @ class = NORMAL # @ initialdir = /N/dc/scratch/namd_example # @ account_no = NONE # @ node_usage = not_shared # @ node = 4 # @ job_type = MPICH # @ checkpoint = no # @ queue mpirun -np $LOADL_TOTAL_TASKS -machinefile $LOADL_HOSTFILE namd2 apoa1.namd

For more, see Batch jobs on Big Red.

Running interactive jobs

Four compute Blades, b509, b510, b511, and b512, are reserved for interactive use on Big Red. Access is available via the Big Red login nodes, and all users may log into these nodes at any time. They are intended specifically for long-running (more than 20 minutes of CPU time) interactive jobs, particularly interactive debugging of parallel jobs run over the Myrinet interconnect.

To use the interactive nodes on Big Red, log into the cluster as you normally would, and then connect (via SSH) to one of these nodes:

b509 b510 b511 b512

For example:

my-host$ ssh bigred.teragrid.iu.edu Password: username@BigRed:~> hostname | host2blade b519 username@BigRed:~> ssh b509 username@BigRed:~> hostname | host2blade b509

These nodes are not running LoadLeveler, and are open for any and all users on the cluster. To run MPI jobs on them, you'll need to create a machine file containing the names of the nodes on which you want your job to run. For example, if you want to run an 8-processor job across two of the nodes, you could use a file containing these lines:

b511 b511 b511 b511 b512 b512 b512 b512

Then, log into any of the nodes (b511 or b512 would probably make the most sense in this example) and run your job:

$ mpirun -np 8 -machinefile <your file> <your MPI-linked binary>

User activity is not restricted on these nodes, so you may run into trouble if you request four tasks on a single node (e.g., another user may have one or more processes running on one or more of the nodes you've requested, using MX ports on the Myrinet adapter). You can see the status of the MX ports on a node with the mt_endpoint_info command:

janedoe@BigRed:~> # mx_endpoint_info 1 Myrinet board installed. The MX driver is configured to support a maximum of: 8 endpoints per NIC, 1152 NICs on the network, 32 NICs per host =================================================================== Board #0: Endpoint PID Command Info ether none none raw 2804 fma 1 26506 su3imp.mpiP There are currently 1 regular endpoint open

In this example, one process (26506) is using one endpoint on the MX adapter (fma process 2804 is a Myrinet mapping daemon; it's often listed and can be safely ignored). Big Red supports up to eight endpoints on the MX adapters, but since there is usually one process associated with each endpoint, you may see some blocking on the node once you have more than four endpoints open (Big Red has four processors per node).

To determine the state of MX endpoints, run the following command from the Big Red login nodes or the interactive nodes:

$ psh interactive '/opt/mx/bin/mx_endpoint_info | grep "open"'

Running bioinformatics software

UITS provides scripts that you can use to run jobs that use BLAST and other bioinformatics software on Big Red. For more, see Bioinformatics support.

This is document avjx in domain all.
Last modified on November 23, 2011.