ARCHIVED: Big Red and the Indiana Economic Development Corporation

Username:
Password:

On this page:

Introduction

Big Red, one of the fastest supercomputers in the United States, is being used to support Indiana's economic growth. A major upgrade to Big Red, which doubled the capacity of the cluster, put 20 teraflops of computational capability in the hands of Indiana businesses seeking to spin out technology, increase research grant competitiveness, and boost the state's growing life sciences community.

Located at Indiana University in Bloomington, the Indiana Economic Development Corporation (IEDC) expansion of Big Red is co-owned and co-managed by IU and Purdue University, and is part of the commitment by IBM and its partners to bring new jobs and new investments to Indiana.

A team of officials from both universities and the IEDC is working to target businesses with significant operations in the state that could use the system to develop new products, improve efficiency, or enhance profitability. The team will review requests for use of the new system from Indiana businesses.

For more about the initiative and the process for submitting project proposals, email IndustrialResearchPartnership@iu.edu , or visit Indiana Initiative for Economic Development.

The Big Red nodes dedicated to the IEDC use LoadLeveler to submit and monitor jobs, and to manage multiple users, processors, and jobs running on the system. LoadLeveler relies on the Moab scheduler software for job scheduling, incorporating a fair share mechanism based on research system time used by each user trying to run a job.

Connecting and logging in

To log in, use SSH2 and your IEDC username and password:

ssh bigred.teragrid.iu.edu -l your_iedc_username

You can also use this hostname to transfer files and submit jobs.

Note: The default shell is bash.

Windows users

If you use an SSH client in Windows, you cannot open tools that need a graphical user interface (GUI), like the TotalView debugger and the Vampir-NG profiler. You'll need X Window emulation software such as Cygwin.

Intra-cluster logins

When you log into your IEDC account for the first time, passphrase-less SSH keys will be automatically created in your home directory. Those keys should enable you to log into compute nodes that you have gained access to through LoadLeveler without entering a password or a passphrase. In other words, parallel jobs should run seamlessly on multiple compute nodes without any manual intervention.

However, you may see the following error message when you try to access LoadLeveler-assigned compute nodes:

Permission denied (publickey,password,keyboard-interactive)

This indicates that the intra-cluster RSA key pair in your home directory is either not present or corrupted. If this happens, enter gensshkeys ; it will generate a passphrase-less key pair for you, allowing you seamless intra-cluster logins between any nodes in the cluster assigned for your use by LoadLeveler.

Forwarding email address for job-related messages

IEDC will send email about your jobs to the address specified in the ~/.forward file in your home directory. (Note the period [ . ] preceding the filename.) By default, this is the email address you provided when you requested your account.

If you'd like to change this email address, enter a command similar to the following, replacing username@host.com with your email address:

IEDCLogin@BigRed:~> echo "username@host.com" > ~/.forward

Be sure to use a valid email address; if you do not, you will not be notified about the status of your jobs.

Changing your password

To change the password of your IEDC account, visit:

https://passphrase.iu.edu/

Enter your IEDC username (e.g., iedc1 or iedc2), and fill in the appropriate fields to change the password. You should see the following message:

IU Network ID
Password changed.
Note: It may take up to twenty (20) minutes for all passphrase changes to take effect.

Disregard any error messages about ADS. You should be able to log in with the new password in 20-30 minutes.

Using SoftEnv to set up your software environment

SoftEnv, an environment management system, lets you customize your environment (i.e., specify the software packages you plan to use) using symbolic keywords. For information about using SoftEnv on IEDC, see On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment?

File storage options

You can store files on your home directory or in scratch space, or use a GPFS file system. For more, see the sections that pertain to Big Red in At IU, how much disk space is available to me on the research systems?

Compiling and running programs

For information about available compilers and how to use them, see Compiling programs on Big Red at IU.

Using message passing libraries

Select message passing packages using SoftEnv. See On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment?
You must link the library that's consistent with the address precision (32-bit or 64-bit) you chose for the compile.
Once an MPI library is added, compiles are made through a wrapper to the IBM/Gnu compiler that built the library. For instance, mpif90 is actually just a wrapper to xlf90_r. The same switches used by xlf90 are available to mpif90_r.

MPICH

Argonne original
MPICH 1 is available; MPICH 2 could be
Uses mpirun
Limited runtime environment

OpenMPI

Replaced LAM
No more lamboot
Looks like MPICH to the user
Improving with each new release

Submitting batch jobs to LoadLeveler

Writing the script

Job scripts are divided into a keyword stanza and an execution section. If any lines exist in the script other than keywords (including just #!/bin/bash), the script is executed. Otherwise, it is sourced.
It's best to write your job in its own script file and tell LoadLeveler to execute that.
Preface all keywords with #@ .
Keywords used in scripts include: output, error, executable, notification, node_usage, node, job_type, checkpoint, queue.

Following is an annotated example of a typical script that you may use as a template:

# Pretty typical file designator. Will take absolute paths or try to write # to your initial directory.; $(Cluster) is unique job id #@ output = test_namd.$(Cluster).out #@ error = test_name.$(Cluster).err # When do you want the job to send an email? Must put address in your # .forward file. #@ notification = complete # How long to run? The default is hh:mm:ss or you could specify in seconds. #@ wall_clock_limit = 1:00:00 # Lots of choices. "COPY_ALL" sources your current environment for the # job's environment. #@ environment = COPY_ALL # What queue to run the job in? #@ class = IEDC # Where to start running the job #@ initialdir = /N/gpfsbr/namd_example # # The name of the executable to be run. #@ executable = My_execution_script.ksh # Whatever else the executable wants put on its command line. #@ arguments = fee fi foe fum # Job control keywords. #@ node = 4 #@ tasks_per_node = 2 #@ job_type = MPICH #@ node_usage = not_shared # What to do if the job halts. Checkpoint is not totally supported yet. #@ checkpoint = no #@ restart = no # The last keyword in the list. Sort of a "stop" key. Anything below this # keyword will be executed. #@ queue

Following is an example of a script that runs an NAMD job:

#@ output = test_namd.$(Cluster).out #@ error = test_namd.$(Cluster).err #@ notification = complete #@ wall_clock_limit = 1:00:00 #@ environment = COPY_ALL # ## Type llclass on command line for up to date information #@ class = IEDC # ## Please change this line to your work directory. #@ initialdir = /N/gpfsbr/namd_example # #@ executable = mpich_namd.bash # #@ node_usage = not_shared #@ node = 4 #@ tasks_per_node = 2 #@ job_type = MPICH #@ checkpoint = no #@ queue

The above script used the following mpich_namd.bash script:

#------------------------------------------------------------------ ## This should point to your work directory if you use it. cd /N/gpfsbr/namd_example # export NAMD2=`which namd2` # ## $LOADL_TOTAL_TASKS and $LOADL_HOSTFILE are defined by LoadLeveler ## for all jobs of type MPICH mpirun -np $LOADL_TOTAL_TASKS -machinefile $LOADL_HOSTFILE $NAMD2 apoa1.namd

Submitting and monitoring the job

To submit your batch job to LoadLeveler, enter:

llsubmit scriptname

For information about tracking your job, see Monitoring LoadLeveler jobs on Big Red at IU.

Output and error files

If your job runs to completion, the messages it tried to print on the console will be directed to the output file you specified in your LoadLeveler script. For example, if you used ${Executable}.${Cluster}.out/err in your script, the output file would be named test_namd.job-id.out and the error file would be named test_namd.job-id.err.

Email notification from your jobs

If you plan to run a large number of jobs through IEDC in a relatively short period of time and will be forwarding notifications to an external mail server, make sure the external mail server can handle the load. If you are not absolutely certain that the external email server can handle a large volume of mail, include this line in your jobs:

# @notification = never

For more about running batch jobs on Big Red, see Batch jobs on Big Red.

Running interactive jobs

Four compute Blades, b509, b510, b511, and b512, are reserved for interactive use on Big Red. Access is available via the IEDC login nodes, and all users may log into these nodes at any time. They are intended specifically for long-running (more than 20 minutes of CPU time) interactive jobs, particularly interactive debugging of parallel jobs run over the Myrinet interconnect.

To use the interactive nodes on the Big Red cluster, log in as you normally would, and then connect (via SSH) to one of these nodes:

b509 b510 b511 b512

These nodes are not running LoadLeveler, and are open for any and all users on the cluster. To run MPI jobs on them, you'll need to create a machine file containing the names of the nodes on which you want your job to run. For example, if you want to run an 8-processor job across two of the nodes, you could use a file containing these lines:

b511 b511 b511 b511 b512 b512 b512 b512

Then, log into any of the nodes (b511 or b512 would probably make the most sense in this example) and run your job:

$ mpirun -np 8 -machinefile your_file your_MPI_linked_binary

User activity is not restricted on these nodes, so you may run into trouble if you request four tasks on a single node (e.g., another user may have one or more processes running on one or more of the nodes you've requested, using MX ports on the Myrinet adapter). You can see the status of the MX ports on a node with the mt_endpoint_info command:

janedoe@BigRed:~> # mx_endpoint_info 1 Myrinet board installed. The MX driver is configured to support a maximum of: 8 endpoints per NIC, 1152 NICs on the network, 32 NICs per host =================================================================== Board #0: Endpoint PID Command Info ether none none raw 2804 fma 1 26506 su3imp.mpiP There are currently 1 regular endpoint open

In this example, one process (26506) is using one endpoint on the MX adapter (fma process 2804 is a Myrinet mapping daemon; it's often listed and can be safely ignored). Big Red supports up to eight endpoints on the MX adapters, but since there is usually one process associated with each endpoint, you may see some blocking on the node once you have more than four endpoints open (Big Red has four processors per node).

To determine the state of MX endpoints, run the following command from the Big Red login nodes or the interactive nodes:

$ psh interactive '/opt/mx/bin/mx_endpoint_info | grep "open"'

This is document awsr in domain all.
Last modified on April 19, 2011.