Getting started on Quarry
On this page:
- Introduction
- Requesting an account or software
- Connecting and logging in
- Managing local and remote files
- Using the vi editor
- Using the GNU Emacs editor
- Command files (shell scripts)
- Using SoftEnv to set up your software environment
- File storage options
- Parallel computing and MPI
- Compiling programs and submitting batch jobs to TORQUE (OpenPBS)
- Compiling and submitting parallel jobs
Introduction
Indiana University's newest supercomputer, Quarry is an IBM HS21 Bladeserver cluster running Red Hat Linux, with TORQUE (PBS) and Moab for job management and SoftEnv to simplify application and environment configuration.
User home directories are NFS-exported from an IBM N5500
Network-Attached Storage (NAS) device. Shared scratch disk is provided
via IBM's General Parallel File System (GPFS), which is mounted
cluster-wide at /N/gpfs/username. The Data Capacitor will be available
late fall 2007 on all nodes at
/N/dc/scratch. Both GPFS and the Data Capacitor are
currently available from Big Red.
For details on Quarry's hardware configuration, see At IU, what is Quarry?
Announcements, down time information, and documentation are available on the Quarry Cluster home page.
Quarry is supported by the UITS High Performance Systems group. If you have system specific questions about Quarry, email High Performance Systems. If you have questions about compilers, programming, scientific/numerical libraries, or debuggers, email the High Performance Applications team.
Requesting an account or software
Quarry serves multiple communities as a research cluster and as a general purpose Linux environment. Accounts are open to all IU faculty, staff, and students, as well as affiliated researchers.
-
Account request: To obtain an account on Quarry,
visit the Account Management Service at:
https://itaccounts.iu.edu/
After you log in (using your Network ID username and passphrase), click
create more accounts. Follow the instructions to create your Quarry accounts. - Software request: If you are at IU and have an account on one of IU's research systems, you can request software using the Research Systems Software Request form.
Connecting and logging in
If you are at IU, you can log into Quarry using your Network ID and passphrase and the Secure Shell client, SSH, using a command similar to one of the following:
ssh your_username@quarry.uits.indiana.eduPlease read the message of the day (MOTD), as it contains news and information regarding the status of the cluster.
The default shell is bash. When you log in for the first
time, you will be prompted to select your preferred login shell by the
changeshell program:
If you use the Bash, Bourne, or Korn shells, the system will
automatically read and execute commands from the
/etc/profile file and your own ~/.profile
(and ~/.bash_profile, in the case of Bash). With the
csh and tcsh shells, the .login
and .cshrc (or .tcshrc) files are read. For
details, see In Unix, what is the shell? or the Unix man pages.
Notes
- Head nodes vs. compute nodes: When you login into Quarry as shown above, you will be logging into one of the four head nodes. The exact head node you are assigned on any given login is determined on a round-robin basis. Quarry's compute nodes are named according to the scheme qXXX, where XXX currently ranges from 005 to 112. These compute nodes are accessible only from the head node, or from each other, not from the outside world; i.e., you cannot use SSH to connect to compute nodes directly. Access to any given compute node is further limited to only those users who have a job currently running on that node.
-
Windows users: If you use an SSH client in
Windows, you cannot open tools that need a graphical user interface
(GUI), like the TotalView debugger. You'll need X Window emulation
software such as Cygwin. UITS recommends using XLiveCD, which the Research
Technologies division of UITS created. See How can I run X Window System applications on Unix systems from a Microsoft Windows workstation?
-
X applications: To use the TotalView
Debugger/Intel Trace Analyzer, or any other graphical application,
from Quarry compute nodes, you must disable X
forwarding by specifying the
-xflag: ssh -x your_username@quarry.uits.indiana.eduTo use common graphical applications, such as Emacs, from the head node, do not use the
-xoption; the default X forwarding must remain enabled. -
Intra-cluster logins: When you log into your
Quarry account for the first time, passphrase-less SSH keys will be
automatically created in your home directory. Those keys should enable
you to log into compute nodes that you have gained access to through
TORQUE without entering a passphrase. I.e., parallel
jobs should run seamlessly on multiple compute nodes without any
manual intervention.
However, you may see the following error message when you try to access assigned compute nodes:
Permission denied (publickey,password,keyboard-interactive)This indicates that the intra-cluster RSA key pair in your home directory is either not present or corrupted. If this happens, enter
gensshkeys(in/opt/xcat/sbin) from any login node. That will generate a passphrase-less key pair and allow you seamless intra-cluster logins between any nodes in the cluster assigned for your use by TORQUE. -
Forwarding email address for job-related
messages: Quarry will send email about your jobs to the
address specified in the
~/.forwardfile in your home directory. (Note the period [.] preceding the filename.) By default, this is the email address you provided when you requested your account.If you'd like to change this email address, enter a command similar to the following, replacing
jdoe@Quarry:~> echo "username@host.com" > ~/.forwardusername@host.comwith your email address:Be sure to use a valid email address; if you do not, you will not be notified about the status of your jobs.
Managing local and remote files
Like other Unix systems, Linux uses a hierarchical, tree-structured directory system to organize files. Data on local disk resides in journalled file systems that contain a root directory from which associated files and subdirectories branch. Directories are catalogs of files that associate names with files and are used to segregate files into related groups. To display information about currently mounted file systems, at the shell prompt, enter:
df
Each file is described by an inode. An inode contains critical
information about the file, such as file type (directory, ordinary,
character special, block special, or pipe), ownership, access
permissions, group ID, file size, file creation, modification, and
access times, and a pointer to its data blocks. To list information
about the files within your current working directory, at the shell
prompt, enter ls -al . Following is a sample of
ls output:
The first character in the first field indicates the file type (e.g.,
- for ordinary file, d for directory). The
next nine characters in the first field indicate the file access
permissions: r for read, w for write, and
x for execute; a hyphen (-) indicates that
the corresponding read, write, or execute permission is denied.
Characters 2, 3, and 4 indicate the file owner's permissions,
characters 5, 6, and 7 indicate the group's permissions, and the last
three characters indicate all others' permissions. Execute permission
for a directory allows users to search through and list its contents.
Field 2 indicates the number of links to the file. The next two fields
show the owner and the group associated with the file. Field 5 gives
the size of the file in bytes. Field 6 is the time the file was
created or last modified. The last field is the name of the file. The
files named . and .. indicate the current
and the parent directory, respectively. For more information about
ls and available options, enter man
ls .
To change directories, use the cd command. You can use an
absolute or relative path name with the cd command. For
example, assume you are currently in the /usr/local
directory. To get to the /usr/local/bin directory, you
could enter either cd /usr/local/bin (using the absolute
path name) or cd bin (using the path name relative to
/usr/local). To get back to /usr/local, you
could enter cd /usr/local or just cd .. (to
go up to the parent directory). Entering cd without an
argument will put you into your home directory. To determine what your
current working directory is, enter pwd .
Following are some common commands for working with files:
- To view a file on your screen, enter
cat. To view one screen at a time, entercat | more.
- To copy a file to the same or a different directory, enter
cp. This command overwrites any existing file of the same name. For more information, enterman cp.
- To rename a file or to move it to a new directory, enter
mv. For more information, enterman mv.
- To delete a file, enter
rm. For more information about deleting single or multiple files, enterman rm.
- To create a directory, use the
mkdircommand. To delete an empty directory, use thermdircommand. For more information, enterman mkdirorman rmdir.
You can also use the commands for managing local files to manage
remote NFS files, provided that the access permissions allow it. For
example, you cannot use rm on a file in a remote file
system that's mounted as read-only.
For access to remote files other than those already accessible to
Quarry, you may access a remote host using the Secure Shell
scp or sftp commands to copy files to or
from the remote system.
To see how many 1KB blocks of disk space you are using, enter du
-k . To see the number of 1KB blocks in your disk quota,
enter nfsquota .
You may exceed your soft quota up to the limit of your hard quota. If you exceed your soft quota, you will receive a message to that effect. You then have seven days to reduce your use below your soft quota. If you do not reduce your disk usage below your soft quota, at the end of the grace period you will not be able to write to your disk until you reduce your disk usage below your soft quota. The grace period is seven days.
Using the vi editor
Before entering vi, make sure your environment variable TERM is set
correctly. When vi starts, it looks at TERM to see what kind of
terminal you are using. It will then load the proper terminal control
information from /etc/termcap (the terminal capabilities
file). Enter the appropriate commands below depending on your shell:
- If you use the Bourne (
sh) or Korn (ksh) shell: $ TERM=vt100 $ export TERM $ vi filename - If you use the C shell (
csh): % setenv TERM vt100 % vi filename
For information about common vi commands, see How do I use the vi text editor?, A quick reference list of vi editor commands, In vi, how can I access deleted text?
You can find more information in Learning the vi Editor by Linda Lamb (O'Reilly and Associates, Inc., 1992), $21.95.
Using the GNU Emacs editor
GNU Emacs is a new version of Emacs written by the author of the original (PDP-10) Emacs, Richard Stallman. GNU Emacs retains all the functionality of other Emacs editors, but it is also customizable and extensible by modifying its Lisp editing commands. GNU Emacs is designed to work well with X Window systems, but it also functions with ASCII terminals.
By default emacs is not in your path. You will need to
add +emacs to the .soft file in your home
directory.
For the change to take effect, enter resoft , or log out
and then back in. For more information on softenv see
On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment?
The account creation process on Quarry copies a
.emacs file into the newly created home directory:
Because GNU Emacs uses the Ctrl-q and Ctrl-s
key sequences as commands, disable the use of Ctrl-q and
Ctrl-s as terminal output start/stop signals before
entering GNU Emacs. To do this, enter stty
-ixon . After completing your GNU Emacs session, you may
re-enable terminal start/stop signals by entering stty
ixon .
GNU Emacs has an extensive interactive help facility, but to use it
you must know how to manipulate Emacs windows and buffers. (For help
with the Emacs notation that follows, see In Emacs, how are keystrokes denoted?) To enter
the help, press C-h . To access an interactive
tutorial that teaches the fundamentals of Emacs, press
C-h t . To find a command given its
functionality, press C-h a . The Help Character
command (C-h c) describes a given character's effect,
and the Help Function command (C-h f) describes a
given Lisp function specified by name.
For information about common Emacs commands, see GNU Emacs Quick Reference Guide.
Command files (shell scripts)
Shell scripts are command files interpreted by a shell. These scripts
may contain any statements valid in the shell in which they are
interpreted. You can use them to perform repetitive tasks, such as
setting up your environment, compiling and executing programs,
submitting batch jobs, and performing system management tasks.
Locally written shell script utilities are contained in
/usr/local/bin.
Shell programming constructs include variable assignment, argument handling, evaluation of integer expressions, conditional execution, flow control, file status checking, string and integer comparisons, reading of input, interrupt signal traps, and menu screen generation.
The following example of a C shell script illustrates some basic shell programming constructs.
C shell script
# Scriptname: showuser # Purpose: display information about a user set USAGE = "Usage: showuser userid" # Test for argument if ($#argv > 1) then echo $USAGE exit 1 endif if ($#argv == 1) then set USERNAME = $1 else if ($#argv == 0) then echo "Enter the user's login name: " set USERNAME = $< endif set IFS = ":" # internal field separator # Use sed to edit /etc/passwd, putting blank in empty field sed "s/$IFS/ /g" < /etc/passwd > $HOME/passwd echo "_________________________________________________________" echo "Userid UID GID Full Name Home Directory Login Shell" echo " " # Use awk to find the correct line and print it out awk "/$USERNAME/" $HOME/passwd echo "_________________________________________________________" rm $HOME/passwd exitThe following reference guides may be of interest:
Using SoftEnv to set up your software environment
SoftEnv, an environment management system, lets you customize your environment (i.e., specify the software packages you plan to use) using symbolic keywords. For information about using SoftEnv, see On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment?
File storage options
You can store files on your home directory or in scratch space, or use a GPFS file system.
-
Home directory: Your Quarry home directory disk
space is allocated on the IBM N5500 NAS storage device. You have a
10GB disk quota which is shared with Big Red, Libra, and
the Research Database Complex (RDC) if you have accounts on
those systems.
-
Local scratch: Scratch disk space is
available locally on each node in
/scratch(19GB). Files in/scratchare automatically deleted once they are 14 days old.
-
Shared scratch: Shared scratch space is
accessible in one 266TB GPFS file system,
/N/gpfs. The path to your scratch space is/N/gpfs/username. Files older than 60 days are purged automatically to free disk space.
For more, see At IU, how much disk space is available to me on the research systems?
Parallel computing and MPI
The Quarry cluster is not equipped with a proprietary low-latency
interconnect. Parallel jobs can be run over gigabit Ethernet; the
cluster is connected to a Force10 E1200 switch. OpenMPI and MPICH1
compiled with the 64-bit Intel v10 compilers are available. The
relevant SoftEnv macros are @openmpi and
@mpich1.
Briefly, use mpicc or mpif77 to compile and
link MPI-enabled software. Then use mpirun to execute
the job on multiple nodes of the cluster. In practice, you must
submit your job through the TORQUE resource manager to
ensure fair and efficient use of the cluster.
Compiling programs and submitting batch jobs to TORQUE (OpenPBS)
Two simple programs are listed below for your convenience:
C program
/* * Copyright 2005, The Trustees of Indiana University. * Original author: Arvind Gopu (UITS-RAC-HPC group)... * . . . [ snip ] . . . */ #include<stdio.h> int main () { double PI2=3.141592654/2.0, theta, sintheta; int i, N=4; for (i=0; i<=N; i++) { theta = i * (PI2/N); sintheta = sin (theta); printf (" Sin (%8.6lf) = %8.6lf \n", theta, sintheta); } return 0; }Fortran program
C C Copyright 2005, The Trustees of Indiana University. C Original author: Don Berry (UITS-RAC-HPC group); C . . . [ snip ] . . . C program sine real, parameter :: PI2=3.141592654/2.0 integer, parameter :: N=4 real x,s integer i do i=0,N x=i*(PI2/N) s=sin(x) write(6,"(f11.6 f11.6)") x,s end do end
These commands use the GNU compilers (gcc and
g77 for C and Fortran programs respectively) to compile
the two example programs:
Submit job(s) to PBS
It's best to write your job in its own script file and tell PBS to
execute that. Following is a script that runs the example
sine_c program:
Use the script to submit your job to PBS:
[jdoe@Quarry]$ qsub submit_sine_c.sh 95.m2.quarry.teragrid.iu.edu
Check the job status with the qstat command:
For more about submitting and tracking jobs, see What is TORQUE?
Output and error files
Assuming your job runs to completion, you can find messages it tried
to print on the console in an output file. You can specify the the
directories where the output file and the error file go using the
-o and -e flags
respectively. The default file name of these files is of the format
job_name.osequence_number and
job_name.esequence_number, where job_name is the name of
the job (check out the -N option) and
sequence_number is the job number assigned when the job
is submitted.
For example, if PBS assigned job id
95.m2.quarry.teragrid.iu.edu to your job, with jobname
job_sine_c, the output file would be named
job_sine_c.o95 and the error file would be named
job_sine_c.e95. (Also see the use of the
-j PBS directive as explained in the section
above.)
Check the output files:
[jdoe@Quarry]$ cat job_sine_c.o348921 Sin (0.000000) = 0.000000 Sin (0.392699) = 0.382683 Sin (0.785398) = 0.707107 Sin (1.178097) = 0.923880 Sin (1.570796) = 1.000000 [jdoe@Quarry]$ cat job_sine_f.o348922 0.000000 0.000000 0.392699 0.382683 0.785398 0.707107 1.178097 0.923880 1.570796 1.000000Compiling and submitting parallel jobs
Here is a simple C program you can submit to use multiple processors:
#include <stdio.h> #include <mpi.h> /* #include "VT.h" */ /* only needed for Vampir Trace */ int main( int argc, char *argv[] ) { /* Remember: We are programming according to the SPMD model! */ /* This means that each and every processor executes this program. */ /* It is convenient to reference the typical processor in the */ /* first person, using "I" and "me" and "my." In other words, */ /* read through this program as if you were one of the processors. */ /* My ID number or my "rank": */ int myrank, length; char myname[BUFSIZ]; /* I initialize for myself the MPI API: */ MPI_Init(&argc, &argv); /* Who am I? I request my ID number: */ MPI_Comm_rank(MPI_COMM_WORLD, &myrank); MPI_Get_processor_name (myname, &length); //VT_symdef( 200, "for-loop", "Calculation" ); /* I print the standard greeting along with my ID: */ printf( "Hello, parallel worlds! This is processor %s and my rank is %d!\n", myname, myrank ); /* Finally I close the MIP API: */ MPI_Finalize(); return 0; }Here is the script file to submit the job:
#PBS -l nodes=2:ppn=2,walltime=5:00 #PBS -m ae #PBS -N job_helloWorlds mpirun C ${HOME}/simple_quarry_jobs/helloWorldsCompile and submit:
[jdoe@Quarry]$ mpicc -o helloWorlds helloWorlds.c [jdoe@Quarry]$ qsub submit_parallel.sh 97.m2.quarry.teragrid.iu.eduLook at the output file:
[jdoe@Quarry]$ cat job_helloWorlds.o348923 LAM 7.1.1/MPI 2 C++/ROMIO - Indiana University Hello, parallel worlds! This is processor bc27 and my rank is 1! Hello, parallel worlds! This is processor bc25 and my rank is 2! Hello, parallel worlds! This is processor bc27 and my rank is 0! Hello, parallel worlds! This is processor bc25 and my rank is 3! LAM 7.1.1/MPI 2 C++/ROMIO - Indiana UniversityLast modified on June 10, 2008.
