Getting started on Quarry

Username:
Password:

On this page:

Introduction

Indiana University's newest supercomputer, Quarry, is an IBM HS21 Bladeserver cluster running Red Hat Linux, with TORQUE (also called PBS) and Moab for job management and SoftEnv to simplify application and environment configuration.

User home directories are NFS-exported from an IBM N5500 Network-Attached Storage (NAS) device. Shared scratch disk is provided via IBM's General Parallel File System (GPFS), which is mounted cluster-wide at /N/gpfs/username. The Data Capacitor is available on all nodes at /N/dc/scratch. Both GPFS and the Data Capacitor are currently available from Big Red.

For details on Quarry's hardware configuration, see At IU, what is Quarry?

Announcements, downtime information, and documentation are available on the Quarry Cluster home page.

Quarry is supported by the UITS High Performance Systems group. If you have system-specific questions about Quarry, email High Performance Systems. If you have questions about compilers, programming, scientific/numerical libraries, or debuggers, email the High Performance Applications team.

Requesting an account or software

Quarry serves multiple communities as a research cluster and as a general-purpose Linux environment. Accounts are open to all IU faculty, staff, and students, as well as affiliated researchers.

Account request: To obtain an account on Quarry, visit the Account Management Service at: https://itaccounts.iu.edu/
After you log in (using your Network ID), click create more accounts. Follow the instructions to create your Quarry accounts.
Software request: If you are at IU and have an account on one of IU's research systems, you can request software using the Research Systems Software Request form.

Connecting and logging in

If you are at IU, log into Quarry with your Network ID over SSH, using a command similar to:

ssh your_username@quarry.uits.indiana.edu

Please read the message of the day (MOTD), as it contains news and information regarding the status of the cluster.

The default shell is bash. When you log in for the first time, you will be prompted to select your preferred login shell by the changeshell program:

jdoe@Quarry:~> changeshell This program will assist you in changing your login shell on all nodes of the Quarry cluster. . . . 1) bash 2) tcsh . . . 5) quit Select 1-5: 2 Changing login shell for jdoe Password: Shell changed. Your shell has been changed to the Cornell tcsh shell This will take effect on all nodes within 15 minutes

If you use the Bash, Bourne, or Korn shells, the system will automatically read and execute commands from the /etc/profile file and your own ~/.profile (and ~/.bash_profile, in the case of Bash). With the csh and tcsh shells, the .login and .cshrc (or .tcshrc) files are read. For details, see In Unix, what is the shell? or the Unix man pages.

Head, compute, and interactive nodes

When you log into Quarry as shown above, you will be logging into one of the four head nodes. The exact head node you are assigned on any given login is determined on a round-robin basis. The login nodes, q001-q004, have a 20-minute time limit on interactive use. Quarry's compute nodes are named according to the scheme qXXX, where XXX currently ranges from 005 to 112. These compute nodes are accessible only from the head nodes, or from each other, not from the outside world; i.e., you cannot use SSH to connect to compute nodes directly. If you need to run interactive jobs longer than 20 minutes, log into one of the interactive nodes (q005-q008) after logging into Quarry, and run long jobs there.

Notes

Windows users: If you use an SSH client in Windows, you cannot open tools that need a graphical user interface (GUI), like the TotalView debugger. You'll need X Window emulation software such as Cygwin. UITS recommends using XLiveCD, which the Research Technologies division of UITS created. See How can I run X Window System applications on Unix systems from a Microsoft Windows workstation?
X applications: To use the TotalView Debugger/Intel Trace Analyzer, or any other graphical application, from Quarry compute nodes, you must disable X forwarding by specifying the -x flag: ssh -x your_username@quarry.uits.indiana.edu
To use common graphical applications, such as Emacs, from the head node, do not use the -x option; the default X forwarding must remain enabled.
Intra-cluster logins: When you log into your Quarry account for the first time, passphrase-less SSH keys will be automatically created in your home directory. Those keys should enable you to log into compute nodes that you have gained access to through TORQUE without entering a passphrase. Thus, parallel jobs should run seamlessly on multiple compute nodes without any manual intervention.
However, you may see the following error message when you try to access assigned compute nodes:
Permission denied (publickey,password,keyboard-interactive)
This indicates that the intra-cluster RSA key pair in your home directory is either not present or corrupted. If this happens, enter gensshkeys (in /opt/xcat/sbin) from any login node. That will generate a passphrase-less key pair and allow you seamless intra-cluster logins between any nodes in the cluster assigned for your use by TORQUE.
Forwarding email address for job-related messages: Quarry will send email about your jobs to the address specified in the ~/.forward file in your home directory. (Note the period [ . ] preceding the filename.) By default, this is the email address you provided when you requested your account.
If you'd like to change this email address, enter a command similar to the following, replacing username@host.com with your email address:
jdoe@Quarry:~> echo "username@host.com" > ~/.forward
Be sure to use a valid email address; if you do not, you will not be notified about the status of your jobs.

Managing local and remote files

Like other Unix systems, Linux uses a hierarchical, tree-structured directory system to organize files. Data on local disk resides in journalled file systems that contain a root directory from which associated files and subdirectories branch. Directories are catalogs of files that associate names with files and are used to segregate files into related groups. To display information about currently mounted file systems, at the shell prompt, enter:

Each file is described by an inode. An inode contains critical information about the file, such as file type (directory, ordinary, character special, block special, or pipe), ownership, access permissions, group ID, file size, file creation, modification, and access times, and a pointer to its data blocks. To list information about the files within your current working directory, at the shell prompt, enter ls -al . Following is a sample of ls output:

$ ls -al drwxr-xr-x 5 jdoe chem 1024 Feb 21 16:06 . drwxr-xr-x 12 bin bin 512 Feb 05 10:56 .. -rw-r--r-- 1 jdoe chem 13250 Feb 14 11:29 .Xdefaults -rw-r--r-- 1 jdoe chem 120 Dec 19 14:41 .aliases.ksh -rwxr----- 1 jdoe chem 24 Jan 13 11:50 .exrc -rw-r--r-- 1 jdoe chem 471 Jan 29 16:37 .profile -rw-r--r-- 1 jdoe chem 255 Jan 29 16:18 .rhosts drwxr-xr-x 3 jdoe chem 512 Dec 19 14:48 bin -rw------- 1 jdoe chem 2054 Feb 21 16:07 .sh_history -rw-r--r-- 1 jdoe chem 37479 Feb 21 14:51 whitepaper.pdf -rwxr-x--- 1 jdoe chem 45 Jan 16 15:22 submit.sh -rwxr-x--- 1 jdoe chem 79 Feb 19 13:02 myjob.c -rwxr-x--- 1 jdoe chem 53 Feb 19 16:45 myjob.c.old -rwxr-xr-x 1 jdoe chem 93 Jan 29 16:10 myjob.o drwxr-xr-x 2 jdoe chem 512 Dec 20 08:41 project1 drwxr-xr-x 2 jdoe chem 512 Dec 20 08:41 project2

The first character in the first field indicates the file type (e.g., - for ordinary file, d for directory). The next nine characters in the first field indicate the file access permissions: r for read, w for write, and x for execute; a hyphen (-) indicates that the corresponding read, write, or execute permission is denied. Characters 2, 3, and 4 indicate the file owner's permissions, characters 5, 6, and 7 indicate the group's permissions, and the last three characters indicate all others' permissions. Execute permission for a directory allows users to search through and list its contents.

Field 2 indicates the number of links to the file. The next two fields show the owner and the group associated with the file. Field 5 gives the size of the file in bytes. Field 6 is the time the file was created or last modified. The last field is the name of the file. The files named . and .. indicate the current and the parent directory, respectively. For more information about ls and available options, enter man ls .

To change directories, use the cd command. You can use an absolute or relative path name with the cd command. For example, assume you are currently in the /usr/local directory. To get to the /usr/local/bin directory, you could enter either cd /usr/local/bin (using the absolute path name) or cd bin (using the path name relative to /usr/local). To get back to /usr/local, you could enter cd /usr/local or just cd .. (to go up to the parent directory). Entering cd without an argument will put you into your home directory. To determine what your current working directory is, enter pwd .

Following are some common commands for working with files:

To view a file on your screen, enter cat . To view one screen at a time, enter cat | more .
To copy a file to the same or a different directory, enter cp . This command overwrites any existing file of the same name. For more information, enter man cp .
To rename a file or to move it to a new directory, enter mv . For more information, enter man mv .
To delete a file, enter rm . For more information about deleting single or multiple files, enter man rm .
To create a directory, use the mkdir command. To delete an empty directory, use the rmdir command. For more information, enter man mkdir or man rmdir .

You can also use the commands for managing local files to manage remote NFS files, provided that the access permissions allow it. For example, you cannot use rm on a file in a remote file system that's mounted as read-only.

For access to remote files other than those already accessible to Quarry, you may access a remote host using the Secure Shell scp or sftp commands to copy files to or from the remote system.

To see how many 1KB blocks of disk space you are using, enter du -k . To see the number of 1KB blocks in your disk quota, enter nfsquota .

You may exceed your soft quota up to the limit of your hard quota. If you exceed your soft quota, you will receive a message to that effect. You then have seven days to reduce your use below your soft quota. If you do not reduce your disk usage below your soft quota, you will receive a grace period of seven days, after which you will not be able to write to your disk until you reduce your disk usage below your soft quota.

Using the vi editor

Before entering vi, make sure your environment variable TERM is set correctly. When vi starts, it looks at TERM to see what kind of terminal you are using. It will then load the proper terminal control information from /etc/termcap (the terminal capabilities file). Enter the appropriate commands below depending on your shell:

If you use the Bourne (sh) or Korn (ksh) shell: $ TERM=vt100 $ export TERM $ vi filename
If you use the C shell (csh): % setenv TERM vt100 % vi filename

For information about common vi commands, see How do I use the vi text editor?, A quick reference list of vi editor commands, In vi, how can I access deleted text?

You can find more information in Learning the vi Editor by Linda Lamb (O'Reilly and Associates, Inc., 1992).

Using the GNU Emacs editor

GNU Emacs is a version of Emacs written by the author of the original (PDP-10) Emacs, Richard Stallman. GNU Emacs retains all the functionality of other Emacs editors, but it is also customizable and extensible by modifying its Lisp editing commands. GNU Emacs is designed to work well with X Window systems, but it also functions with ASCII terminals.

By default emacs is not in your path. You will need to add +emacs to the .soft file in your home directory.

soft add +keyword

For the change to take effect, enter resoft , or log out and then back in. For more, see On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment?

The account creation process on Quarry copies a .emacs file into the newly created home directory:

[jdoe@Quarry] cat .emacs ;; .emacs ;;; uncomment this line to disable loading of "default.el" at startup ;; (setq inhibit-default-init t) ;; turn on font-lock mode (when (fboundp 'global-font-lock-mode) (global-font-lock-mode t)) ;; enable visual feedback on selections ;(setq transient-mark-mode t) ;; default to better frame titles (setq frame-title-format (concat "%b - emacs@" system-name))

Because GNU Emacs uses the Ctrl-q and Ctrl-s key sequences as commands, disable the use of Ctrl-q and Ctrl-s as terminal output start/stop signals before entering GNU Emacs. To do this, enter stty -ixon . After completing your GNU Emacs session, you may re-enable terminal start/stop signals by entering stty ixon .

GNU Emacs has an extensive interactive help facility, but to use it you must know how to manipulate Emacs windows and buffers. (For help with the Emacs notation that follows, see In Emacs, how are keystrokes denoted?) To enter the help, press C-h . To access an interactive tutorial that teaches the fundamentals of Emacs, press C-h t . To find a command given its functionality, press C-h a . The Help Character command (C-h c) describes a given character's effect, and the Help Function command (C-h f) describes a given Lisp function specified by name.

For information about common Emacs commands, see GNU Emacs Quick Reference Guide.

Command files (shell scripts)

Shell scripts are command files interpreted by a shell. These scripts may contain any statements valid in the shell in which they are interpreted. You can use them to perform repetitive tasks, such as setting up your environment, compiling and executing programs, submitting batch jobs, and performing system management tasks. Locally written shell script utilities are contained in /usr/local/bin.

Shell programming constructs include variable assignment, argument handling, evaluation of integer expressions, conditional execution, flow control, file status checking, string and integer comparisons, reading of input, interrupt signal traps, and menu screen generation.

The following example of a C shell script illustrates some basic shell programming constructs.

C shell script

# Scriptname: showuser # Purpose: display information about a user set USAGE = "Usage: showuser userid" # Test for argument if ($#argv > 1) then echo $USAGE exit 1 endif if ($#argv == 1) then set USERNAME = $1 else if ($#argv == 0) then echo "Enter the user's login name: " set USERNAME = $< endif set IFS = ":" # internal field separator # Use sed to edit /etc/passwd, putting blank in empty field sed "s/$IFS/ /g" < /etc/passwd > $HOME/passwd echo "_________________________________________________________" echo "Userid UID GID Full Name Home Directory Login Shell" echo " " # Use awk to find the correct line and print it out awk "/$USERNAME/" $HOME/passwd echo "_________________________________________________________" rm $HOME/passwd exit

The following reference guides may be of interest:

Using SoftEnv to set up your software environment

SoftEnv, an environment management system, lets you customize your environment (i.e., specify the software packages you plan to use) using symbolic keywords. For more, see On Big Red and Quarry at IU, how can I use SoftEnv to customize my software environment?

File storage options

You can store files on your home directory or in scratch space, or use a GPFS file system.

Home directory: Your Quarry home directory disk space is allocated on the IBM N5500 NAS storage device. You have a 10GB disk quota, which is shared with Big Red and the Research Database Complex (RDC) if you have accounts on those systems.
Local scratch: Scratch disk space is available locally on each node in /scratch (19GB). Files in /scratch are automatically deleted once they are 14 days old.
Shared scratch: Shared scratch space is accessible in one 266TB GPFS file system, /N/gpfs. The path to your scratch space is /N/gpfs/username. Files older than 60 days are purged automatically to free disk space.

For more, see At IU, how much disk space is available to me on the research systems?

Parallel computing and MPI

The Quarry cluster is not equipped with a proprietary low-latency interconnect. Parallel jobs can be run over gigabit Ethernet; the cluster is connected to a Force10 E1200 switch. OpenMPI and MPICH1 compiled with the 64-bit Intel v10 compilers are available. The relevant SoftEnv macros are @openmpi and @mpich1.

Briefly, use mpicc or mpif77 to compile and link MPI-enabled software. Then use mpirun to execute the job on multiple nodes of the cluster. In practice, you must submit your job through the TORQUE resource manager to ensure fair and efficient use of the cluster.

Compiling programs and submitting batch jobs to TORQUE (OpenPBS)

Two simple programs are listed below for your convenience:

C program

/* * Copyright 2005, The Trustees of Indiana University. * Original author: Arvind Gopu (UITS-RAC-HPC group)... * . . . [ snip ] . . . */ #include<stdio.h> int main () { double PI2=3.141592654/2.0, theta, sintheta; int i, N=4; for (i=0; i<=N; i++) { theta = i * (PI2/N); sintheta = sin (theta); printf (" Sin (%8.6lf) = %8.6lf \n", theta, sintheta); } return 0; }

Fortran program

C C Copyright 2005, The Trustees of Indiana University. C Original author: Don Berry (UITS-RAC-HPC group); C . . . [ snip ] . . . C program sine real, parameter :: PI2=3.141592654/2.0 integer, parameter :: N=4 real x,s integer i do i=0,N x=i*(PI2/N) s=sin(x) write(6,"(f11.6 f11.6)") x,s end do end

These commands use the GNU compilers (gcc and g77 for C and Fortran programs respectively) to compile the two example programs:

jdoe@Quarry: ~> gcc -o sine_c sine.c -lm jdoe@Quarry: ~> g77 -o sine_f sine.f

Submit job(s) to TORQUE

It's best to write your job in its own script file and tell TORQUE to execute that. Following is a script that runs the example sine_c program:

[jdoe@Quarry]$ cat submit_sine_c.sh #PBS -l nodes=1:ppn=1,walltime=5:00 #PBS -m ae #PBS -N job_sine_c ${HOME}/simple_quarry_jobs/sine_c

Use the script to submit your job to TORQUE:

[jdoe@Quarry]$ qsub submit_sine_c.sh 95.m2.quarry.teragrid.iu.edu

Check the job status with the qstat command:

[jdoe@Quarry]$ qstat | grep jdoe [jdoe@Quarry]$ qstat -u jdoe Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time --------------- -------- -------- ---------- ------ --- --- ------ ----- - ----- 95.m2.quarry.teragri jdoe batch job_sine_c -- 1 1 -- 00:05 R -- 96.m2.quarry.teragri jdoe batch job_sine_f -- 1 1 -- 00:05 Q --

For more about submitting and tracking jobs, see What is TORQUE/PBS?

Output and error files

Assuming your job runs to completion, you can find messages it tried to print on the console in an output file. You can specify the the directories where the output file and the error file go using the -o and -e flags respectively. The default file name of these files is of the format job_name.osequence_number and job_name.esequence_number, where job_name is the name of the job (check out the -N option) and sequence_number is the job number assigned when the job is submitted.

For example, if TORQUE assigned job id 95.m2.quarry.teragrid.iu.edu to your job, with jobname job_sine_c, the output file would be named job_sine_c.o95 and the error file would be named job_sine_c.e95. (Also see the use of the -j TORQUE directive as explained in the section above.)

[jdoe@Quarry]$ ls job_sine_* job_sine_c.e348921 job_sine_c.o348921 job_sine_f.e348922 job_sine_f.o348922

Check the output files:

[jdoe@Quarry]$ cat job_sine_c.o348921 Sin (0.000000) = 0.000000 Sin (0.392699) = 0.382683 Sin (0.785398) = 0.707107 Sin (1.178097) = 0.923880 Sin (1.570796) = 1.000000 [jdoe@Quarry]$ cat job_sine_f.o348922 0.000000 0.000000 0.392699 0.382683 0.785398 0.707107 1.178097 0.923880 1.570796 1.000000

Compiling and submitting parallel jobs

Here is a simple C program you can submit to use multiple processors:

#include <stdio.h> #include <mpi.h> /* #include "VT.h" */ /* only needed for Vampir Trace */ int main( int argc, char *argv[] ) { /* Remember: We are programming according to the SPMD model! */ /* This means that each and every processor executes this program. */ /* It is convenient to reference the typical processor in the */ /* first person, using "I" and "me" and "my." In other words, */ /* read through this program as if you were one of the processors. */ /* My ID number or my "rank": */ int myrank, length; char myname[BUFSIZ]; /* I initialize for myself the MPI API: */ MPI_Init(&argc, &argv); /* Who am I? I request my ID number: */ MPI_Comm_rank(MPI_COMM_WORLD, &myrank); MPI_Get_processor_name (myname, &length); //VT_symdef( 200, "for-loop", "Calculation" ); /* I print the standard greeting along with my ID: */ printf( "Hello, parallel worlds! This is processor %s and my rank is %d!\n", myname, myrank ); /* Finally I close the MIP API: */ MPI_Finalize(); return 0; }

Here is the script file to submit the job:

#! /bin/bash #PBS -l nodes=1:ppn=4,walltime=1:00 #PBS -m ea #PBS -N My_mpi_job # Get the number of processors NP=`wc -l $PBS_NODEFILE | awk '{print $1}'` # print out the nodes where the job runs echo "Execute node list:" sort -u $PBS_NODEFILE # change to the working directory cd simple_quarry_jobs # run an mpi job: mpirun -np $NP -machinefile $PBS_NODEFILE helloWorlds

Compile and submit:

[jdoe@Quarry]$ mpicc -o helloWorlds helloWorlds.c [jdoe@Quarry]$ qsub submit_parallel.sh 97.m2.quarry.teragrid.iu.edu

Look at the output file:

[jdoe@Quarry]$ cat job_helloWorlds.o348923 LAM 7.1.1/MPI 2 C++/ROMIO - Indiana University Hello, parallel worlds! This is processor bc27 and my rank is 1! Hello, parallel worlds! This is processor bc25 and my rank is 2! Hello, parallel worlds! This is processor bc27 and my rank is 0! Hello, parallel worlds! This is processor bc25 and my rank is 3! LAM 7.1.1/MPI 2 C++/ROMIO - Indiana University

This is document avkx in domain all.
Last modified on November 05, 2009.