ARCHIVED: Rockhopper at Indiana University

On this page:

• System overview
• System information
• System access
• Available software
• Computing environment
• Transferring your files to Rockhopper
• Application development
• Running your applications
• Reference
• Policies
• Support

System overview

Rockhopper consists of 11 Penguin Computing Altus 1804 servers, each containing four AMD Opteron 6172 12-core processors and 128 GB of RAM. The total RAM in the system is 1.5 TB. Each server chassis has a QDR (40 Gbps) InfiniBand interconnect to the cluster's switch fabric, which is then connected via four trunked 10 Gbps Ethernet links to IU's network infrastructure. For hardware configuration details, see this document's System information section.

The Rockhopper nodes run CentOS 5. Job management and scheduling are provided by the Sun Grid Engine (SGE) resource manager. The Modules system is used to simplify application and environment configuration. Users may log into the cluster via SSH, using their Penguin POD user IDs.

Back to top

System information

Back to top

System access

Requesting an account

Researchers at US institutions of higher education (with .edu domain names) or Federally Funded Research and Development Centers (FFRDCs) can purchase computing time from Penguin Computing, and then receive access to the on-demand HPC cloud service hosted on Rockhopper at IU. Prospective users can request accounts by filling out and submitting Penguin Computing's account request form. To pay for your account, you need to enter your credit card information when completing the account request form. To request an alternate financial arrangement, email Penguin Computing directly.

Methods of access

Users may log into the Rockhopper POD via SSH, using their Penguin POD user IDs. Users create SSH keys and obtain instructions on how to use them as part of the account creation process.

SSH2 clients may be used to connect to rockhopper.uits.iu.edu, which resolves to one of Rockhopper's two login nodes:

  login0.rockhopper.uits.iu.edu
  login1.rockhopper.uits.iu.edu

File transfer is supported via SCP or SFTP.

Note: Public key authentication is the only permitted authentication mechanism available on the Rockhopper cluster:

• Rockhopper does not use local passwords/passphrases for user authentication.
• Rockhopper does not use IU ADS for authentication. Rockhopper is not an IU resource; you cannot use your IU Network ID to access the POD system.

Back to top

Available software

Software packages installed on Rockhopper are made available to users via Modules, an environment management system that lets you easily and dynamically add software packages to your user environment. For a list of software modules available on Rockhopper, see Rockhopper Applications.

For more about using Modules on Rockhopper, see Modules in the Computing environment section of this document.

Back to top

Computing environment

Unix shell

Rockhopper supports the Bourne-again (bash) and Tenex C (tcsh) shells. New user accounts are assigned the bash shell by default. For more on bash, see the Bash Reference Manual.

To change your shell to tcsh on Rockhopper, email Penguin Computing directly.

Environment variables

Use the following commands, depending on your shell, to display and change the values of environment variables:

Startup scripts

• Bash: On login, bash reads and executes commands from the following directories by default (and in this order):

    /etc/profile
    ~/.bash_profile
    ~/.bashrc

  Note: The ~ (tilde) represents your home directory (e.g., ~/.bash_profile is the .bash_profile file in your home directory).

  On logout, bash reads and executes ~/.bash_logout. For more on bash startup files, see the Bash Startup Files section of the Bash Reference Manual.

• Tcsh: On login, tcsh reads and executes commands from the following directories by default (and in this order):

    /etc/csh.cshrc
    /etc/csh.login
    ~/.tcshrc (if it exists, otherwise ~/.cshrc)
    ~/.history
    ~/.login
    ~/.cshdirs

  Note: In practice, on Rockhopper, only the first two files exist. You may create the others, and add commands and variables to them as you see fit.

Modules

Rockhopper uses the Modules environment package to provide a convenient method for dynamically modifying your environment and specifying which versions of software you use.

Some common Modules commands include:

  module avail openmpi

  module load intel

  module load intel/11.1

  module switch package_A package_B

  module display package

For more, see Use modules to manage your software environment on IU research supercomputers.

Back to top

Transferring your files to Rockhopper

Rockhopper supports SCP and SFTP for transferring files. SCP is a command line utility included with OpenSSH. Basic use is:

  scp -i ~/.ssh/key-you-downloaded-from-pod [[user@]host1:]file1 [[user@]host2:]file2

For example, to copy foo.txt from the current directory on your computer to your home directory on Rockhopper, use (replacing username with your Rockhopper username):

  scp -i ~/.ssh/key-you-downloaded-from-pod foo.txt username@rockhopper.uits.iu.edu:foo.txt

You may specify absolute paths, or paths relative to your home directory:

  scp -i ~/.ssh/key-you-downloaded-from-pod foo.txt username@rockhopper.uits.iu.edu:/some/path/for/data/foo.txt

You also may leave the destination filename unspecified, in which case it will become the same as the source filename. For more, see Use SCP to securely transfer files between two Unix computers

SFTP provides file access, transfer, and management, and offers client functionality similar to FTP. For example, from a computer with a command line SFTP client (e.g., a Linux or Mac OS X workstation), you could transfer files as follows:

  $ sftp -i ~/.ssh/key-you-downloaded-from-pod  username@rockhopper.uits.iu.edu:
  Connected to rockhopper.uits.iu.edu.
  Changing to: /home/username/
  sftp> ls -l
  -rw------- 1 username group 113 May 19 2011 loadit.pbs.e897
  -rw------- 1 username group 695 May 19 2011 loadit.pbs.o897
  -rw-r--r-- 1 username group 693 May 19 2011 local_limits
  sftp> put foo.txt
  Uploading foo.txt to /home/username/foo.txt
  foo.txt                                100%   95MB   8.7MB/s   00:11    
  sftp> exit

You can also ship detachable hard drives to Penguin Computing. To make such an arrangement, email Penguin Computing directly.

Graphical SFTP clients are available for many systems. For more, see Use SFTP to transfer files

Back to top

Application development

Programming models

Rockhopper is designed to support codes that have reasonably large shared memory and/or distributed memory parallelism.

Compiling

The GNU Compiler Collection (GCC), Intel Compiler Suite, and Portland Group (PGI) compilers for C, C++, and Fortran codes are installed on the Rockhopper POD. Open MPI wrappers for these compilers are available for MPI programs. Use the -O3 switch for optimization of serial and parallel codes. The Intel Math Kernel Library (MKL) and AMD Core Math Library (ACML) are available. For debugging, the GNU Project Debugger (GDB) and Intel Debugger (IDB) are available.

For more about compiling programs on the Rockhopper POD, see Compiling on POD IU on Penguin Computing's POD IU wiki page.

Back to top

Running your applications

Rockhopper has one general-purpose queue (all.q) with "First In First Out" (FIFO) scheduling and no maximum walltime limit.

Rockhopper uses the Sun Grid Engine (SGE) for job management and scheduling. If you're familiar with TORQUE (or OpenPBS), you'll most likely be comfortable working with SGE. Many job submission (qsub) parameters are identical in TORQUE and SGE; the only difference is SGE replaces #PBS with #$. For complete user information, see the Grid Engine manual pages.

Penguin Computing also provides the PODTools package for submitting and managing jobs from your personal workstation without connecting to the Rockhopper login node. The package includes POD Shell for remote job submission, data staging tool, and file management, and POD Report for querying your core-hour and storage usage.

Submitting jobs

To submit a job to run on the Rockhopper POD, use the qsub command. If the command exits successfully, it will return a job ID, for example:

  [jdoe@login1 ~]$ qsub job.script
  Your job 2651 ("job.script") has been submitted
  [jdoe@login1 ~]$

If you need attribute values different from the defaults, but less than the maximum allowed, specify these either in the job script using SGE directives, or on the command line with the -l switch. For example, to submit a job requiring 10 hours of walltime, use:

  qsub -l time=10:00:00 job.script

Note: Command-line arguments override directives in the job script, and you may specify many attributes on the command line, either as comma-separated options following the -l switch, or each with its own -l switch. The following commands are equivalent:

  qsub -l time=10:00:00, mem=1G job.script
  qsub -l time=10:00:00 -l mem=1G job.script

Useful qsub switches include:

For a complete list of qsub parameters, see this qsub manual page.

Monitoring jobs

Use qstat for monitoring the status of a queued or running job. Switches include:

For a complete list of qstat parameters, see this qstat manual page.

Deleting jobs

Use qdel to delete queued or running jobs. Occasionally, a node will become unresponsive and unable to respond to SGE's requests to kill a job. In such cases, try qdel -f.

For a complete list of qdel parameters, see this qdel manual page.

Back to top

Reference

User and support information is available on Penguin Computing's POD IU wiki page.

Back to top

Policies

Accounts

Researchers at US institutions of higher education (with .edu domain names) or Federally Funded Research and Development Centers (FFRDCs) can purchase computing time from Penguin Computing, and then receive access to the on-demand HPC cloud service hosted on Rockhopper at IU. Prospective users can request accounts by filling out and submitting Penguin Computing's account request form. To pay for your account, you need to enter your credit card information when completing the account request form. To request an alternate financial arrangement, email Penguin Computing directly.

Acceptable use

The Rockhopper cluster is a Penguin Computing resource. For information about your responsibilities as a user of this resource, see POD Terms of Use.

Home directories

Home directories reside on a Lustre file system, with no quotas or backups.

Note: Rockhopper does not provide a separate scratch file system. Disk usage is a billable item; to make financial arrangements, email Penguin Computing directly.

Computational resources (queues)

Rockhopper has only one queue, and all jobs submitted will execute in the default queue. The only restriction is that individual jobs are limited to 128 cores.

Mail usage

Rockhopper does not provide a production mail service; however, SGE communicates via email. Mail forwarding is not configured during account creation; you should consider establishing a mail forwarding .forward file; see ARCHIVED: How do I forward my mail from a Unix account?

Scheduled downtime

Rockhopper does not have a regularly scheduled maintenance window. Information about pending outages is sent via email to account holders.

Back to top

Support

For user and support information, see Penguin Computing's POD IU wiki page, or email Penguin Computing directly.

Back to top