ARCHIVED: Rockhopper at Indiana University

On this page:


System overview


Rockhopper (rockhopper.uits.iu.edu) is Penguin Computing's Penguin-On-Demand (POD) supercomputing cloud appliance hosted by Indiana University. The Rockhopper POD is a collaborative effort between Penguin Computing, IU, the University of Virginia, the University of California Berkeley, and the University of Michigan to provide supercomputing cloud services in a secure US facility. Researchers at US institutions of higher education and Federally Funded Research and Development Centers (FFRDCs) can purchase computing time from Penguin Computing, and receive access via high-speed national research networks operated by IU.

Rockhopper consists of 11 Penguin Computing Altus 1804 servers, each containing four AMD Opteron 6172 12-core processors and 128 GB of RAM. The total RAM in the system is 1.5 TB. Each server chassis has a QDR (40 Gbps) InfiniBand interconnect to the cluster's switch fabric, which is then connected via four trunked 10 Gbps Ethernet links to IU's network infrastructure. For hardware configuration details, see this document's System information section.

The Rockhopper nodes run CentOS 5. Job management and scheduling are provided by the Sun Grid Engine (SGE) resource manager. The Modules system is used to simplify application and environment configuration. Users may log into the cluster via SSH, using their Penguin POD user IDs.

Back to top

System information

System configuration Aggregate information Per-node information
Machine type High-performance, on-demand computing
Penguin Computing Altus 1804 MPP cluster
4 x 2.1 GHz 12-core AMD Opteron 6172 processors
Operating system CentOS 5  
Memory model Distributed  
Processor cores 528 48
CPUs 2.1 GHz 12-core AMD Opteron 6172 processors  
Nodes 11 compute nodes; 2 login nodes  
RAM 1.4 TB 1,333 MHz DDR3 ECC memory 128 GB 1,333 MHz DDR3 ECC memory
Computational systems details Total Per node
Processing capability Rmax = 5242 gigaFLOPS Rmax = 403 gigaFLOPS
Benchmark data HPL gigaFLOPS 2216.53 (8 nodes) HPL gigaFLOPS 288.67
Power usage To be determined 0.000230 teraFLOPS per watt
Disk space 67 TB (local) 6 TB (local)
Login nodes Rmax = 806 gigaFLOPS Rmax = 403 gigaFLOPS
Homogeneous compute nodes Rmax = 4435 gigaFLOPS Rmax = 403 gigaFLOPS
Heterogeneous compute nodes Rmax = 4435 gigaFLOPS  
Storage information
File systems Home directories are on a local Lustre file system with no quotas.
Disk space 67 TB (local)
Total scratch space Rockhopper does not include a separate scratch file system. Disk usage is a billable item.
Data Capacitor II Note: Shared scratch space on Data Capacitor II is available only to IU students, faculty, and staff.

Shared scratch space is hosted on the Data Capacitor II (DC2) file system. The DC2 scratch directory is a temporary workspace. Scratch space is not allocated, and its total capacity fluctuates based on project space requirements. The DC2 file system is mounted on IU research systems as /N/dc2/scratch and behaves like any other disk device. If you have an account on an IU research system, you can access /N/dc2/scratch/username (replace username with your IU Network ID username). Access to /N/dc2/projects requires an allocation. For details, see The Data Capacitor II and DC-WAN high-speed file systems at Indiana University. Files in shared scratch space may be purged if they have not been accessed for more than 60 days.

Scholarly Data Archive (SDA) Note: Archival storage space on IU's HPSS-based Scholarly Data Archive (SDA) is available only to IU students, faculty, and staff
Backup and purge policies Home directories are not backed up.
Quotas None
Networking Information
QDR (40 Gbps) InfiniBand Switch: Mellanox IS5030
HCA: Mellanox ConnectX 2, 1-port QSFP, QDR
1 GBps Ethernet

Back to top

System access

Requesting an account

Researchers at US institutions of higher education (with .edu domain names) or Federally Funded Research and Development Centers (FFRDCs) can purchase computing time from Penguin Computing, and then receive access to the on-demand HPC cloud service hosted on Rockhopper at IU. Prospective users can request accounts by filling out and submitting Penguin Computing's account request form. To pay for your account, you need to enter your credit card information when completing the account request form. To request an alternate financial arrangement, email Penguin Computing directly.

Methods of access

Users may log into the Rockhopper POD via SSH, using their Penguin POD user IDs. Users create SSH keys and obtain instructions on how to use them as part of the account creation process.

SSH2 clients may be used to connect to rockhopper.uits.iu.edu, which resolves to one of Rockhopper's two login nodes:

  login0.rockhopper.uits.iu.edu
  login1.rockhopper.uits.iu.edu

File transfer is supported via SCP or SFTP.

Note: Public key authentication is the only permitted authentication mechanism available on the Rockhopper cluster:

  • Rockhopper does not use local passwords/passphrases for user authentication.
  • Rockhopper does not use IU ADS for authentication. Rockhopper is not an IU resource; you cannot use your IU Network ID to access the POD system.

Back to top

Available software

Software packages installed on Rockhopper are made available to users via Modules, an environment management system that lets you easily and dynamically add software packages to your user environment. For a list of software modules available on Rockhopper, see Rockhopper Applications.

For more about using Modules on Rockhopper, see Modules in the Computing environment section of this document.

Back to top

Computing environment

Unix shell

Rockhopper supports the Bourne-again (bash) and Tenex C (tcsh) shells. New user accounts are assigned the bash shell by default. For more on bash, see the Bash Reference Manual.

To change your shell to tcsh on Rockhopper, email Penguin Computing directly.

Environment variables

Use the following commands, depending on your shell, to display and change the values of environment variables:

Shell Display value Change value
bash echo $VARNAME export VARNAME=VALUE
tcsh echo $VARNAME setenv VARNAME VALUE

Startup scripts

  • Bash: On login, bash reads and executes commands from the following directories by default (and in this order):
      /etc/profile
      ~/.bash_profile
      ~/.bashrc

    Note: The ~ (tilde) represents your home directory (e.g., ~/.bash_profile is the .bash_profile file in your home directory).

    On logout, bash reads and executes ~/.bash_logout. For more on bash startup files, see the Bash Startup Files section of the Bash Reference Manual.

  • Tcsh: On login, tcsh reads and executes commands from the following directories by default (and in this order):
      /etc/csh.cshrc
      /etc/csh.login
      ~/.tcshrc (if it exists, otherwise ~/.cshrc)
      ~/.history
      ~/.login
      ~/.cshdirs

    Note: In practice, on Rockhopper, only the first two files exist. You may create the others, and add commands and variables to them as you see fit.

Modules

Rockhopper uses the Modules environment package to provide a convenient method for dynamically modifying your environment and specifying which versions of software you use.

Some common Modules commands include:

Command Action
module avail List all software packages available on the system.
module avail package List all versions of package available on the system; for example:

  module avail openmpi
module list List all packages currently added to your user environment.
module load package Add the default version of the package to your user environment; for example:

  module load intel
module load package/version Add the specified version of the package to your user environment; for example:

  module load intel/11.1
module unload package Remove the specified package from your user environment.
module swap package_A package_B Swap the loaded package (package_A) with another package (package_B). This is synonymous with:

  module switch package_A package_B
module show package Show the changes loading the specified package makes to your user environment (e.g., environment variables set, library paths added). This is synonymous with:

  module display package

To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

For more about using Modules to configure your user environment, see On Big Red II, Big Red II+, Karst, and Mason at IU, how do I use Modules to manage my software environment?

Back to top

Transferring your files to Rockhopper

Rockhopper supports SCP and SFTP for transferring files. SCP is a command line utility included with OpenSSH. Basic use is:

  scp -i ~/.ssh/key-you-downloaded-from-pod [[user@]host1:]file1 [[user@]host2:]file2

For example, to copy foo.txt from the current directory on your computer to your home directory on Rockhopper, use (replacing username with your Rockhopper username):

  scp -i ~/.ssh/key-you-downloaded-from-pod foo.txt username@rockhopper.uits.iu.edu:foo.txt

You may specify absolute paths, or paths relative to your home directory:

  scp -i ~/.ssh/key-you-downloaded-from-pod foo.txt username@rockhopper.uits.iu.edu:/some/path/for/data/foo.txt

You also may leave the destination filename unspecified, in which case it will become the same as the source filename. For more, see In Unix, how do I use SCP to securely transfer files between two computers?

SFTP provides file access, transfer, and management, and offers client functionality similar to FTP. For example, from a computer with a command line SFTP client (e.g., a Linux or Mac OS X workstation), you could transfer files as follows:

  $ sftp -i ~/.ssh/key-you-downloaded-from-pod  username@rockhopper.uits.iu.edu:
  Connected to rockhopper.uits.iu.edu.
  Changing to: /home/username/
  sftp> ls -l
  -rw------- 1 username group 113 May 19 2011 loadit.pbs.e897
  -rw------- 1 username group 695 May 19 2011 loadit.pbs.o897
  -rw-r--r-- 1 username group 693 May 19 2011 local_limits
  sftp> put foo.txt
  Uploading foo.txt to /home/username/foo.txt
  foo.txt                                100%   95MB   8.7MB/s   00:11    
  sftp> exit

You can also ship detachable hard drives to Penguin Computing. To make such an arrangement, email Penguin Computing directly.

Graphical SFTP clients are available for many systems. For more, see What is SFTP, and how do I use it to transfer files?

Back to top

Application development

Programming models

Rockhopper is designed to support codes that have reasonably large shared memory and/or distributed memory parallelism.

Compiling

The GNU Compiler Collection (GCC), Intel Compiler Suite, and Portland Group (PGI) compilers for C, C++, and Fortran codes are installed on the Rockhopper POD. Open MPI wrappers for these compilers are available for MPI programs. Use the -O3 switch for optimization of serial and parallel codes. The Intel Math Kernel Library (MKL) and AMD Core Math Library (ACML) are available. For debugging, the GNU Project Debugger (GDB) and Intel Debugger (IDB) are available.

For more about compiling programs on the Rockhopper POD, see Compiling on POD IU on Penguin Computing's POD IU wiki page.

Back to top

Running your applications

Rockhopper has one general-purpose queue (all.q) with "First In First Out" (FIFO) scheduling and no maximum walltime limit.

Rockhopper uses the Sun Grid Engine (SGE) for job management and scheduling. If you're familiar with TORQUE (or OpenPBS), you'll most likely be comfortable working with SGE. Many job submission (qsub) parameters are identical in TORQUE and SGE; the only difference is SGE replaces #PBS with #$. For complete user information, see the Grid Engine manual pages.

Penguin Computing also provides the PODTools package for submitting and managing jobs from your personal workstation without connecting to the Rockhopper login node. The package includes POD Shell for remote job submission, data staging tool, and file management, and POD Report for querying your core-hour and storage usage.

Submitting jobs

To submit a job to run on the Rockhopper POD, use the qsub command. If the command exits successfully, it will return a job ID, for example:

  [jdoe@login1 ~]$ qsub job.script
  Your job 2651 ("job.script") has been submitted
  [jdoe@login1 ~]$

If you need attribute values different from the defaults, but less than the maximum allowed, specify these either in the job script using SGE directives, or on the command line with the -l switch. For example, to submit a job requiring 10 hours of walltime, use:

  qsub -l time=10:00:00 job.script

Note: Command-line arguments override directives in the job script, and you may specify many attributes on the command line, either as comma-separated options following the -l switch, or each with its own -l switch. The following commands are equivalent:

  qsub -l time=10:00:00, mem=1G job.script
  qsub -l time=10:00:00 -l mem=1G job.script

Useful qsub switches include:

Switch Action
-q <queue_name> Specify a user-selectable queue.
-r Make job re-runnable.
-a <date_time> Execute the job only after a specified date and time.
-V Export environment variables in your current environment to the job.

For a complete list of qsub parameters, see this qsub manual page.

Monitoring jobs

Use qstat for monitoring the status of a queued or running job. Switches include:

Switch Action
-u <user_list> Display jobs for users in the user list.
-sa Display all jobs.
-sr Display running jobs.
-r Display the full listing of jobs (excessive detail).
-n Display nodes allocated to jobs.

For a complete list of qstat parameters, see this qstat manual page.

Deleting jobs

Use qdel to delete queued or running jobs. Occasionally, a node will become unresponsive and unable to respond to SGE's requests to kill a job. In such cases, try qdel -f.

For a complete list of qdel parameters, see this qdel manual page.

Back to top

Reference

User and support information is available on Penguin Computing's POD IU wiki page.

Back to top

Policies

Accounts

Researchers at US institutions of higher education (with .edu domain names) or Federally Funded Research and Development Centers (FFRDCs) can purchase computing time from Penguin Computing, and then receive access to the on-demand HPC cloud service hosted on Rockhopper at IU. Prospective users can request accounts by filling out and submitting Penguin Computing's account request form. To pay for your account, you need to enter your credit card information when completing the account request form. To request an alternate financial arrangement, email Penguin Computing directly.

Acceptable use

The Rockhopper cluster is a Penguin Computing resource. For information about your responsibilities as a user of this resource, see POD Terms of Use.

Home directories

Home directories reside on a Lustre file system, with no quotas or backups.

Note: Rockhopper does not provide a separate scratch file system. Disk usage is a billable item; to make financial arrangements, email Penguin Computing directly.

Computational resources (queues)

Rockhopper has only one queue, and all jobs submitted will execute in the default queue. The only restriction is that individual jobs are limited to 128 cores.

Mail usage

Rockhopper does not provide a production mail service; however, SGE communicates via email. Mail forwarding is not configured during account creation; you should consider establishing a mail forwarding .forward file; see ARCHIVED: How do I forward my mail from a Unix account?

Scheduled downtime

Rockhopper does not have a regularly scheduled maintenance window. Information about pending outages is sent via email to account holders.

Back to top

Support

For user and support information, see Penguin Computing's POD IU wiki page, or email Penguin Computing directly.

Back to top

This document was developed with support from National Science Foundation (NSF) grants 1053575 and 1548562. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.

This is document bbjb in the Knowledge Base.
Last modified on 2015-06-30 00:00:00.

  • Fill out this form to submit your issue to the UITS Support Center.
  • Please note that you must be affiliated with Indiana University to receive support.
  • All fields are required.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.

  • Fill out this form to submit your comment to the IU Knowledge Base.
  • If you are affiliated with Indiana University and need help with a computing problem, please use the I need help with a computing problem section above, or contact your campus Support Center.

Please provide your IU email address. If you currently have a problem receiving email at your IU account, enter an alternate email address.