On Big Red II at IU, how do I use PCP to bundle multiple serial jobs to run them in parallel?
On this page:
- Adding PCP to your Big Red II user environment
- Bundling and running multiple serial jobs in parallel on Big Red II
- Getting help
Parallel Command Processor (PCP), developed by the Ohio Supercomputer Center (OSC) and the National Institute for Computational Sciences (NICS), is an application that lets you bundle multiple serial jobs and run them concurrently.
Using PCP to bundle and run your serial jobs in parallel lets you make efficient use of all the cores on a compute node. Conversely, running one serial job at a time can waste more than 90% of a node's computational power. PCP also lets you request multiple nodes for your jobs.
PCP is especially useful for running parametric studies and Monte Carlo simulations.
Adding PCP to your Big Red II user environment
At Indiana University, PCP is available on Big Red
II. To add PCP to your user environment on Big Red II, load the
pcp/2008 module; on the command line, enter:
To make permanent changes to your environment, edit your
~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?
For more about the Modules package, see the
module manual page
page. Additionally, see On Big Red II, Mason, Quarry, and Rockhopper at IU, how do I use Modules to manage my software environment?
Bundling and running multiple serial jobs in parallel on Big Red II
Following is an example illustrating how you can bundle multiple serial jobs and run them in parallel on Big Red II using PCP:
- Assume you have a set of 32 serial jobs to run; list the jobs in a
text file (e.g.,
list.txt): ./a.out > o1.txt ./a.out > o2.txt ./a.out > o3.txt ....... ./a.out > o31.txt ./a.out > o32.txt
- Write a TORQUE job script (e.g.,
pbs_script.sh) that requests the cores needed to run all of your serial jobs (32) and launches PCP to execute the jobs listed in
list.txt: #!/bin/bash #PBS -l nodes=1:ppn=32 #PBS -l walltime=00:10:00 #PBS -N my_job #PBS -q cpu #PBS -V aprun -n 32 pcp list.txt
Note: Big Red II uses the TORQUE resource manager in combination with the Moab job scheduler for job management.
- Submit the script (e.g.,
pbs_script.sh) using the
qsubcommand; on the command line, enter: qsub pbs_script.sh
As a result, the 32 serial jobs in
list.txt will run
in parallel on Big Red II.
Support for this system is provided by the UITS High Performance Systems (HPS) and Scientific Applications and Performance Tuning (SciAPT) groups. If you have system-specific questions, email the HPS group. If you have questions about compilers, programming, scientific/numerical libraries, or debuggers on this system, email the SciAPT group.
Last modified on September 30, 2013.