Use PCP to bundle multiple serial jobs to run in parallel on IU research supercomputers

On this page:


Overview

Parallel Command Processor (PCP), developed by the Ohio Supercomputer Center (OSC) and the National Institute for Computational Sciences (NICS), is an application that lets you bundle multiple serial jobs and run them concurrently.

Using PCP to bundle and run your serial jobs in parallel lets you make efficient use of all the cores on a compute node. Conversely, running one serial job at a time can waste more than 90% of a node's computational power. PCP also lets you request multiple nodes for your jobs.

PCP is especially useful for running parametric studies and Monte Carlo simulations.

Add PCP to your user environment

At Indiana University, PCP is available on the IU research supercomputers. To use PCP on an IU research supercomputer, you must add it to your user environment.

  • On Carbonate, to add the default version of PCP and its prerequisite module (openmpi/intel) to your user environment, enter:
    module load openmpi/intel pcp
    
  • On Big Red 3 or Quartz, to add the default version of PCP to your user environment, enter:
    module load pcp
    

You can save your customized user environment so that it loads every time you start a new session; for instructions, see Use modules to manage your software environment on IU research supercomputers.

Run multiple serial jobs in parallel with PCP

For a parallel job with N processors allocated, the PCP manager process reads the first N-1 commands in the command stream and distributes them to the other N-1 processors. As processes complete, the PCP manager reads the next process in the stream and runs it on an idle core.

For example, to run 47 serial jobs in parallel:

  1. Create a text file that lists each job, one per line (for example, list.txt):
    ./a.out > o1.txt 
    ./a.out > o2.txt
    ./a.out > o3.txt
    
    .......
    
    ./a.out > o46.txt
    ./a.out > o47.txt
    
  2. Create a Slurm job script (for example, slurm_script.sh). In your script:
    • Use the --ntasks-per-node option to request the cores needed to run all the jobs in your job list and the PCP manager process. (For this example, the script would need to request 48 cores to run the 47 jobs in list.txt plus the PCP manager process.)
    • Use srun to launch PCP and execute the jobs in your job list file.

    For example, on Big Red 3, the following Slurm script will launch PCP and run the 47 jobs in the job list file (list.txt):

    #!/bin/bash
    
    #SBATCH -J job_name
    #SBATCH -p general
    #SBATCH -o filename_%j.txt
    #SBATCH -e filename_%j.err
    #SBATCH --mail-type=ALL
    #SBATCH --mail-user=username@iu.edu
    #SBATCH --nodes=1
    #SBATCH --ntasks-per-node=48
    #SBATCH --cpus-per-task=1
    #SBATCH --time=02:00:00
    #SBATCH --mem=58G
    
    module load pcp
    srun --tasks-per-node=48 pcp list.txt
    
  3. Submit the script (for example, slurm_script.sh) from the command line using sbatch:
    sbatch slurm_script.sh
    

When the job starts, the 47 serial jobs in list.txt will run in parallel. When the PCP manager runs out of commands to run, it will wait for any remaining running processes to complete, and then shut itself down.

Get help

Support for IU research supercomputers, software, and services is provided by various teams within the Research Technologies division of UITS.

For general questions about research computing at IU, contact UITS Research Technologies.

For more options, see Research computing support at IU.

This is document bdka in the Knowledge Base.
Last modified on 2021-07-29 17:08:51.