Bioops

Bioinformatics=(ACGAAG->AK)+(#!/bin/sh)+(P(A|B)=P(B|A)*P(A)/P(B))

Introduction to Using PBS

| Comments

Introduction to Using PBS

Introduction

PBS is the batch scheduler running on the Sun Opteron cluster, midnight. Unlike interactive jobs, batch jobs are controlled via scripts. These scripts tell the system which resources a job will require and how long they will be needed.

Command Interface to PBS

Frequently used PBS commands:

Command	Purpose
qmap	Displays a grid of current jobs along with a list of queued and running jobs.(ARSC Only) (see qmap -h for additional job and queue information)
qsub	Submits a job to the PBS batch scheduler. (see also man qsub)
qdel	Removes a job from the queue. This includes all running, waiting, and held jobs.(also see man qdel)
qstat -f	Displays more information about a particular job (also see man qstat)

Queues

A user’s job is submitted to the queue with the “qsub” command. The user must be certain that the specific resources requested (such as number of nodes and walltime hours) are within the ranges offered by the particular queue.

The command “qmap -r” shows the names of the queues available and their maximum walltimes. For a more verbose narrative on queue usage, type the following at the command prompt:

news queues

Submitting Jobs to PBS

The command:

qsub <PBS script>

will submit the given script for processing. The script contains information PBS needs to allocate resources for your job, directions for handling standard I/O streams, and instructions to run the job. Example scripts are included below .

Running Interactive Jobs

You are encouraged to use the PBS batch system, but may run interactive jobs as well. An interactive command is simply typed at the prompt in a terminal window. Standard error and standard output are printed to the terminal, redirected to a file, or piped to another command using appropriate Unix shell syntax.

You can spawn an interactive job using the following command:

qsub -q debug -l select=1:ncpus=4:node_type=4way -I

Once your job is started, you may then run interactive commands on the compute node(s) PBS assigned to your session.

Monitoring Queues and Requests

The command:

qmap

will show all jobs currently running or queued on the system. For details about your particular jobs, issue the command:

qmap -u <user name>

Canceling Queued and Running Jobs

The command:

qdel <job id>

where <job id> is obtained from the “Job Id” field of the qmap output. This command will remove the job from the queue and terminate the job if it is running.

Example Scripts

Example #1 MPI using Sun Fire x2200 nodes (4way nodes)

#!/bin/bash
#PBS -q standard
#PBS -l select=8:ncpus=4:node_type=4way
#PBS -l walltime=08:00:00
#PBS -j oe
cd $PBS_O_WORKDIR

mpirun -np 32 ./myprog

Here is a line-by-line breakdown of the keywords and their assigned values listed in this MPI script:

#!/bin/bash Specifies the shell to be used when executing the command portion of the script.

#PBS -q standard Specifies which queue the job will be submitted to.

#PBS -l select=8:ncpus=4:node_type=4way Requests 8 “blocks” of 4 processors on x2200 nodes. You can also think of this as requesting 8 nodes, running 4 tasks on each of those nodes, with the nodes being 4way (i.e. x2200) only.

#PBS -l walltime=08:00:00 Requests that the running job be allowed to run for a maximum of 8 hours.

#PBS -j oe Joins the output and error files.

cd $PBS_O_WORKDIR Change to the initial working directory.

mpirun -np 32 ./myprog Runs the mpi program with a total of 32 tasks.

Example #2 OpenMP using Sun Fire x4600 nodes (16way nodes)

#!/bin/bash
#PBS -q standard
#PBS -l select=1:ncpus=16:node_type=16way
#PBS -l walltime=08:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=16
export PSC_OMP_AFFINITY=TRUE
./myprog

Here is a line-by-line breakdown of the example OpenMP script:

#!/bin/bash Specifies the shell to be used when executing the command portion of the script.

#PBS -q standard Specifies which queue the job will be submitted to.

#PBS -l select=1:ncpus=16:node_type=16way Requests 1”block” of 16 processors on an x4600 node. You can also think of this as requesting 1 node, running 16 tasks on the node, with the node being 16way.

#PBS -l walltime=08:00:00 Requests that the running job be allowed to run for a maximum of 8 hours.

#PBS -j oe Joins the output and error files.

cd $PBS_O_WORKDIR Change to the initial working directory.

export OMP_NUM_THREADS=16 Sets the number of OpenMP threads to 16.

export PSC_OMP_AFFINITY=TRUE Sets the threads to have CPU affinity.

./myprog Runs the OpenMP program.

Example #3 Data Staging Script

#!/bin/bash
#PBS -q transfer
#PBS -l select=1:ncpus=1
#PBS -l walltime=04:00:00
#PBS -j oe
cd $PBS_O_WORKDIR
cp -r $ARCHIVE_HOME/mydataset/* . || exit 1
qsub mpi_job.pbs

Here is a line-by-line breakdown of the keywords and their assigned values listed in this data staging script:

#!/bin/bash Specifies the shell to be used when executing the command portion of the script.

#PBS -q transfer Specifies to run a job in the transfer queue.

#PBS -l select=1:ncpus=1 Requests 1 node to run 1 process in the queue. Data transfer jobs must be run in serial.

#PBS -l walltime=04:00:00 Requests that the running job be allowed to run for a maximum of 4 hours.

#PBS -j oe Joins the output and error files.

cd $PBS_O_WORKDIR Change to the initial working directory.

cp -r $ARCHIVE_HOME/mydataset/* . || exit 1 Copies files from long term storage to the current working directory.

qsub mpi_job.pbs Submits a new job to the batch scheduler once the data transfer is complete.

Comments