The qsub Command

Specifying Job Resource Requirements

Specifying Per-Job CPU Time Limit

Specifying Per-Job Memory Size Limit

Specifying a Time For Submitting Batch Job

PBS Environment Variables

Submitting a Job

Submitting PBS Jobs With Input From a File

Submitting a PBS Job With Input From the Terminal

The qsub Command

The qsub command submits a series of commands to be executed as a batch job.

The qsub Command Format

The syntax for the qsub command is:

qsub [ option ] [ script-file ]

where option is one or more of the options to the qsub command, and script-file is the name of the file that contains the shell script (if one is used)  for the job submission. If you do not specify a script-file, then the commands are taken directly from the terminal (standard input). If the job commands are entered from the command line, the end of job is indicated by typing control-d on a blank line. Submission of a script file is the preferred method as it avoids errors due to typing mistakes.

The option field is usually of the format:

-flag [ param ]

Example:

-A g12345

Some options have multiple parameters which may be entered using a comma separated list in the following format:

-flag [ param=value[,param=value,...] ]

Example:

-l mem=50mw,cput=02:00:00

Note that there are no spaces on either side of the comma.

This may also be expressed as individual instances of the same option:

-flag param=value -flag param=value

Example:

-l mem=50mw -l cput=02:00:00

The most frequently used option for the qsub command are identified in Table below.

Options for the qsub command may be entered on the command-line, in the forms shown above, or embedded as comments at the top of the script file before the first executable command using the identifying string #PBS followed by the desired options. For example:

#PBS -l cput=2:30:00
...
# first executable command comes next

#PBS directives encountered after the first executable command will be considered comments. PBS options entered on the command line take precedence over those found in the script file.

Explanation of qsub Options

Specifying Job Resource Requirements

One of the most important aspects of creating a PBS batch job  is to accurately specify the system resource requirements of the job. The reason for being as accurate as possible, rather than, for example, asking for the maximums, is to allow the scheduler to maximize system usage.

The -l flag, described in the Table above, is used to request the system resources the job needs. Multiple resources may be requested in a single instance of the flag with a comma separated list, or multiple instances of the flag may be used.

For example:

% qsub -l cput=1:00:00,mem=40MW runjob         OR
% qsub -l cput=1:00:00 -l mem=40MW runjob

will produce the same result of requesting 1 hour of CPU time and 40 megawords of memory for the job that is contained in a script file named runjob.

The following table lists most of the resource request parameters recongized by the scheduler for the qsub -l flag.

Frequently Used Job Resource(-l) Options

Specifying Per-Job CPU Time Limit

The -lcput is used with the qsub command to specify a per-job maximum CPU time limit. If the combined total time of all processes associated with the job exceed the time limit set by this parameter, the job will terminate. The syntax of qsub -lcput is:

qsub -lcput=time-limit [ script-file ] where time-limit is of the format:

[[hours:]minutes:]seconds[.milliseconds]

For example:

-lcput=4:30:00

PBS will take the smallest unit possible for any specification, meaning, for example:

-lcput=4:30:00
-lcput=4:30

are not equivalent. The former reads: four hours, thirty minutes, while the latter reads: four minutes, thirty seconds.

The run-time of any program may vary, relative to the system load at the time of the run. Therefore, be sure to include a little extra time to account for these variations.

There is also a per-process time limit parameter, pcput, which has the same format, but places a limit on the amount of time any single process may use, as opposed to the limit for the whole job. This parameter should always be used in conjunction with the "job" time limit (cput). If cput is omitted, the default time of 5 minutes is set. Any job requiring more than this amount will abort even if the "process" time limit (pcput) has been set to a higher value.
 

Specifying Per-Job Memory Size Limit

The -lmem flag is used with the qsub command to specify a per-job maximum memory size limit for all processes required to complete the batch job. If any of process, or combination of concurrent processes, in the job exceed the size limit, the request terminates. The syntax of qsub -lmem is:

qsub -lmem=size-limit [ script-file ] where size-limit refers to the maximum amount of memory allowed for the job. The default size unit is measured in bytes, although it is possible to specify other units. Acceptable memory size units include: words (w), kilobytes (kb), kilowords (kw), megabytes (mb), megawords (mw), gigabytes (gb), and gigawords (gw).

The following example specifies the size limit in megawords:

qsub -lmem=16mw -lcput=1200 myjob There is also a per-process memory limit, pmem, which follows the same format, but places a limit on any one process, as opposed to the whole job. Failing to specify the "job" memory limit results in the default value, 4 MW. This will cause a job requiring more than the default to fail regardless of whether the "process" memory limit has been set to something higher.

Specifying a Time For Submitting Batch Job

qsub -a is used to specify the date and time after which the batch job should be submitted. This does not mean that the job will start at this time. If your job is still queued when the specified time arrives, it will remain queued and will not run until PBS initiates it. If it is designated by PBS to run before the specified time, then it will go into a wait state until the specified time arrives. The format of qsub -a is:

qsub -a date-time [ script-file ]

or in a script:

#PBS -a date-time

where date-time corresponds to a particular date and time. The date_time argument is in the following form:

[[[[CC]YY]MM]DD]hhmm[.SS]

The following are examples of valid date and time specifications:

qsub -a 0800                   Submit at the next 8:00 A.M.
qsub -a 199604010000     Submit at 12:00 A.M. on April 1, 1996
qsub -a 09301800            Submit at 6:00 P.M. on September 30

Without the -a date-time option, the current date and time are assumed (that is, the job is submitted immediately). The job is also submitted immediately if the date and time specified have already passed.
 

 Specifying Job Dependency and File Staging With the -W Flag

The -W option provides two categories of options that allow you to control the order your jobs run and that identify specific files to be copied to temporary locations during job execution. The first category is designated by the depend= string and provides dependency controls based on the PBS job identifier and the state of the job or jobs identified. The second category is designated by the strings stagein and stageout= and provides the ability to identify files to copy files to temporary locations before (stagein) execution for use during job processing, and copy out (stageout) to permanent storage after job completion.

The following table describes some of the dependency options. These options make it possible to control the order in which your jobs are run, or to make one job dependent on the successful completion or failure of another job.

 Job Dependency Options (-W depend=option)

      The following example illustrates one way to use the dependency options. In this example the first job in a series is submitted to the machine TIFR C7  through the pbs server, TIFR C7. The second job is then submitted specifying that it may run only after the successful completion of the first job.

tifrc7 %  qsub job_01
4500.tifrc7.tifr.res.in
tifrc7% qsub -W depend=afterok:4500@tifrc7 job_02
4502.tifrc7.tifr.res.in
tifrc7 %

A couple other dependency options provide the ability to indicate that you want certain jobs to run at the same time.  For more information on the -W depend= options see the man page on qsub.
 

 PBS Environment Variables

PBS provides several environment variables that may be used during job execution. These variables are created when the job begins to run prior to the execution of the first command in the job script. These variables are included in the job's global envirnoment and may be used in the script for a variety of purposes including user identification, the location of the directory from which the job was submitted, the job's PBS ID, and other useful information.

The following is a table of the environment variables provided by PBS:

PBS Environment Variables

       An alternative to PBS_O_WORKDIR is to use /R/machine_name/u/userid/pwd  as the full path to the directory from which the job was submitted. 

 Submitting a Job

A job is submitted to the batch queues by executing the qsub command. This may be done by entering commands interactively through standard input (your terminal), or by specifying a script file on the qsub command-line. In general, a script is considered safer. For both cases, the submitted commands are stored in a temporary file so that later changes will not affect previously queued batch jobs. Immediately after you submit a PBS request, the system returns a confirmation message specifying the identification number of the job. After completing the batch job execution, PBS returns standard output and the standard error to the directory from which the qsub command was invoked or as directed by qsub directives (-e path -o path).

Whether the mode of input is from a terminal or a file, a job always starts in your home directory. Therefore, if the job requires access to a specific file not residing in your home directory, either change to the directory (cd) containing the file, as a part of the batch job, or give an explicit path to the file. PBS provides an environment variable, $PBS_O_WORKDIR, which is set to the directory from which the job is submitted.

Secondly, you must remember which shell PBS is using for interpreting the submitted batch commands. The default shell is determined by the first character of the script, unless you use the -S option to specify a particular shell. You can specify /bin/sh (POSIX shell), /bin/csh (C shell), or /bin/ksh (korn shell).

The following two sections describe how to submit a job, first using a script file, and secondly, by entering input at the terminal. 

Submitting PBS Jobs With Input From a File

The qsub Command Format above, the qsub command format was described as:

qsub [ option ] [ script-file ]

In order to provide job input from a file the script-file field must be specified on the command line. For example:

% qsub runjob

where runjob is the name of a file located in the current directory, containing comments and executable commands. The option field is used to supply resource and other requirements for the job. #PBS directives specified in the script are overridden by options specified on the command-line. A common practice is to set up a script that includes all the directives necessary to run the job. On occasion, these directives will be overridden by entering the particular option on the command-line.

When the qsub command execution completes a PBS job identifier is returned for tracking purposes. EG:

% qsub runjob
9012.tifrc7.tifr.res.in

A copy of the script is saved by PBS for later execution. The originating script can then be modified without affecting the job you just submitted. When the job is scheduled to run the commands in the script will be executed by the designated command shell. At job completion, or job failure, the standard output and standard error of the job are returned to the owner. These files (or file, depending on qsub options used) are returned as the first 15 characters of the job name followed by dot (.) followed by the letter "o" for standard output, or the letter "e" for standard error. For example:

% ls   in the directory the from which job was submitted  will list these output files -- > job.e9012 job.o9012

Submitting a PBS job with input from a file allows you to submit a number of commands and options to be processed collectively. This speeds up the submission of multiple commands and saves repetitive typing. In the event of an unsuccessful run, it also allows you to check for errors made during job submission, or in the commands executed.

Submitting a PBS Job With Input From the Terminal

The second method of submitting a job to the batch queues is by entering it by hand at the terminal. This method is most useful for short scripts that require minimal typing. By it's very nature, this method is subject to typographical errors, and is not recommended for regular use. Saving a script in a file makes it easier to debug should problems arise.

The steps to submitting a job via terminal input are:
enter the qsub command and press return (enter).

You may include any of the PBS options, but, you must not specify a job script. When the qsub command does not find a script file it will prompt you for input from the keyboard.

enter shell commands one line at a time

Anything valid in a script may be entered. The only problem is that you can not go back and fix a typographical error once you have hit return.

enter control-d on a blank line

At this point, PBS will return the job identifier, just as it does when submitting a script file.

The following example submits a short job to the long  queue with a memory limit of 10MW and a job CPU time limit of 100 seconds.

% qsub -l mem=10MW,cput=100
./testjob
246.tifrc7.tifr.res.in

% qstat -au xyz
-------------------------
PBS Batch Request Summary
-----------------------------------
PBSid  Jobname    Username Queue      NDS TSK  SID  REQMEM  REMTIME S
------ ---------- -------- ---------- --- --- ----- ------ -------- - ---- ------------------------
246         test              xyz             long            -           -        0        10mw          00:01:25     R

As with submitting a script file, the standard output and standard error files are returned to the directory from which the job was submitted unless otherwise modified by qsub options or #PBS directives.