Tiger Tutorial

Running Programs

Programs are scheduled to run on Tiger using the sbatch command, a component of Slurm. Your job will be put into the appropriate quality of service, based on the requirements that you describe. See the information about Job Scheduling on the main Tiger page and the sbatch man page for details.

Programming Tiger

The Gnu and Intel compilers are installed on Tiger. The standard MPI implementation for Tiger is OpenMPI, an MPICH compatible library that supports the Infiniband infrastructure.
 
To set up your environment correctly on Tiger, it is highly recommended to use the module facility. This is a utility to correctly set your environment without having to know all the paths to the executables. Different environments can be set quickly allowing useful comparisons of code compiled with different executables. In most cases a simple module load openmpi command can be issued setting up your environment to use the latest openmpi as well as the Intel compilers.

Compiling parallel MPI programs

# loads the openmpi environment as well as the Intel compilers
module load openmpi
mpif90 myMPIcode.f
mpicc myMPIcode.c
 

Submitting a Job

Once the executable is compiled, a job script will need to be created for the scheduler. For this machine there are a total of 40 processor cores per node. Special node properties include the large memory nodes and GPUs. These nodes can be chosen by adding those special flags to the requirements: #SBATCH -C fat (for the large memory nodes) and - -gres=gpu:2 (for 2 gpus per node, see below for more).
Here is a sample script which uses 80 processors allocated as 40 processors/node on 2 nodes:
#!/bin/bash
# parallel job using 80 processors. and runs for 4 hours (max)
#SBATCH -N 2
#SBATCH --ntasks-per-node=40
#SBATCH -t 4:00:00
# sends mail when process begins, and
# when it ends. Make sure you define your email
# address.
#SBATCH --mail-type=begin
#SBATCH --mail-type=end
#SBATCH --mail-user= yourNetID@princeton.edu


module load openmpi
cd /home/yourNetID/mpi_directory
mpiexec ./a.out

(Change "NetID" to your NetID in the above script.)
To submit the job to the batch queuing system use:
 
sbatch parallel.cmd

GPU Usage

To allocate a job using the GPUs (up to 4 per node) you will need to add another specifier to the #SBATCH command requesting the number of GPUs as well as sending the job to the GPU partition. Because of the way that the GPU's are laid out on a node, you must allocate both sockets if you require more than 2 GPUs.

#SBATCH -N 1
#SBATCH --ntasks-per-node=4
#SBATCH --ntasks-per-socket=2
#SBATCH --gres=gpu:4
Command Synopsis
sinfo Shows how nodes are being used..
sshare/sprio Shows the priority assigned to queued jobs.
squeue or qstat Shows jobs in the queues.
smap/sview

A graphical display of the queues.

slurmtop A text based view of the cluster nodes.