Cassini Cluster: Introduction
We have a shared computational cluster available for your use. It consists of 128 nodes, each equipped with a AMD Phenom quad core processor, 8 GB of memory and a local scratch disk. There are 512 job slots available for general use. The nodes are connected via a Gigabit Ethernet network. Additionally, there are 33 dual CPU, dual core AMD Opteron nodes with 4 GB of memory (132 job slots), also available for general use.
To obtain an account on this system, contact Luki, Alex or Duilio.
Click here for a brief crash course to get started.
Current cluster status can be found here (accessible from wired LAN only).
Fair Use Policy
The cluster uses the Sun Grid Engine, which manages the job queue and dispatches jobs to compute nodes according to availability and user-defined criteria such as CPU and memory requirements, which are specified during job submission. The job queues have time and resource limits to ensure fair use. In addition, the job priorities themselves are adjusted based on the user's recent cluster usage (over the last 2 days) by giving light users higher priority. If you have questions or concerns regarding this policy, please contact Luki.
Important: If you are new to this cluster environment, please review the usage guidelines to find out what to do, what not to do, and where to find things.
Queues
There are three queues set up, each with different restrictions. Choose a queue that meets your requirements. If you do not specify a queue, your job will be sent to the rude, short, medium or long queue (in that order), depending on availability and requested parameters.
The rude queue has a 1 hour CPU time limit, and a 1.5 hour wall-clock time limit. Jobs exceeding these limits will be automatically killed. Any job in the short queue has the highest priority, thus such jobs will automatically suspend other jobs in the short, medium and/or long queues. You may submit as many short jobs as you wish (up to 10,000 per user).
The short queue has a 6 hour CPU time limit, and a 9 hour wall-clock time limit. Jobs exceeding these limits will be automatically killed. Any job in the short queue has the highest priority, thus such jobs will automatically suspend other jobs in the medium and/or long queues. You may submit as many short jobs as you wish (up to 10,000 per user).
The medium queue has a 24 hour CPU time and a 48 hour wall-clock time limit. Jobs in this queue will be suspended if there are any jobs in the short queue waiting to be run and there are no free nodes available. The jobs will resume automatically, however, they are not guaranteed to have uninterrupted 24 hours of run time. If this queue gets full, jobs in the long queue will be suspended to provide resources to run jobs waiting in the medium queue. You may submit as many medium jobs as you wish (up to 10,000 per user).
The long queue has no time limit, however, there is a limit of 32 concurrent jobs per user and 256 jobs total. This queue is for long-running jobs that can be interrupted to run shorter jobs with higher priority.
A queue is specified during job submission as follows:
qsub ... -q medium script_to_run.sh
You may of course specify additional arguments.
Memory Requirements
Unless requested otherwise during job submission, each job is allocated 1 GB of physical memory. If a job exceeds this allocation, it will be silently killed by the operating system (perhaps producing a core dump first). You may request up to 8 GB of memory for your job. The grid engine keeps track of the available and requested memory and will dispatch your job to a node meeting your requirements. Note that there may be no nodes with sufficient free memory ready to take on your job even if they have available CPU capacity, and hence your job may have to wait until a node opens up. Therefore, do not request more memory than you actually need (add a fudge factor to be sure).
Memory is requested during job submission using the mem= value as follows:
qsub ... -l mem=2G script_to_run.sh
To check how much memory a job is using (and the maximum usage over the lifetime of the job), run qstat on the job of interest and look for the "usage" line showing the "vmem" (current usage) and maxvmem (maximum usage):
qstat -j 1234
usage 1: cpu=12:11:03:53, mem=446671.27015 GBs, io=0.00000, vmem=1.828G, maxvmem=1.828G
Job Output and Errors
Your job should write any output that you want to keep to a file. Anything sent to standard output and standard error will be discarded (sent to /dev/null) unless you request during job submission that this output be written to a file. This is accomplished using the -e, -o and -j flags. The -j flag will merge the stadard output and standard error into a single file.
Example:
qsub ... -o output.log -e errors.log
Additional Tips
qsub
qsub is used to submit jobs for processing to the queue. Read the manual page for qsub (man qsub). Here are some useful flags:
-cwd: Run the script in the current directory. Without it, the script will be run in your home directory.
-N: Give your job a name for easy identification in qstat.
-l h=compute-X-Y: Requires that this job only runs on compute-X-Y (X = room = 1, Y = machine number). This is primary useful if you have manually copied data needed by a job to a set of nodes. We are working on setting up wildcard field for the host name.
-hold_jid JOBID: Hold this job until job with JOBID finished.
-t: Run an array of jobs (see below).
-a <time>: Define a start time of a job.
-now: Only queue this job if it can run immediately.
qstat
qstat shows job status information.
-u USERNAME: Only show jobs for user USERNAME.
-j JOBID: Get detailed information about a job with job ID JOBID.
-f: Show detailed cluster queue information.
qdel
qdel deletes a queued, running or errored job from the queue.
-u USERNAME: Delete all jobs submitted by USERNAME (yourself).
Array Jobs
An array job is a set of identical tasks being differentiated only by an index number, which will be available to the script at the $SGE_TASK_ID environment variable. This is primarily useful if you need to execute the same script many times with different indices such as run ID.
Example:
qsub -cwd -t 1-100:1 script.sh
This will run script.sh on 100 nodes in parallel (or fewer if there are no suffient nodes available), each job will be given a SGE_TASK_ID between 1 and 100, in increments of 1.
