We have a shared computational cluster available for your use with about 140 nodes and 1400 CPUs. It consists of a blend of machines that we acquired over the years, with AMD or Intel processors, 4 – 24 cores and 8 – 256 GB of RAM per node, gigabit networking and a local scratch disk. Some machines have 24 CPU cores, 256 GB of RAM, 10 Gigabit Networking and/or an NVIDIA GPU.

Node details:

    • 8 x Dual Intel E5-2680v2 (2 x 10 core), 256 GB RAM, 10 Gigabit Ethernet
    • 16 x Dual Intel E5-2670 (2 x 8 core), 128 GB RAM, QDR InfiniBand
    • 10 x Dual Intel E5-2670 (2 x 8 core), 128 GB RAM, NDIVIA GTX680 GPU
    • 6 x Intel Core i7 2600K (quad core), 16 GB RAM, NVIDIA GTX580 GPU
    • 4 x AMD Opteron 6128 (16 core), 64 GB RAM
    • 96 x AMD Phenom II 9950 (quad core), 8 GB RAM

To obtain an account on this system, contact Luki, Alex or Duilio.

Click here for a brief crash course to get started. Current cluster status can be found here (accessible from wired LAN only).

Fair Use Policy

The cluster uses the SLURM job scheduler, which manages the job queue and dispatches jobs to compute nodes according to availability and user-defined criteria such as CPU and memory requirements specified during job submission. The job queues have time and resource limits to ensure fair use. In addition, the job priorities themselves are adjusted based on the user’s recent cluster usage (over the last 2 days) by giving light users higher priority. If you have questions or concerns regarding this policy, please contact Luki.

Important: If you are new to this cluster environment, please review the usage guidelines to find out what to do, what not to do, and where to find things. Please do NOT log in directly to compute nodes to run jobs without going through qsub or qlogin. Such activity is not being accounted for by the scheduler and will likely result in overbooking of the node’s resources, thus slowing everyone down!

Queues

There are three general purpose queues. You can submit as many jobs as you’d like at once, and they will be processed based available capacity, your overall limits and usage in the last 24 hours. If you plan to submit more than 500 similar jobs, consider structuring your work in tasks (see the “Array Jobs” section below).

Available general-purpose queues are:

  • Short Queue
    Jobs in this queue are limited to a run time of 2 hours per job and task. You can submit as many jobs as you like, but each job can not run for longer than 2 hours. The scheduler will terminate jobs that doesn’t complete within 2 hours. An error message will be logged in the job output file.
  • Medium Queue
    Similar to the short queue, but jobs are limited to 24 hours and there is also a limit of 750 jobs per user.
  • Long Queue
    No run time limit, but each user is limited to 250 jobs.

You can specify the queue you’d like to use with the sbatch command with the -p (partition) flag. If you do not specify a queue, the short queue will be selected by default, with the applicable limits.

We strongly encourage you to try to split up your work into into small tasks, but we realize that this is not always possible or practical. Short tasks ensure higher job turn over (less overall waiting) and also allow you to more easily resume work in case of node crashes or power failures.

In addition, there are specialty queues for jobs wanting to use a NVIDIA GPU, QDR InfiniBand interconnect (for MPI, etc.), or for web services that need higher priority. Scroll down for info how to submit multi-CPU jobs. If in doubt, please inquire if you need to use those queues.

Memory Requirements

Unless requested otherwise during job submission, each job is allocated 1 GB of physical memory. If a job exceeds this allocation, it will be killed by the job scheduler and an error will be logged in the job output file. You may request up to 256 GB of memory for your job. The scheduler keeps track of the available and requested memory and will dispatch your job to a compute node meeting your requirements. Note that there may be no nodes with sufficient free memory ready to take on your job even if they have available CPU capacity, and hence your job may have to wait until a node opens up. Therefore, do not request more memory than you actually need (add a fudge factor to be sure).

Memory is requested during job submission using the --mem=value as follows:

sbatch … –mem=2G script_to_run.sh

See also memory requirements for multi-CPU jobs below.

Job Output and Errors

Your job should write any output that you want to keep to a file. Anything sent to standard output and standard error will be discarded (sent to /dev/null) unless you request during job submission that this output be written to a file. This is accomplished using the -e and -o flags. To send all output to one file, use only the -o flag.

Example:

sbatch … -o output.log

Array Jobs

An array job is a set of identical tasks being differentiated only by an index number, which will be available to the script at the $SLURM_ARRAY_TASK_ID environment variable. This is primarily useful if you need to execute the same script many times with different indices such as run ID.

Example:

sbatch -a 1-100:1 script.sh

This will run script.sh on 100 nodes in parallel (or fewer if there are no suffient nodes available), each job will be given a SLURM_ARRAY_TASK_ID between 1 and 100, in increments of 1.

The task concurrency can be limited by appending the limit as follows: -a 1-100%10; this will run 100 tasks total, but only 10 concurrently. Please consider using this option to leave some nodes available for other users.

Multi-CPU Jobs

You can request more than one CPU slot for your job or task by specifying the desired number of CPU slots with the -n and -N flags to sbatch as follows:

sbatch -n 8 -N 1 script.sh

This will allocate 8 CPUs on 1 node for one job. You may request up to 20 CPUs on a single node (for example for mult-threaded applications like BLAST); if you require more CPUs, then your job will need to be split across multiple nodes. This only makes sense if your jobs uses MPI.

Important: The total amount of memory allocated to your job equals the requested memory (see above) * number of CPUs. Thus the submission below will make a total of 16 GB of memory and 8 CPUs available to your jobs:

sbatch -n 8 –mem=2G script.sh