Hardware

A 380 node, 3144 core, high-performance computing cluster, modeled after national supercomputing centers, supporting large-scale computation, low-latency networking for MPI workloads, large memory systems, and high-performance parallel filesystems.

• 32 dual-processor, 12-core (Intel E5-2650 v4) Dell PowerEdge R430 nodes, with 64 GB each, 10Gbit/s Ethernet-connected.
• 8 dual-processor, 12-core (Intel E5-2650 v4) Dell PowerEdge R430 nodes, with 256 GB each, 10Gbit/s Ethernet-connected.
• 32 dual-processor, 10-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 64 GB each, Infiniband 4XFDR (56Gbit/s)-connected. (Reserved for jobs that use IB.)
• 8 dual-processor, 10-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 64 GB each, Infiniband 4XFDR (56Gbit/s)-connected. (Reserved for jobs that use IB.)
• 3 dual-processor, 10-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 256 GB each, Ethernet-connected.
• 22 dual-processor, 6-core (Intel E5-2630) IBM dx360m4 nodes, with 32GB each,
• Infiniband 4XFDR (56Gbit/s)-connected. (Reserved for jobs that use IB.)
• 130 dual-processor, 6-core (Intel X5650) IBM dx360m3 nodes, with 24GB each, Ethernet- connected.
• 2 dual-processor, 12-core (Intel E5-2650 v4) Dell R730, with 1TB.
• 1 dual-processor, 8-core (Intel E7-8837) IBM x3690 x5, with 512GB.
• 2 dual-processor, 12-core (Intel E5-2650 v4) Dell R730 GPU nodes, each with 2 NVidia Tesla P100 GPUs. (Each GPU has 3584 CUDA cores and 16GB RAM.)
• 2 dual-processor, 6-core (Intel X5650) GPU nodes, each with 2 NVidia Tesla M209

GPUs (each GPU has 512 CUDA cores and 5GB RAM):

• 2 user nodes (2 x Dell R430s, each with 2X 12-core Intel E5-2650 and 128GB RAM.)
• 2 I/O nodes (Dell R430s, 10G ethernet connected) along with:
• 2 I/O nodes (IBM x3655s, 10G ethernet connected) connected to:
  • 1 IBM DS4800 providing 260 terabytes of raw storage to GPFS (roughly 197TB usable).
  • 1 IBM DS4700 providing 104 terabytes of raw storage (roughly 76TB usable).
  • 1 IBM DCS3850 providing 240 terabytes of raw storage to GPFS (roughly 164TB usable).
  • 1 Dell MD3460 providing 357.5 terabytes of raw storage to GPFS (roughly 260.5TB usable), and 43 terabytes of solid-state disk to GPFS (for fast random-access data and metadata, roughly 27.5 TB usable.)
  • 1 IBM V3700 providing 10 terabytes of solid-state disk to GPFS (for fast random- access data and metadata.)
• 2 Flash-storage GPFS Metadata nodes (IBM x3655s, 10G ethernet connected)

Software

• Operating System: RedHat Enterprise Linux 7 (64-bit) with the GNU compilers (gcc, f77)
• Resources Manager: TORQUE v6.1 (TORQUE is an extension of PBS)
• Workload Scheduler: Moab v9
• Package Manager: Spack v0.11

Hardware

DeepGreen is a new massively parallel cluster deployed in Summer 2019 with 80 GPUs capable of over 8 petaflops of mixed-precision calculations based on the NVIDIA Tesla V100 architecture. Its hybrid design can expedite high-throughput artificial intelligence and machine learning workflows, and its extreme parallelism will forge new and transformative research pipelines.

• 10 GPU nodes (Penguin Relion XE4118GTS) each with:
  • 2 Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz (2x 16 cores, 22M cache)
  • 768GB RAM (256GB for GPFS pagepool)
  • 8 NVIDIA Tesla V100s with 32GB RAM
  • 4 2-lane HDR (100Gb/s, so 400Gb/s/node) Infiniband links to QM8700 switch
  • 2 NVMe nodes, each with 64TB NVMe devices (8x8TB), replicated to provide 64TB /gpfs3 filesystem
• Mellanox QM8700 switch running at HDR speeds

Software

• Operating System: RedHat Enterprise Linux 7 (64-bit) with the GNU compilers (gcc, f77)
• Resources Manager: Slurm v19
• Package Manager: Spack v0.11