The VACC includes three clusters: BlackDiamond, Bluemoon, and DeepGreen.

For a download of cluster specs, as well as logos and requirements for citing the VACC in publications, talks or proceedings, see Assets for Grants & Publications. 

BlackDiamond Specs

BlackDiamond is a high-performance computing cluster made possible by a gift from microchip manufacturer AMD. This cluster is built using AMD's 2nd Gen AMD EPYC processor, which pushes the boundaries for x86 performance, efficiency, security features, and overall system throughput.

Hardware

6 GPU nodes, each with:

  • 1 AMD EPYC 7642 48-core processor
  • 8 AMD Radeon Instinct MI50 Accelerators (32GB)
  • 512GB DDR4-3200MHz RAM
  • NVME via RDMA storage

Software

  • Operating System: RedHat Enterprise Linux 7.9 (64-bit) with the GNU compilers (gcc, f77)
  • Resources Manager: Slurm v20.11.8
  • Package Manager: Spack v0.11
  • ROCm 4.3

Bluemoon Specs

Bluemoon is a 300 node, 4004 core, high-performance computing cluster modeled after national supercomputing centers. This cluster supports large-scale computation, low-latency networking for MPI workloads, large memory systems, and high-performance parallel filesystems.

Hardware

  • 32 dual-processor, 12-core (Intel E5-2650 v4) Dell PowerEdge R430 nodes, with 64 GB each, 10Gbit/s Ethernet-connected
  • 8 dual-processor, 12-core (Intel E5-2650 v4) Dell PowerEdge R430 nodes, with 256 GB each, 10Gbit/s Ethernet-connected
  • 9 dual-processor, 20 core (Intel 6230), PowerEdge R440, with 10GB , 10Gbit/s Ethernet-connected
  • 3 dual-processor, 10-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 256 GB each, Ethernet-connected
  • 130 dual-processor, 6-core (Intel X5650) IBM dx360m3 nodes, with 24GB each, Ethernet- connected
  • Infiniband: 8 dual-processor, 160-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 64 GB each, Infiniband 4XFDR (56Gbit/s)-connected
  • Infiniband: 32 dual-processor, 640-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 64 GB each, Infiniband 4XFDR (56Gbit/s)-connected
  • Infiniband: 22 dual-processor, 252-core (Intel E5-2630) IBM dx360m4 nodes, with 32GB each, Infiniband 4XFDR (56Gbit/s)-connected
  • 2 dual-processor, 12-core (Intel E5-2650 v4) Dell R730, with 1TB
  • 1 dual-processor, 8-core (Intel E7-8837) IBM x3690 x5, with 512GB
  • 2 dual-processor, 12-core (Intel E5-2650 v4) Dell R730 GPU nodes, each with 2 NVidia Tesla P100 GPUs. (Each GPU has 3584 CUDA cores and 16GB RAM)
  • 2 I/O nodes (Dell R430s, 10G ethernet connected) along with 2 I/O nodes (IBM x3655s, 10G ethernet connected) connected to:
    • 1 IBM DS4800 providing 260 terabytes of raw storage to GPFS (roughly 197TB usable)
    • 1 IBM DS4700 providing 104 terabytes of raw storage (roughly 76TB usable)
    • 1 IBM DCS3850 providing 240 terabytes of raw storage to GPFS (roughly 164TB usable)
    • 1 Dell MD3460 providing 357.5 terabytes of raw storage to GPFS (roughly 260.5TB usable), and 43 terabytes of solid-state disk to GPFS (for fast random-access data and metadata, roughly 27.5 TB usable)
    • 1 IBM V3700 providing 10 terabytes of solid-state disk to GPFS (for fast random- access data and metadata)
  • 2 Flash-storage GPFS Metadata nodes (IBM x3655s, 10G Ethernet connected)

Software

  • Operating System: RedHat Enterprise Linux 7.9 (64-bit) with the GNU compilers (gcc, f77)
  • Resources Manager: Slurm v20.11.8
  • Package Manager: Spack v0.11

 

DeepGreen Specs

DeepGreen is a massively parallel cluster deployed in Summer 2019 with 80 GPUs capable of over 8 petaflops of mixed-precision calculations based on the NVIDIA Tesla V100 architecture. Its hybrid design can expedite high-throughput artificial intelligence and machine learning workflows, and its extreme parallelism will forge new and transformative research pipelines.

Hardware

  • 10 GPU nodes (Penguin Relion XE4118GTS) each with:
    • 2 Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz (2x 16 cores, 22M cache)
    • 768GB RAM (256GB for GPFS pagepool)
    • 8 NVIDIA Tesla V100s with 32GB RAM
    • 4 2-lane HDR (100Gb/s, so 400Gb/s/node) Infiniband links to QM8700 switch
    • 2 NVMe nodes, each with 64TB NVMe devices (8x8TB), replicated to provide 64TB /gpfs3 filesystem
  • Mellanox QM8700 switch running at HDR speeds

Software

  • Operating System: RedHat Enterprise Linux 7.9 (64-bit) with the GNU compilers (gcc, f77)
  • Resources Manager: Slurm v20.11.8
  • Package Manager: Spack v0.11
  • CUDA 11.4