BlackDiamond is a high-performance computing cluster made possible by a gift from microchip manufacturer AMD. This cluster is built using AMD's 2nd Gen AMD EPYC processor, which pushes the boundaries for x86 performance, efficiency, security features, and overall system throughput.

Hardware

6 GPU nodes, each with:

• 1 AMD EPYC 7642 48-core processor
• 8 AMD Radeon Instinct MI50 Accelerators (32GB)
• 512GB DDR4-3200MHz RAM
• NVME via RDMA storage

Software

• Operating System: RedHat Enterprise Linux 7.9 (64-bit) with the GNU compilers (gcc, f77)
• Resources Manager: Slurm v20.11.8
• Package Manager: Spack v0.11
• ROCm 4.3

Bluemoon is a 300 node, 4004 core, high-performance computing cluster modeled after national supercomputing centers. This cluster supports large-scale computation, low-latency networking for MPI workloads, large memory systems, and high-performance parallel filesystems.

Hardware

• 32 dual-processor, 12-core (Intel E5-2650 v4) Dell PowerEdge R430 nodes, with 64GB RAM each, 10Gb Ethernet-connected
• 8 dual-processor, 12-core (Intel E5-2650 v4) Dell PowerEdge R430 nodes, with 256GB RAM each, 10Gb Ethernet-connected
• 9 dual-processor, 20 core (Intel 6230), PowerEdge R440, with 10GB RAM, 10Gb Ethernet-connected
• 3 dual-processor, 10-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 256GB RAM each, Ethernet-connected
• 38 dual-processor, 128-core AMD Epyc 7763 PowerEdge R6525 nodes, with 1TB RAM each. Mixed use: Infiniband connected 100Gb/s (2X HDR) for file access as well as MPI communication along with 10/25Gb Ethernet
• 1 dual-processor, 64-core EPYC 7543 PowerEdge R7525 node, with 4TB RAM. Infiniband connected 100 Gb/s (2X HDR) for file access, 10/25Gb Ethernet
• Infiniband: 8 dual-processor, 160-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 64GB RAM each, Infiniband 4XFDR (56Gb)-connected
• Infiniband: 32 dual-processor, 640-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 64GB RAM each, Infiniband 4XFDR (56Gb)-connected
• Infiniband: 22 dual-processor, 252-core (Intel E5-2630) IBM dx360m4 nodes, with 32GB RAM each, Infiniband 4XFDR (56Gb)-connected
• 2 dual-processor, 12-core (Intel E5-2650 v4) Dell R730, with 1TB RAM
• 1 dual-processor, 8-core (Intel E7-8837) IBM x3690 x5, with 512GB RAM
• 2 dual-processor, 12-core (Intel E5-2650 v4) Dell R730 GPU nodes, each with 2 Nvidia Tesla P100 GPUs. (Each GPU has 3584 CUDA cores and 16GB RAM)
• 2 I/O nodes (Dell R430s, 10Gb Ethernet-connected) along with 2 I/O nodes (IBM x3655s, 10Gb Ethernet-connected) connected to:
  • 1 IBM DS4800 providing 260TB of raw storage to GPFS (roughly 197TB usable)
  • 1 IBM DS4700 providing 104TB of raw storage (roughly 76TB usable)
  • 1 IBM DCS3850 providing 240TB of raw storage to GPFS (roughly 164TB usable)
  • 1 Dell MD3460 providing 357.5TB of raw storage to GPFS (roughly 260.5TB usable), and 43TB of solid-state disk to GPFS (for fast random-access data and metadata, roughly 27.5TB usable)
  • 1 IBM V3700 providing 10TB of solid-state disk to GPFS (for fast random- access data and metadata)
• 2 Flash-storage GPFS Metadata nodes (IBM x3655s, 10Gb Ethernet-connected)

Software

• Operating System: RedHat Enterprise Linux 7.9 (64-bit) with the GNU compilers (gcc, f77)
• Resources Manager: Slurm v20.11.8
• Package Manager: Spack v0.11

DeepGreen is a massively parallel cluster deployed in Summer 2019 with 80 GPUs capable of over 8 petaflops of mixed-precision calculations based on the NVIDIA Tesla V100 architecture. Its hybrid design can expedite high-throughput artificial intelligence and machine learning workflows, and its extreme parallelism will forge new and transformative research pipelines.