BlackDiamond is a high-performance computing cluster made possible by a gift from microchip manufacturer AMD. This cluster is built using AMD's 2nd Gen AMD EPYC processor, which pushes the boundaries for x86 performance, efficiency, security features, and overall system throughput.

Hardware

6 GPU nodes, each with:

• 1 AMD EPYC 7642 48-core processor
• 8 AMD Radeon Instinct MI50 Accelerators (32GB)
• 512GB DDR4-3200MHz RAM
• NVME via RDMA storage

Software

• Operating System: RedHat Enterprise Linux 7.9 (64-bit) with the GNU compilers (gcc, f77)
• Resources Manager: Slurm v20.11.8
• Package Manager: Spack v0.11
• ROCm 4.3

Bluemoon is a 161 node, 8392 core, high-performance computing cluster modeled after national supercomputing centers. This cluster supports large-scale computation, low-latency networking for MPI workloads, large memory systems, and high-performance parallel filesystems.

Hardware

• 39 dual-processor, 128-core AMD Epyc 7763 PowerEdge R6525 nodes, with 1TB RAM each. Mixed use: Infiniband connected 100Gb/s (2X HDR) for file access as well as MPI communication along with 10/25Gb Ethernet
• 1 dual-processor, 64-core EPYC 7543 PowerEdge R7525 node, with 4TB RAM. Infiniband connected 100 Gb/s (2X HDR) for file access, 10/25Gb Ethernet
• 32 dual-processor, 12-core (Intel E5-2650 v4) Dell PowerEdge R430 nodes, with 64GB RAM each, 10Gb Ethernet-connected
• 8 dual-processor, 12-core (Intel E5-2650 v4) Dell PowerEdge R430 nodes, with 256GB RAM each, 10Gb Ethernet-connected
• 9 dual-processor, 20 core (Intel 6230), PowerEdge R440, with 10GB RAM, 10Gb Ethernet-connected
• 3 dual-processor, 10-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 256GB RAM each, Ethernet-connected
• Infiniband: 8 dual-processor, 160-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 64GB RAM each, Infiniband 4XFDR (56Gb)-connected
• Infiniband: 32 dual-processor, 640-core (Intel E5-2650 v3) Dell PowerEdge R630 nodes, with 64GB RAM each, Infiniband 4XFDR (56Gb)-connected
• Infiniband: 22 dual-processor, 252-core (Intel E5-2630) IBM dx360m4 nodes, with 32GB RAM each, Infiniband 4XFDR (56Gb)-connected
• 2 dual-processor, 12-core (Intel E5-2650 v4) Dell R730, with 1TB RAM
• 1 dual-processor, 8-core (Intel E7-8837) IBM x3690 x5, with 512GB RAM
• 2 dual-processor, 12-core (Intel E5-2650 v4) Dell R730 GPU nodes, each with 2 Nvidia Tesla P100 GPUs. (Each GPU has 3584 CUDA cores and 16GB RAM)
• 2 I/O nodes (Dell R430s, 10Gb Ethernet-connected) along with 2 I/O nodes (IBM x3655s, 10Gb Ethernet-connected) connected to:
  • 1 Dell MD3460 providing 260TB storage to GPFS, and 27TB of solid-state disk to GPFS (for fast random-access data and metadata)
  • 1 Dell ME4084 providing 751TB of spinning disk storage
  • 1 IBMFS7200 providing 187TB of NVMe-attached FlashCore Module storage

Software

• Operating System: RedHat Enterprise Linux 7.9 (64-bit) with the GNU compilers (gcc, f77)
• Resources Manager: Slurm v20.11.8
• Package Manager: Spack v0.11

DeepGreen is a massively parallel cluster deployed in Summer 2019 with 80 GPUs capable of over 8 petaflops of mixed-precision calculations based on the NVIDIA Tesla V100 architecture. Its hybrid design can expedite high-throughput artificial intelligence and machine learning workflows, and its extreme parallelism will forge new and transformative research pipelines.