PyTorch 2.0 Performance on GPUs with MVAPICH-Plus

Machine Specifications: TACC Vista

CPU Model CPU Core Info Memory GPU Model GPU Memory IB Card
NVIDIA Grace CPU 1x72@3.1 GHz 116 GB DDR5 NVIDIA H200 GPU (1/Node) 96 GB HBM 3 Mellanox NDR (400 Gb/s)
Model Batch Size Block Size Benchmark Dataset DL Framework
GPT-2 12 1024 NanoGPT OpenWebText PyTorch 2.6.0
NanoGPT results on TACC Vista

Machine Specifications: OLCF Frontier

CPU Model CPU Core Info Memory GPU Model GPU Memory IB Card
AMD EPYC 7A53 CPU 1x64@2GHz 512 GB DDR4 AMD MI250X (4/Node) 128 GB HBM 2e HPE Slingshot (200 Gb/s)
Model Batch Size Block Size Benchmark Dataset DL Framework
GPT-2 12 1024 NanoGPT OpenWebText PyTorch 2.6.0
NanoGPT results on OLCF Frontier (1)
NanoGPT results on OLCF Frontier (2)