ParaInfer-X Performance

Machine Specifications

CPU Model CPU Core Info Memory IB Card OS GPU
AMD EPYC 7763 2x64 @ 2.5Ghz 1000 GB Mellonox HDR (200 Gbps) Centos Linux 7.9 NVIDIA A100-SXM4-80GB(4/Node)
compare in posisson
mem shuffle
progress
result
Tensor RT vs Flover 32 requests
Tensor RT vs Flover 64 requests
Tensor RT Flover mem usage