ParaInfer-X Performance
Machine Specifications
CPU Model | CPU Core Info | Memory | IB Card | OS | GPU |
AMD EPYC 7763 | 2x64 @ 2.5Ghz | 1000 GB | Mellonox HDR (200 Gbps) | Centos Linux 7.9 | NVIDIA A100-SXM4-80GB(4/Node) |
![compare in posisson](/static/images/hidl/performance/ParaInfer-X/compare_in_poisson.png)
![mem shuffle](/static/images/hidl/performance/ParaInfer-X/mem_shuffle.png)
![progress](/static/images/hidl/performance/ParaInfer-X/progress.png)
![result](/static/images/hidl/performance/ParaInfer-X/result.png)
![Tensor RT vs Flover 32 requests](/static/images/hidl/performance/ParaInfer-X/TensorRT-LLM_vs._Flover_32_requests.png)
![Tensor RT vs Flover 64 requests](/static/images/hidl/performance/ParaInfer-X/TensorRT-LLM_vs._Flover_64_requests.png)
![Tensor RT Flover mem usage](/static/images/hidl/performance/ParaInfer-X/TensorRT-LLM_vs._Flover_Mem._Usage.png)
CPU Model | CPU Core Info | Memory | IB Card | OS | GPU |
AMD EPYC 7763 | 2x64 @ 2.5Ghz | 1000 GB | Mellonox HDR (200 Gbps) | Centos Linux 7.9 | NVIDIA A100-SXM4-80GB(4/Node) |