MVAPICH-Plus 3.0 ================ Overview -------- MVAPICH-Plus 3.0 binary release is based on MVAPICH 3.0 and incorporates GPU accelerator support. Upcoming releases will enahance existing GPU support and add other advanced features. Supported Platfoms ------------------ CPU: 1. Intel and AMD x86 Systems 2. OpenPOWER 8 and 9 Systems System Requirements ------------------- MVAPICH-Plus 3.0 binary release requires the following software to be installed on your system: NVIDIA GPUs ~~~~~~~~~ 1. http://www.nvidia.com/Download/driverResults.aspx/69372/[NVIDIA Driver 367.48 or later] 2. https://developer.nvidia.com/cuda-toolkit[NVIDIA CUDA Toolkit 7.5 and later] List of Mellanox InfiniBand adapters and NVIDIA GPU devices which support GPUDirect RDMA can be found http://www.mellanox.com/page/products_dyn?product_family=116[here]. AMD GPUs ~~~~~~ 1. https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html[AMD ROCm installation (for AMD GPUs)] Installing MVAPICH-Plus library ------------------------------- To install the MVAPICH-Plus library you simply need to select the correct library (MOFED version, Compiler version, etc.) for your system and install the RPM using your favorite RPM tool. Please use the downloads page (http://mvapich.cse.ohio-state.edu/downloads/) to find the appropriate RPM link and follow the instructions below. [source,bash] .Install library using the downloaded RPM file ------------------------------------------------------------------------------- $ wget http://mvapich.cse.ohio-state.edu/download/mvapich/gdr/3.0/.rpm $ rpm -Uvh --nodeps .rpm ------------------------------------------------------------------------------- The RPMs contained in our libraries are relocatable and can be installed using a prefix other than the default of ./opt/mvapich/ used by the library in the previous example. [source,bash] .Install library specifying custom prefix -------------------------------------------------------------------------------- $ rpm --prefix /custom/install/prefix -Uvh --nodeps .rpm -------------------------------------------------------------------------------- If you do not have root permission you can use rpm2cpio to extract the library. [source,bash] .Use rpm2cpio to extract the library -------------------------------------------------------------------------------- $ rpm2cpio .rpm | cpio -id -------------------------------------------------------------------------------- When using the rpm2cpio method, you will need to update the MPI compiler scripts, such as mpicc, in order to point to the correct path of where you place the library. [TIP] If you are using a Debian based system such as Ubuntu you can convert the rpm to a deb using a tool such as alien or follow the rpm2cpio instructions above. Installing MVAPICH-Plus using Spack ----------------------------------- MVAPICH-Plus can be installed using Spack without building it from source. See the Spack userguide for details: https://mvapich.cse.ohio-state.edu/userguide/userguide_spack/ Running applications -------------------- Here are some examples running applications with the MVAPICH-Plus software. Example running OSU Micro Benchmark for NVIDIA GPUs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To run point-to-point and collective benchmarks for measuring internode latency between GPUs when enabling GPUDirect RDMA-based designs in MVAPICH-Plus 3.0 [source,bash,subs="+attributes"] -------------------------------------------------------------------------------- $ export MVP_PATH=/opt/mvapich/plus/3.0/gnu $ export MVP_ENABLE_GPU=1 $ $MVP_PATH/bin/mpirun_rsh -export -np 2 hostA hostB \ $MVP_PATH/libexec/osu-micro-benchmarks/get_local_rank \ $MVP_PATH/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency D D $ $MVP_PATH/bin/mpirun_rsh -export -np 2 hostA hostB \ $MVP_PATH/libexec/osu-micro-benchmarks/get_local_rank \ $MVP_PATH/libexec/osu-microbenchmarks/mpi/collective/osu_allreduce -d cuda -------------------------------------------------------------------------------- Example running OSU Micro Benchmark for AMD GPUs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To run point-to-point and collective tests running on AMD GPUs using ROCm-aware designs in MVAPICH-Plus 3.0 [source,bash,subs="+attributes"] -------------------------------------------------------------------------------- $ export MVP_PATH=/opt/mvapich/plus/3.0/gnu $ export MVP_ENABLE_GPU=1 $ $MVP_PATH/bin/mpirun_rsh -export -np 2 hostA hostB \ $MVP_PATH/libexec/osu-micro-benchmarks/get_local_rank \ $MVP_PATH/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency D D $ $MVP_PATH/bin/mpirun_rsh -export -np 2 hostA hostB \ $MVP_PATH/libexec/osu-micro-benchmarks/get_local_rank \ $MVP_PATH/libexec/osu-microbenchmarks/mpi/collective/osu_allreduce -d rocm