Marshall2, John (SSC/SPC)
2018-04-20 17:03:16 UTC
Hi
I am trying to verify/determine what the proper setting is for btl_openib_ib_include.
Some background:
* openmpi 2.1.1 (and 1.6.5 - yes it is old)
* lxc containers
* SRIOV (virtual functions) being used
* dedicated IB interface (e.g., ib2) per container
Should the mlx4_X:1 correspond to a specific ibY interface? E.g., for ib26, I find
mlx4_13:1 by:
$ ls /sys/class/net/ib26/device/infiniband
mlx4_13
Does the mlx4_X have to be determined at each location where an mpi task
would run? I suppose it would because the ibY is likely to be different.
On some tests, I have found that the setting:
export OMPI_MCA_btl_openib_if_include=mlx4_0:1
provides better performance than not specifying a value or letting mpirun/orted
figure it out at runtime.
Thanks,
John
I am trying to verify/determine what the proper setting is for btl_openib_ib_include.
Some background:
* openmpi 2.1.1 (and 1.6.5 - yes it is old)
* lxc containers
* SRIOV (virtual functions) being used
* dedicated IB interface (e.g., ib2) per container
Should the mlx4_X:1 correspond to a specific ibY interface? E.g., for ib26, I find
mlx4_13:1 by:
$ ls /sys/class/net/ib26/device/infiniband
mlx4_13
Does the mlx4_X have to be determined at each location where an mpi task
would run? I suppose it would because the ibY is likely to be different.
On some tests, I have found that the setting:
export OMPI_MCA_btl_openib_if_include=mlx4_0:1
provides better performance than not specifying a value or letting mpirun/orted
figure it out at runtime.
Thanks,
John