[OMPI users] Concerning the performance of the one-sided communications

[Ahmed Hamdy Mohamed Eleliemy]

Hi all,

I used to work with MPICH. I am trying now to compare and switch to OpenMPI. I am using OpenMPI version 2.1.1.
To have progress for one-sided communications (in the passive mode). I used to export the following environment variable MPICH_ASYNC_PROGRESS=1
My questions are:
1. What would be the equivalent option for openmpi?
2. How openmpi handle the progress of the one-sided communications?

Thanks in advance

Best regards
Ahmed

[Nathan Hjelm]

It depends on the transport used. If there is a high-performance network (Cray Aries, Infiniband, etc) then the progress is handled by the hardware. For other networks (Infinipath, Omnipath, TCP, etc) there are options.

For TCP you can set:

 --mca btl_tcp_progress_thread 1

No such option currently exists for Infinipath and Omnipath. Though with those networks you can use verbs but Intel does not support that mode of operation and the performance can be bad for some operations (> 100 us latency on compare-and-swap for example).

I don't know how well tested this support is. There haven't been many requests for this feature so not much work has been done on it.

-Nathan

On Feb 16, 2018, at 08:46 AM, Ahmed Hamdy Mohamed Eleliemy <***@unibas.ch> wrote:

Hi all,

I used to work with MPICH. I am trying now to compare and switch to OpenMPI. I am using OpenMPI version 2.1.1.
To have progress for one-sided communications (in the passive mode). I used to export the following environment variable MPICH_ASYNC_PROGRESS=1
My questions are:
1. What would be the equivalent option for openmpi?
2. How openmpi handle the progress of the one-sided communications?

Thanks in advance

Best regards
Ahmed

[Jeff Hammond]

Post by Nathan Hjelm
It depends on the transport used. If there is a high-performance network
(Cray Aries, Infiniband, etc) then the progress is handled by the hardware.
For other networks (Infinipath, Omnipath, TCP, etc) there are options.
--mca btl_tcp_progress_thread 1
No such option currently exists for Infinipath and Omnipath. Though with
those networks you can use verbs but Intel does not support that mode of
operation and the performance can be bad for some operations (> 100 us
latency on compare-and-swap for example).

If Intel Omni Path is the network of interest, I'd recommend using the
OFI/libfabric transport layer in Open-MPI and asking the libfabric list (
http://lists.openfabrics.org/mailman/listinfo/libfabric-users) for
instructions on how to enable asynchronous progress inside of libfabric.

When I was testing RMA in Open-MPI on Omni Path a while ago, the
performance of both PSM2 and OFI was quite good relative to other
implementations, as measured by the OSU MPI benchmarks. I was using Casper
(http://www.mcs.anl.gov/project/casper/) for asynchronous progress,
although those tests don't necessary reveal a lack of asynchronous progress.

Best,

Jeff

I don't know how well tested this support is. There haven't been many

Post by Nathan Hjelm
requests for this feature so not much work has been done on it.
-Nathan
On Feb 16, 2018, at 08:46 AM, Ahmed Hamdy Mohamed Eleliemy <
Hi all,
I used to work with MPICH. I am trying now to compare and switch to
OpenMPI. I am using OpenMPI version 2.1.1.
To have progress for one-sided communications (in the passive mode). I
used to export the following environment variable MPICH_ASYNC_PROGRESS=1
1. What would be the equivalent option for openmpi?
2. How openmpi handle the progress of the one-sided communications?
Thanks in advance
Best regards
Ahmed
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

[Nathan Hjelm]

How was the latency? That is the best metric to use because osc/pt2pt does put aggregation. Makes the result of osu_put_bw relatively garbage.

Post by Nathan Hjelm
It depends on the transport used. If there is a high-performance network (Cray Aries, Infiniband, etc) then the progress is handled by the hardware. For other networks (Infinipath, Omnipath, TCP, etc) there are options.
--mca btl_tcp_progress_thread 1
No such option currently exists for Infinipath and Omnipath. Though with those networks you can use verbs but Intel does not support that mode of operation and the performance can be bad for some operations (> 100 us latency on compare-and-swap for example).

If Intel Omni Path is the network of interest, I'd recommend using the OFI/libfabric transport layer in Open-MPI and asking the libfabric list (http://lists.openfabrics.org/mailman/listinfo/libfabric-users) for instructions on how to enable asynchronous progress inside of libfabric.
When I was testing RMA in Open-MPI on Omni Path a while ago, the performance of both PSM2 and OFI was quite good relative to other implementations, as measured by the OSU MPI benchmarks. I was using Casper (http://www.mcs.anl.gov/project/casper/) for asynchronous progress, although those tests don't necessary reveal a lack of asynchronous progress.
Best,
Jeff

Post by Nathan Hjelm
I don't know how well tested this support is. There haven't been many requests for this feature so not much work has been done on it.
-Nathan

Post by Ahmed Hamdy Mohamed Eleliemy
Hi all,
I used to work with MPICH. I am trying now to compare and switch to OpenMPI. I am using OpenMPI version 2.1.1.
To have progress for one-sided communications (in the passive mode). I used to export the following environment variable MPICH_ASYNC_PROGRESS=1
1. What would be the equivalent option for openmpi?
2. How openmpi handle the progress of the one-sided communications?
Thanks in advance
Best regards
Ahmed
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
Jeff Hammond
http://jeffhammond.github.io/
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users