[OMPI users] Is building with "--enable-mpi-thread-multiple" recommended?

[Mark Dixon]

Hi,

We have some users who would like to try out openmpi MPI_THREAD_MULTIPLE
support on our InfiniBand cluster. I am wondering if we should enable it
on our production cluster-wide version, or install it as a separate "here
be dragons" copy.

I seem to recall openmpi folk cautioning that MPI_THREAD_MULTIPLE support
was pretty crazy and that enabling it could have problems for
non-MPI_THREAD_MULTIPLE codes (never mind codes that explicitly used it),
so such an install shouldn't be used unless for codes that actually need
it.

Is that still the case, please?

Thanks,

Mark

[r***@open-mpi.org]

Depends on the version, but if you are using something in the v2.x range, you should be okay with just one installed version

Hi,
We have some users who would like to try out openmpi MPI_THREAD_MULTIPLE support on our InfiniBand cluster. I am wondering if we should enable it on our production cluster-wide version, or install it as a separate "here be dragons" copy.
I seem to recall openmpi folk cautioning that MPI_THREAD_MULTIPLE support was pretty crazy and that enabling it could have problems for non-MPI_THREAD_MULTIPLE codes (never mind codes that explicitly used it), so such an install shouldn't be used unless for codes that actually need it.
Is that still the case, please?
Thanks,
Mark
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Mark Dixon]

Post by r***@open-mpi.org
Depends on the version, but if you are using something in the v2.x
range, you should be okay with just one installed version

Thanks Ralph.

How good is MPI_THREAD_MULTIPLE support these days and how far up the
wishlist is it, please?

We don't get many openmpi-specific queries from users but, other than core
binding, it seems to be the thing we get asked about the most (I normally
point those people at mvapich2 or intelmpi instead).

Cheers,

Mark

[r***@open-mpi.org]

We have been making a concerted effort to resolve outstanding issues as the interest in threaded applications has grown. It should be pretty good now, but we do see occasional bug reports, so it isn’t perfect.

Post by Mark Dixon

Post by r***@open-mpi.org
Depends on the version, but if you are using something in the v2.x range, you should be okay with just one installed version

Thanks Ralph.
How good is MPI_THREAD_MULTIPLE support these days and how far up the wishlist is it, please?
We don't get many openmpi-specific queries from users but, other than core binding, it seems to be the thing we get asked about the most (I normally point those people at mvapich2 or intelmpi instead).
Cheers,
Mark
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Michel Lesoinne]

I am also a proponent of the multiple thread support. For many reasons:
- code simplification
- easier support of computation/communication overlap with fewer
synchronization points
- possibility of creating exception aware MPI Code (I think the MPI
standard cruelly lacks constructs for a natural clean handling of
application exceptions across processes)

So it is good to hear there is progress.

Post by r***@open-mpi.org
We have been making a concerted effort to resolve outstanding issues as
the interest in threaded applications has grown. It should be pretty good
now, but we do see occasional bug reports, so it isn’t perfect.

Post by Mark Dixon

Post by r***@open-mpi.org
Depends on the version, but if you are using something in the v2.x

range, you should be okay with just one installed version

Post by Mark Dixon
Thanks Ralph.
How good is MPI_THREAD_MULTIPLE support these days and how far up the

wishlist is it, please?

Post by Mark Dixon
We don't get many openmpi-specific queries from users but, other than

core binding, it seems to be the thing we get asked about the most (I
normally point those people at mvapich2 or intelmpi instead).

Post by Mark Dixon
Cheers,
Mark
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[r***@open-mpi.org]

FWIW: have you taken a look at the event notification mechanisms in PMIx yet? The intent there, among other features, is to provide async notification of events generated either by the system (e.g., node failures and/or congestion) or other application processes.

https://pmix.github.io/master <https://pmix.github.io/master>

OMPI includes PMIx support beginning with OMPI v2.0, and various RMs are releasing their integrated support as well.
Ralph

Post by Michel Lesoinne
- code simplification
- easier support of computation/communication overlap with fewer synchronization points
- possibility of creating exception aware MPI Code (I think the MPI standard cruelly lacks constructs for a natural clean handling of application exceptions across processes)
So it is good to hear there is progress.
We have been making a concerted effort to resolve outstanding issues as the interest in threaded applications has grown. It should be pretty good now, but we do see occasional bug reports, so it isn’t perfect.

Post by Mark Dixon

Post by r***@open-mpi.org
Depends on the version, but if you are using something in the v2.x range, you should be okay with just one installed version

Thanks Ralph.
How good is MPI_THREAD_MULTIPLE support these days and how far up the wishlist is it, please?
We don't get many openmpi-specific queries from users but, other than core binding, it seems to be the thing we get asked about the most (I normally point those people at mvapich2 or intelmpi instead).
Cheers,
Mark
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Paul Kapinos]

Hi Mark,

Post by Mark Dixon

Post by r***@open-mpi.org
Depends on the version, but if you are using something in the v2.x range, you
should be okay with just one installed version

How good is MPI_THREAD_MULTIPLE support these days and how far up the wishlist
is it, please?

Note that on 1.10.x series (even on 1.10.6), enabling of MPI_THREAD_MULTIPLE in
lead to (silent) shutdown of the InfiniBand fabric for that application => SLOW!

2.x versions (tested: 2.0.1) handle MPI_THREAD_MULTIPLE on InfiniBand the right
way up, however due to absence of memory hooks (= nut aligned memory allocation)
we get 20% less bandwidth on IB with 2.x versions compared to 1.10.x versions of
Open MPI (regardless with or without support of MPI_THREAD_MULTIPLE).

On Intel OmniPath network both above issues seem to be not present, but due to a
performance bug in MPI_Free_mem your application can be horribly slow (seen:
CP2K) if the InfiniBand failback of OPA not disabled manually, see
https://www.mail-archive.com/***@lists.open-mpi.org//msg30593.html

Best,

Paul Kapinos

[Mark Dixon]

On Fri, 3 Mar 2017, Paul Kapinos wrote:
...

Post by Paul Kapinos
Note that on 1.10.x series (even on 1.10.6), enabling of
MPI_THREAD_MULTIPLE in lead to (silent) shutdown of the InfiniBand
fabric for that application => SLOW!
2.x versions (tested: 2.0.1) handle MPI_THREAD_MULTIPLE on InfiniBand
the right way up, however due to absence of memory hooks (= nut aligned
memory allocation) we get 20% less bandwidth on IB with 2.x versions
compared to 1.10.x versions of Open MPI (regardless with or without
support of MPI_THREAD_MULTIPLE).
On Intel OmniPath network both above issues seem to be not present, but
due to a performance bug in MPI_Free_mem your application can be
horribly slow (seen: CP2K) if the InfiniBand failback of OPA not
disabled manually, see

...

Hi Paul,

All very useful - thanks :)

Our (limited) testing seems to show no difference on 2.x with
MPI_THREAD_MULTIPLE enabled vs. disabled as well, which is good news. Glad
to hear another opinion.

Your 20% memory bandwidth performance hit on 2.x and the OPA problem are
concerning - will look at that. Are there tickets open for them?

Cheers,

Mark

[Paul Kapinos]

This post might be inappropriate. Click to display it.