[OMPI users] MPI_THREAD_MULTIPLE: Fatal error in MPI_Win_flush

[Joseph Schuchart]

All,

We are trying to combine MPI_Put and MPI_Win_flush on locked (using
MPI_Win_lock_all) dynamic windows to mimic a blocking put. The
application is (potentially) multi-threaded and we are thus relying on
MPI_THREAD_MULTIPLE support to be available.

When I try to use this combination (MPI_Put + MPI_Win_flush) in our
application, I am seeing threads occasionally hang in MPI_Win_flush,
probably waiting for some progress to happen. However, when I try to
create a small reproducer (attached, the original application has
multiple layers of abstraction), I am seeing fatal errors in
MPI_Win_flush if using more than one thread:

```
[beryl:18037] *** An error occurred in MPI_Win_flush
[beryl:18037] *** reported by process [4020043777,2]
[beryl:18037] *** on win pt2pt window 3
[beryl:18037] *** MPI_ERR_RMA_SYNC: error executing rma sync
[beryl:18037] *** MPI_ERRORS_ARE_FATAL (processes in this win will now
abort,
[beryl:18037] *** and potentially your MPI job)
```

I could only trigger this on dynamic windows with multiple concurrent
threads running.

So: Is this a valid MPI program (except for the missing clean-up at the
end ;))? It seems to run fine with MPICH but maybe they are more
tolerant to some programming errors...

If it is a valid MPI program, I assume there is some race condition in
MPI_Win_flush that leads to the fatal error (or the hang that I observe
otherwise)?

I tested this with OpenMPI 1.10.5 on single node Linux Mint 18.1 system
with stock kernel 4.8.0-36 (aka my laptop). OpenMPI and the test were
both compiled using GCC 5.3.0. I could not run it using OpenMPI 2.0.2
due to the fatal error in MPI_Win_create (which also applies to
MPI_Win_create_dynamic, see my other thread, not sure if they are related).

Please let me know if this is a valid use case and whether I can provide
you with additional information if required.

Many thanks in advance!

Cheers
Joseph

[Nathan Hjelm]

You can not perform synchronization at the same time as communication on the same target. This means if one thread is in MPI_Put/MPI_Get/MPI_Accumulate (target) you can’t have another thread in MPI_Win_flush (target) or MPI_Win_flush_all(). If your program is doing that it is not a valid MPI program. If you want to ensure a particular put operation is complete try MPI_Rput instead.

-Nathan

All,
We are trying to combine MPI_Put and MPI_Win_flush on locked (using MPI_Win_lock_all) dynamic windows to mimic a blocking put. The application is (potentially) multi-threaded and we are thus relying on MPI_THREAD_MULTIPLE support to be available.
```
[beryl:18037] *** An error occurred in MPI_Win_flush
[beryl:18037] *** reported by process [4020043777,2]
[beryl:18037] *** on win pt2pt window 3
[beryl:18037] *** MPI_ERR_RMA_SYNC: error executing rma sync
[beryl:18037] *** MPI_ERRORS_ARE_FATAL (processes in this win will now abort,
[beryl:18037] *** and potentially your MPI job)
```
I could only trigger this on dynamic windows with multiple concurrent threads running.
So: Is this a valid MPI program (except for the missing clean-up at the end ;))? It seems to run fine with MPICH but maybe they are more tolerant to some programming errors...
If it is a valid MPI program, I assume there is some race condition in MPI_Win_flush that leads to the fatal error (or the hang that I observe otherwise)?
I tested this with OpenMPI 1.10.5 on single node Linux Mint 18.1 system with stock kernel 4.8.0-36 (aka my laptop). OpenMPI and the test were both compiled using GCC 5.3.0. I could not run it using OpenMPI 2.0.2 due to the fatal error in MPI_Win_create (which also applies to MPI_Win_create_dynamic, see my other thread, not sure if they are related).
Please let me know if this is a valid use case and whether I can provide you with additional information if required.
Many thanks in advance!
Cheers
Joseph
--
Dipl.-Inf. Joseph Schuchart
High Performance Computing Center Stuttgart (HLRS)
Nobelstr. 19
D-70569 Stuttgart
Tel.: [phone]
Fax: [phone]
<ompi_flush_hang.c>_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Joseph Schuchart]

Nathan,

Thanks for your clarification. Just so that I understand where my
misunderstanding of this matter comes from: can you please point me to
the place in the standard that prohibits thread-concurrent window
synchronization using MPI_Win_flush[_all]? I can neither seem to find
such a passage in 11.5.4 (Flush and Sync), nor in 12.4 (MPI and
Threads). The latter explicitly excludes waiting on the same request
object (which it does not) and collective operations on the same
communicator (which MPI_Win_flush is not) but it fails to mention
one-sided non-collective sync operations. Any hint would be much
appreciated.

We will look at MPI_Rput and MPI_Rget. However, having a single put
paired with a flush is just the simplest case. We also want to support
multiple asynchronous operations that are eventually synchronized on a
per-thread basis where keeping the request handles might not be feasible.

Thanks,
Joseph

Post by Nathan Hjelm
You can not perform synchronization at the same time as communication on the same target. This means if one thread is in MPI_Put/MPI_Get/MPI_Accumulate (target) you can’t have another thread in MPI_Win_flush (target) or MPI_Win_flush_all(). If your program is doing that it is not a valid MPI program. If you want to ensure a particular put operation is complete try MPI_Rput instead.
-Nathan

All,
We are trying to combine MPI_Put and MPI_Win_flush on locked (using MPI_Win_lock_all) dynamic windows to mimic a blocking put. The application is (potentially) multi-threaded and we are thus relying on MPI_THREAD_MULTIPLE support to be available.
```
[beryl:18037] *** An error occurred in MPI_Win_flush
[beryl:18037] *** reported by process [4020043777,2]
[beryl:18037] *** on win pt2pt window 3
[beryl:18037] *** MPI_ERR_RMA_SYNC: error executing rma sync
[beryl:18037] *** MPI_ERRORS_ARE_FATAL (processes in this win will now abort,
[beryl:18037] *** and potentially your MPI job)
```
I could only trigger this on dynamic windows with multiple concurrent threads running.
So: Is this a valid MPI program (except for the missing clean-up at the end ;))? It seems to run fine with MPICH but maybe they are more tolerant to some programming errors...
If it is a valid MPI program, I assume there is some race condition in MPI_Win_flush that leads to the fatal error (or the hang that I observe otherwise)?
I tested this with OpenMPI 1.10.5 on single node Linux Mint 18.1 system with stock kernel 4.8.0-36 (aka my laptop). OpenMPI and the test were both compiled using GCC 5.3.0. I could not run it using OpenMPI 2.0.2 due to the fatal error in MPI_Win_create (which also applies to MPI_Win_create_dynamic, see my other thread, not sure if they are related).
Please let me know if this is a valid use case and whether I can provide you with additional information if required.
Many thanks in advance!
Cheers
Joseph
--
Dipl.-Inf. Joseph Schuchart
High Performance Computing Center Stuttgart (HLRS)
Nobelstr. 19
D-70569 Stuttgart
Tel.: [phone]
Fax: [phone]
<ompi_flush_hang.c>_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Joseph Schuchart]

Ping :) I would really appreciate any input on my question below. I
crawled through the standard but cannot seem to find the wording that
prohibits thread-concurrent access and synchronization.

Using MPI_Rget works in our case but MPI_Rput only guarantees local
completion, not remote completion. Specifically, a thread-parallel
application would have to go into some serial region just to issue an
MPI_Win_flush before a thread can read a value previously written to the
same target. Re-reading remote values in the same process/thread might
not be efficient but is a valid use-case for us.

Best regards,
Joseph

Post by Joseph Schuchart
Nathan,
Thanks for your clarification. Just so that I understand where my
misunderstanding of this matter comes from: can you please point me to
the place in the standard that prohibits thread-concurrent window
synchronization using MPI_Win_flush[_all]? I can neither seem to find
such a passage in 11.5.4 (Flush and Sync), nor in 12.4 (MPI and
Threads). The latter explicitly excludes waiting on the same request
object (which it does not) and collective operations on the same
communicator (which MPI_Win_flush is not) but it fails to mention
one-sided non-collective sync operations. Any hint would be much
appreciated.
We will look at MPI_Rput and MPI_Rget. However, having a single put
paired with a flush is just the simplest case. We also want to support
multiple asynchronous operations that are eventually synchronized on a
per-thread basis where keeping the request handles might not be feasible.
Thanks,
Joseph

Post by Nathan Hjelm
You can not perform synchronization at the same time as communication
on the same target. This means if one thread is in
MPI_Put/MPI_Get/MPI_Accumulate (target) you can’t have another thread
in MPI_Win_flush (target) or MPI_Win_flush_all(). If your program is
doing that it is not a valid MPI program. If you want to ensure a
particular put operation is complete try MPI_Rput instead.
-Nathan

Post by Joseph Schuchart
All,
We are trying to combine MPI_Put and MPI_Win_flush on locked (using
MPI_Win_lock_all) dynamic windows to mimic a blocking put. The
application is (potentially) multi-threaded and we are thus relying
on MPI_THREAD_MULTIPLE support to be available.
When I try to use this combination (MPI_Put + MPI_Win_flush) in our
application, I am seeing threads occasionally hang in MPI_Win_flush,
probably waiting for some progress to happen. However, when I try to
create a small reproducer (attached, the original application has
multiple layers of abstraction), I am seeing fatal errors in
```
[beryl:18037] *** An error occurred in MPI_Win_flush
[beryl:18037] *** reported by process [4020043777,2]
[beryl:18037] *** on win pt2pt window 3
[beryl:18037] *** MPI_ERR_RMA_SYNC: error executing rma sync
[beryl:18037] *** MPI_ERRORS_ARE_FATAL (processes in this win will now abort,
[beryl:18037] *** and potentially your MPI job)
```
I could only trigger this on dynamic windows with multiple
concurrent threads running.
So: Is this a valid MPI program (except for the missing clean-up at
the end ;))? It seems to run fine with MPICH but maybe they are more
tolerant to some programming errors...
If it is a valid MPI program, I assume there is some race condition
in MPI_Win_flush that leads to the fatal error (or the hang that I
observe otherwise)?
I tested this with OpenMPI 1.10.5 on single node Linux Mint 18.1
system with stock kernel 4.8.0-36 (aka my laptop). OpenMPI and the
test were both compiled using GCC 5.3.0. I could not run it using
OpenMPI 2.0.2 due to the fatal error in MPI_Win_create (which also
applies to MPI_Win_create_dynamic, see my other thread, not sure if
they are related).
Please let me know if this is a valid use case and whether I can
provide you with additional information if required.
Many thanks in advance!
Cheers
Joseph
--
Dipl.-Inf. Joseph Schuchart
High Performance Computing Center Stuttgart (HLRS)
Nobelstr. 19
D-70569 Stuttgart
Tel.: [phone]
Fax: [phone]
<ompi_flush_hang.c>_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Jeff Hammond]

This post might be inappropriate. Click to display it.

[Joseph Schuchart]

Jeff, Nathan,

Thank you for the positive feedback. I took a chance to look at current
master and tried (based on my humble understanding of the OpenMPI
internals) to remove the error check in ompi_osc_pt2pt_flush. Upon
testing with the example code I sent initially, I saw a Segfault that
stemmed from infinite recursion in ompi_osc_pt2pt_frag_alloc. I fixed it
locally (see attached patch) and now I am seeing a Segfault somewhere
below opal_progress():

```
Thread 5 "a.out" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffdcf08700 (LWP 23912)]
0x00007fffe77b48ad in mca_pml_ob1_recv_frag_callback_match () from
/home/joseph/opt/openmpi-master/lib/openmpi/mca_pml_ob1.so
(gdb) bt
#0 0x00007fffe77b48ad in mca_pml_ob1_recv_frag_callback_match () from
/home/joseph/opt/openmpi-master/lib/openmpi/mca_pml_ob1.so
#1 0x00007fffec985d0d in mca_btl_vader_component_progress () from
/home/joseph/opt/openmpi-master/lib/openmpi/mca_btl_vader.so
#2 0x00007ffff6d4220c in opal_progress () from
/home/joseph/opt/openmpi-master/lib/libopen-pal.so.0
#3 0x00007fffe6754c55 in ompi_osc_pt2pt_flush_lock () from
/home/joseph/opt/openmpi-master/lib/openmpi/mca_osc_pt2pt.so
#4 0x00007fffe67574df in ompi_osc_pt2pt_flush () from
/home/joseph/opt/openmpi-master/lib/openmpi/mca_osc_pt2pt.so
#5 0x00007ffff7b608bc in PMPI_Win_flush () from
/home/joseph/opt/openmpi-master/lib/libmpi.so.0
#6 0x0000000000401149 in put_blocking ()
#7 0x000000000040140f in main._omp_fn ()
#8 0x00007ffff78bfe46 in gomp_thread_start (xdata=<optimized out>) at
../../../src/libgomp/team.c:119
#9 0x00007ffff76936ba in start_thread (arg=0x7fffdcf08700) at
pthread_create.c:333
#10 0x00007ffff73c982d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109
```
I guess there is a race condition somewhere. Unfortunately, I cannot see
symbols in the standard build (which is probably to be expected). I
recompiled master with --enable-debug and now I am facing a Segfault in
mpirun:

```
Thread 1 "mpirun" received signal SIGSEGV, Segmentation fault.
0x00007fffd1e2590e in external_close () from
/home/joseph/opt/openmpi-master/lib/openmpi/mca_pmix_pmix3x.so
(gdb) bt
#0 0x00007fffd1e2590e in external_close () from
/home/joseph/opt/openmpi-master/lib/openmpi/mca_pmix_pmix3x.so
#1 0x00007ffff77f23f5 in mca_base_component_close
(component=0x7fffd20afde0 <mca_pmix_pmix3x_component>, output_id=-1) at
../../../../opal/mca/base/mca_base_components_close.c:53
#2 0x00007ffff77f24b5 in mca_base_components_close (output_id=-1,
components=0x7ffff7abe250 <opal_pmix_base_framework+80>,
skip=0x7fffd23c9dc0 <mca_pmix_pmix2x_component>)
at ../../../../opal/mca/base/mca_base_components_close.c:85
#3 0x00007ffff77f2867 in mca_base_select (type_name=0x7ffff789652c
"pmix", output_id=-1, components_available=0x7ffff7abe250
<opal_pmix_base_framework+80>, best_module=0x7fffffffd6a0,
best_component=0x7fffffffd698,
priority_out=0x0) at
../../../../opal/mca/base/mca_base_components_select.c:141
#4 0x00007ffff786d083 in opal_pmix_base_select () at
../../../../opal/mca/pmix/base/pmix_base_select.c:35
#5 0x00007ffff5339ca8 in rte_init () at
../../../../../orte/mca/ess/hnp/ess_hnp_module.c:640
#6 0x00007ffff7ae0b62 in orte_init (pargc=0x7fffffffd86c,
pargv=0x7fffffffd860, flags=4) at ../../orte/runtime/orte_init.c:243
#7 0x00007ffff7b2290b in orte_submit_init (argc=6, argv=0x7fffffffde58,
opts=0x0) at ../../orte/orted/orted_submit.c:535
#8 0x00000000004012d7 in orterun (argc=6, argv=0x7fffffffde58) at
../../../../orte/tools/orterun/orterun.c:133
#9 0x0000000000400fd6 in main (argc=6, argv=0x7fffffffde58) at
../../../../orte/tools/orterun/main.c:13
```

I'm giving up here for tonight. Please let me know if I can help with
anything else.


Joseph

Nathan and I discussed at the MPI Forum last week. I argued that your
usage is not erroneous, although certain pathological cases (likely
concocted) can lead to nasty behavior. He indicated that he would
remove the error check, but it may require further discussion/debate
with others.
You can remove the error check from the source and recompile if you
are in a hurry, or you can use an MPICH-derivative (I have not
checked, but I doubt MPICH errors on this code.).
Jeff
Ping :) I would really appreciate any input on my question below.
I crawled through the standard but cannot seem to find the wording
that prohibits thread-concurrent access and synchronization.
Using MPI_Rget works in our case but MPI_Rput only guarantees
local completion, not remote completion. Specifically, a
thread-parallel application would have to go into some serial
region just to issue an MPI_Win_flush before a thread can read a
value previously written to the same target. Re-reading remote
values in the same process/thread might not be efficient but is a
valid use-case for us.
Best regards,
Joseph
Nathan,
Thanks for your clarification. Just so that I understand where
my misunderstanding of this matter comes from: can you please
point me to the place in the standard that prohibits
thread-concurrent window synchronization using
MPI_Win_flush[_all]? I can neither seem to find such a passage
in 11.5.4 (Flush and Sync), nor in 12.4 (MPI and Threads). The
latter explicitly excludes waiting on the same request object
(which it does not) and collective operations on the same
communicator (which MPI_Win_flush is not) but it fails to
mention one-sided non-collective sync operations. Any hint
would be much appreciated.
We will look at MPI_Rput and MPI_Rget. However, having a
single put paired with a flush is just the simplest case. We
also want to support multiple asynchronous operations that are
eventually synchronized on a per-thread basis where keeping
the request handles might not be feasible.
Thanks,
Joseph
You can not perform synchronization at the same time as
communication on the same target. This means if one thread
is in MPI_Put/MPI_Get/MPI_Accumulate (target) you can’t
have another thread in MPI_Win_flush (target) or
MPI_Win_flush_all(). If your program is doing that it is
not a valid MPI program. If you want to ensure a
particular put operation is complete try MPI_Rput instead.
-Nathan
On Feb 19, 2017, at 2:34 PM, Joseph Schuchart
All,
We are trying to combine MPI_Put and MPI_Win_flush on
locked (using MPI_Win_lock_all) dynamic windows to
mimic a blocking put. The application is (potentially)
multi-threaded and we are thus relying on
MPI_THREAD_MULTIPLE support to be available.
When I try to use this combination (MPI_Put +
MPI_Win_flush) in our application, I am seeing threads
occasionally hang in MPI_Win_flush, probably waiting
for some progress to happen. However, when I try to
create a small reproducer (attached, the original
application has multiple layers of abstraction), I am
seeing fatal errors in MPI_Win_flush if using more
```
[beryl:18037] *** An error occurred in MPI_Win_flush
[beryl:18037] *** reported by process [4020043777,2]
[beryl:18037] *** on win pt2pt window 3
[beryl:18037] *** MPI_ERR_RMA_SYNC: error executing
rma sync
[beryl:18037] *** MPI_ERRORS_ARE_FATAL (processes in
this win will now abort,
[beryl:18037] *** and potentially your MPI job)
```
I could only trigger this on dynamic windows with
multiple concurrent threads running.
So: Is this a valid MPI program (except for the
missing clean-up at the end ;))? It seems to run fine
with MPICH but maybe they are more tolerant to some
programming errors...
If it is a valid MPI program, I assume there is some
race condition in MPI_Win_flush that leads to the
fatal error (or the hang that I observe otherwise)?
I tested this with OpenMPI 1.10.5 on single node Linux
Mint 18.1 system with stock kernel 4.8.0-36 (aka my
laptop). OpenMPI and the test were both compiled using
GCC 5.3.0. I could not run it using OpenMPI 2.0.2 due
to the fatal error in MPI_Win_create (which also
applies to MPI_Win_create_dynamic, see my other
thread, not sure if they are related).
Please let me know if this is a valid use case and
whether I can provide you with additional information
if required.
Many thanks in advance!
Cheers
Joseph
--
Dipl.-Inf. Joseph Schuchart
High Performance Computing Center Stuttgart (HLRS)
Nobelstr. 19
D-70569 Stuttgart
Tel.: [phone]
Fax: [phone]
<ompi_flush_hang.c>_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Dipl.-Inf. Joseph Schuchart
High Performance Computing Center Stuttgart (HLRS)
Nobelstr. 19
D-70569 Stuttgart
Tel.: [phone]
Fax: [phone]
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Jeff Hammond
http://jeffhammond.github.io/
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Jeff Hammond]

This is fine if each thread interacts with a different window, no?

Jeff

Post by Nathan Hjelm
You can not perform synchronization at the same time as communication on
the same target. This means if one thread is in
MPI_Put/MPI_Get/MPI_Accumulate (target) you can’t have another thread in
MPI_Win_flush (target) or MPI_Win_flush_all(). If your program is doing
that it is not a valid MPI program. If you want to ensure a particular put
operation is complete try MPI_Rput instead.
-Nathan

Post by Joseph Schuchart
All,
We are trying to combine MPI_Put and MPI_Win_flush on locked (using

MPI_Win_lock_all) dynamic windows to mimic a blocking put. The application
is (potentially) multi-threaded and we are thus relying on
MPI_THREAD_MULTIPLE support to be available.

Post by Joseph Schuchart
When I try to use this combination (MPI_Put + MPI_Win_flush) in our

application, I am seeing threads occasionally hang in MPI_Win_flush,
probably waiting for some progress to happen. However, when I try to create
a small reproducer (attached, the original application has multiple layers
of abstraction), I am seeing fatal errors in MPI_Win_flush if using more

Post by Joseph Schuchart
```
[beryl:18037] *** An error occurred in MPI_Win_flush
[beryl:18037] *** reported by process [4020043777,2]
[beryl:18037] *** on win pt2pt window 3
[beryl:18037] *** MPI_ERR_RMA_SYNC: error executing rma sync
[beryl:18037] *** MPI_ERRORS_ARE_FATAL (processes in this win will now

abort,

Post by Joseph Schuchart
[beryl:18037] *** and potentially your MPI job)
```
I could only trigger this on dynamic windows with multiple concurrent

threads running.

Post by Joseph Schuchart
So: Is this a valid MPI program (except for the missing clean-up at the

end ;))? It seems to run fine with MPICH but maybe they are more tolerant
to some programming errors...

Post by Joseph Schuchart
If it is a valid MPI program, I assume there is some race condition in

MPI_Win_flush that leads to the fatal error (or the hang that I observe
otherwise)?

Post by Joseph Schuchart
I tested this with OpenMPI 1.10.5 on single node Linux Mint 18.1 system

with stock kernel 4.8.0-36 (aka my laptop). OpenMPI and the test were both
compiled using GCC 5.3.0. I could not run it using OpenMPI 2.0.2 due to the
fatal error in MPI_Win_create (which also applies to
MPI_Win_create_dynamic, see my other thread, not sure if they are related).

Post by Joseph Schuchart
Please let me know if this is a valid use case and whether I can provide

you with additional information if required.

Post by Joseph Schuchart
Many thanks in advance!
Cheers
Joseph
--
Dipl.-Inf. Joseph Schuchart
High Performance Computing Center Stuttgart (HLRS)
Nobelstr. 19
D-70569 Stuttgart
Tel.: [phone]
Fax: [phone]
<ompi_flush_hang.c>_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users