Andre Kessler
2016-10-06 23:44:19 UTC
Hi folks,
I've been running into some issues using the MPI_IN_PLACE option with MPI_Ireduce, in what seems to be a regression from OpenMPI v1.10.1. In particular, I encounter data corruption if non-root ranks supply the same pointers for 'send' and 'recv', even though this should have no bearing on the results since they do not expect to receive anything. In particular, take this scenario:
Root rank receiving results of reduction, say rank = 0, does MPI_Ireduce( MPI_IN_PLACE, data, N, mpiType<T>(), MPI_MIN, sink, MPI_COMM_WORLD, &req );
Ranks sharing data for reduction, say ranks 1, 2, 3, do MPI_Ireduce( data, recv, N, mpiType<T>(), MPI_MIN, sink, MPI_COMM_WORLD, &req );
As I understand it from the documentation, the value of the 'recv' pointer on ranks sharing data with the root rank should not be relevant. However, when 'recv' is set equal to the 'data' pointer on non-root ranks, I get non-deterministic garbage values as the result of the reduction.
This problem goes away if you supply 'nullptr' as the recv pointer on non-root ranks; it also goes away if you replace MPI_Ireduce with MPI_Reduce.
Source code to reproduce is here:
https://gist.github.com/akessler/3e911102892f3d6442feeb254f74665f
mpic++ --std=c++11 -Wall -Wextra -oreducebug reducebug.cpp
mpiexec -np 3 ./reducebug
RESULT:
<snip>
Case with T = int32_t
Expect: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
Actual: 0, 0, -649746440, 0, -649746440, 0, 0, 0, 0, 0, [FAILURE] : Data mismatch at sink <another run> Case with T = uint64_t
Expect: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
Actual: 1, 1, 0, 1, 1, 1, 1, 1, 1, 0,
[FAILURE] : Data mismatch at sink
You may need to run this a couple times to get it to fail, although it does so about 95% of the time.
Furthermore, with just two ranks, I get a consistent immediate segfault which seems to suggest a memory copy is being done between these pointers:
*** Process received signal ***
Signal: Segmentation fault (11)
Signal code: Address not mapped (1)
Failing at address: 0x90
[ 0] /usr/lib64/libpthread.so.0(+0xf100)[0x2ac11c395100]
[ 1] ./reduce(__intel_ssse3_rep_memcpy+0x23fe)[0x41022e]
[ 2] <snip>/openmpi/intel/2015.6.233/2.0.1/lib/libopen-pal.so.20(opal_cuda_memcpy+0x7d)[0x2ac11ce4cb0d]
[ 3] <snip>/openmpi/intel/2015.6.233/2.0.1/lib/libopen-pal.so.20(opal_convertor_pack+0x166)[0x2ac11ce443a6]
[ 4] <snip>/openmpi/intel/2015.6.233/2.0.1/lib/libmpi.so.20(PMPI_Pack+0x17d)[0x2ac11b8bf85d]
[ 5] <snip>/openmpi/intel/2015.6.233/2.0.1/lib/openmpi/mca_coll_libnbc.so(ompi_coll_libnbc_ireduce+0x74e)[0x2ac12f093a3e]
[ 6] <snip>/openmpi/intel/2015.6.233/2.0.1/lib/libmpi.so.20(PMPI_Ireduce+0x98)[0x2ac11b8c1e18]
[ 7] ./reducebug[0x40270d]
[ 8] /usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x2ac11c5c3b15]
[ 9] ./reducebug[0x402509]
*** End of error message ***
System environment:
# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
# icc -V
Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.6.233 Build 20151119
Open MPI: 2.0.1
Open MPI repo revision: v2.0.0-257-gee86e07 Open MPI release date: Sep 02, 2016 Open RTE: 2.0.1
Any insight here would be appreciated!
Thanks,
Andre Kessler
Software Engineer
Space Exploration Technologies
I've been running into some issues using the MPI_IN_PLACE option with MPI_Ireduce, in what seems to be a regression from OpenMPI v1.10.1. In particular, I encounter data corruption if non-root ranks supply the same pointers for 'send' and 'recv', even though this should have no bearing on the results since they do not expect to receive anything. In particular, take this scenario:
Root rank receiving results of reduction, say rank = 0, does MPI_Ireduce( MPI_IN_PLACE, data, N, mpiType<T>(), MPI_MIN, sink, MPI_COMM_WORLD, &req );
Ranks sharing data for reduction, say ranks 1, 2, 3, do MPI_Ireduce( data, recv, N, mpiType<T>(), MPI_MIN, sink, MPI_COMM_WORLD, &req );
As I understand it from the documentation, the value of the 'recv' pointer on ranks sharing data with the root rank should not be relevant. However, when 'recv' is set equal to the 'data' pointer on non-root ranks, I get non-deterministic garbage values as the result of the reduction.
This problem goes away if you supply 'nullptr' as the recv pointer on non-root ranks; it also goes away if you replace MPI_Ireduce with MPI_Reduce.
Source code to reproduce is here:
https://gist.github.com/akessler/3e911102892f3d6442feeb254f74665f
mpic++ --std=c++11 -Wall -Wextra -oreducebug reducebug.cpp
mpiexec -np 3 ./reducebug
RESULT:
<snip>
Case with T = int32_t
Expect: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
Actual: 0, 0, -649746440, 0, -649746440, 0, 0, 0, 0, 0, [FAILURE] : Data mismatch at sink <another run> Case with T = uint64_t
Expect: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
Actual: 1, 1, 0, 1, 1, 1, 1, 1, 1, 0,
[FAILURE] : Data mismatch at sink
You may need to run this a couple times to get it to fail, although it does so about 95% of the time.
Furthermore, with just two ranks, I get a consistent immediate segfault which seems to suggest a memory copy is being done between these pointers:
*** Process received signal ***
Signal: Segmentation fault (11)
Signal code: Address not mapped (1)
Failing at address: 0x90
[ 0] /usr/lib64/libpthread.so.0(+0xf100)[0x2ac11c395100]
[ 1] ./reduce(__intel_ssse3_rep_memcpy+0x23fe)[0x41022e]
[ 2] <snip>/openmpi/intel/2015.6.233/2.0.1/lib/libopen-pal.so.20(opal_cuda_memcpy+0x7d)[0x2ac11ce4cb0d]
[ 3] <snip>/openmpi/intel/2015.6.233/2.0.1/lib/libopen-pal.so.20(opal_convertor_pack+0x166)[0x2ac11ce443a6]
[ 4] <snip>/openmpi/intel/2015.6.233/2.0.1/lib/libmpi.so.20(PMPI_Pack+0x17d)[0x2ac11b8bf85d]
[ 5] <snip>/openmpi/intel/2015.6.233/2.0.1/lib/openmpi/mca_coll_libnbc.so(ompi_coll_libnbc_ireduce+0x74e)[0x2ac12f093a3e]
[ 6] <snip>/openmpi/intel/2015.6.233/2.0.1/lib/libmpi.so.20(PMPI_Ireduce+0x98)[0x2ac11b8c1e18]
[ 7] ./reducebug[0x40270d]
[ 8] /usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x2ac11c5c3b15]
[ 9] ./reducebug[0x402509]
*** End of error message ***
System environment:
# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
# icc -V
Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.6.233 Build 20151119
Open MPI: 2.0.1
Open MPI repo revision: v2.0.0-257-gee86e07 Open MPI release date: Sep 02, 2016 Open RTE: 2.0.1
Any insight here would be appreciated!
Thanks,
Andre Kessler
Software Engineer
Space Exploration Technologies