Stanfield, Clyde
2018-07-06 14:08:55 UTC
We are using MPI_Ialltoallv for an image processing algorithm. When doing this we pass in an MPI_Type_contiguous with an MPI_Datatype of MPI_C_FLOAT_COMPLEX which ends up being the size of multiple rows of the image (based on the number of nodes used for distribution). In addition sendcounts, sdispls, resvcounts, and rdispls all fit within a signed int. Usually this works without any issues, but when we lower our number of nodes we sometimes see failures.
What we found is that even though we can fit everything into signed ints, line 528 of nbc_internal.h ends up calling a malloc with an int that appears to be the size of the (num_distributed_rows * num_columns * sizeof(std::complex<float>)) which in very large cases wraps back to negative. As a result we end up seeing "Error in malloc()" (line 530 of nbc_internal.h) throughout our output.
We can get around this issue by ensuring the sum of our contiguous type never exceeds 2GB. However, this was unexpected to us as our understanding was that all long as we can fit all the parts into signed ints we should be able to transfer more than 2GB at a time. Is it intended that MPI_Ialltoallv requires your underlying data to be less than 2GB or is this in error in how malloc is being called (should be called with a size_t instead of an int)?
Thanks,
Clyde Stanfield
[Loading Image...
]<http://www.radiantsolutions.com/>
Clyde Stanfield
Software Engineer
734-480-5100 office<tel://734-480-5100>
***@mdaus.com<mailto:***@mdaus.com>
[Loading Image...
]<https://twitter.com/radiant_maxar> [Loading Image...
] <https://www.linkedin.com/company/radiant-solutions/>
The information contained in this communication is confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please re-send this communication to the sender and delete the original message or any copy of it from your computer system.
What we found is that even though we can fit everything into signed ints, line 528 of nbc_internal.h ends up calling a malloc with an int that appears to be the size of the (num_distributed_rows * num_columns * sizeof(std::complex<float>)) which in very large cases wraps back to negative. As a result we end up seeing "Error in malloc()" (line 530 of nbc_internal.h) throughout our output.
We can get around this issue by ensuring the sum of our contiguous type never exceeds 2GB. However, this was unexpected to us as our understanding was that all long as we can fit all the parts into signed ints we should be able to transfer more than 2GB at a time. Is it intended that MPI_Ialltoallv requires your underlying data to be less than 2GB or is this in error in how malloc is being called (should be called with a size_t instead of an int)?
Thanks,
Clyde Stanfield
[Loading Image...
]<http://www.radiantsolutions.com/>Clyde Stanfield
Software Engineer
734-480-5100 office<tel://734-480-5100>
***@mdaus.com<mailto:***@mdaus.com>
[Loading Image...
]<https://twitter.com/radiant_maxar> [Loading Image...
] <https://www.linkedin.com/company/radiant-solutions/>The information contained in this communication is confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please re-send this communication to the sender and delete the original message or any copy of it from your computer system.