Discussion:
[OMPI users] ERR_TRUNCATE with MPI_Pack
Florian Lindner
2018-02-14 10:19:00 UTC
Permalink
Hello,

I have this example code:

#include <vector>
#include <mpi.h>

int main(int argc, char *argv[])
{
MPI_Init(&argc, &argv);
{
MPI_Request req1, req2;
std::vector<int> vec = {1, 2, 3};
int packSize = sizeof(int) * vec.size();
int position = 0;
std::vector<char> packSendBuf(packSize);
int vecSize = vec.size();
MPI_Pack(vec.data(), vec.size(), MPI_INT, packSendBuf.data(), packSize, &position, MPI_COMM_WORLD);

int estimatedPackSize = 0;
MPI_Pack_size(vec.size(), MPI_INT, MPI_COMM_WORLD, &estimatedPackSize);
std::cout << "packSize = " << packSize << ", estimatedPackSize = " << estimatedPackSize << std::endl;

MPI_Isend(&vecSize, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &req1);
MPI_Isend(packSendBuf.data(), position, MPI_PACKED, 0, 0, MPI_COMM_WORLD, &req2);
}
{
int vecSize, msgSize;
int packSize = 0, position = 0;

MPI_Recv(&vecSize, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

MPI_Status status;
MPI_Probe(0, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
MPI_Get_count(&status, MPI_PACKED, &packSize);
char packBuffer[packSize];
std::cout << "packSize from get_count = " << packSize << std::endl;

std::vector<int> vec(vecSize);
MPI_Recv(packBuffer, packSize, MPI_PACKED, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Unpack(packBuffer, vecSize, &position, vec.data(), vecSize, MPI_INT, MPI_COMM_WORLD);
}
MPI_Finalize();
}


Which gives an MPI_ERR_TRUNCATE even when running on 1 rank only. Background is that I want to send multiple differently
sized objects, also with more complex types that to not map to MPI_*, for which I plan to use MPI_BYTES. I plan to pack
them into one stream and unpack them one after one.

I suspect I got somthig with the sizes wrong. The lines

int estimatedPackSize = 0;
MPI_Pack_size(vec.size(), MPI_INT, MPI_COMM_WORLD, &estimatedPackSize);
std::cout << "packSize = " << packSize << ", estimatedPackSize = " << estimatedPackSize << std::endl;

Return the same number, that is 12, the packSize from get_cont is also 12.

Could you give a hint, what the is problem is here?

OpenMPI 3.0.0 @ Arch or OpenMPI 1.1.0.2 @ Ubuntu 16.04

Thanks,
Florian
Gilles Gouaillardet
2018-02-14 10:46:08 UTC
Permalink
Florian,

You send position=0 MPI_PACKED instead of estimatedPackSize, so it is very odd you see get_count = 12

Can you please double check that part ?

Also, who returns MPI_ERR_TRUNCATE ? MPI_Recv ? MPI_Unpack ?


Cheers,

Gilles
Post by Florian Lindner
Hello,
#include <vector>
#include <mpi.h>
int main(int argc, char *argv[])
{
MPI_Init(&argc, &argv);
{
MPI_Request req1, req2;
std::vector<int> vec = {1, 2, 3};
int packSize = sizeof(int) * vec.size();
int position = 0;
std::vector<char> packSendBuf(packSize);
int vecSize = vec.size();
MPI_Pack(vec.data(), vec.size(), MPI_INT, packSendBuf.data(), packSize, &position, MPI_COMM_WORLD);
int estimatedPackSize = 0;
MPI_Pack_size(vec.size(), MPI_INT, MPI_COMM_WORLD, &estimatedPackSize);
std::cout << "packSize = " << packSize << ", estimatedPackSize = " << estimatedPackSize << std::endl;
MPI_Isend(&vecSize, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &req1);
MPI_Isend(packSendBuf.data(), position, MPI_PACKED, 0, 0, MPI_COMM_WORLD, &req2);
}
{
int vecSize, msgSize;
int packSize = 0, position = 0;
MPI_Recv(&vecSize, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Status status;
MPI_Probe(0, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
MPI_Get_count(&status, MPI_PACKED, &packSize);
char packBuffer[packSize];
std::cout << "packSize from get_count = " << packSize << std::endl;
std::vector<int> vec(vecSize);
MPI_Recv(packBuffer, packSize, MPI_PACKED, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Unpack(packBuffer, vecSize, &position, vec.data(), vecSize, MPI_INT, MPI_COMM_WORLD);
}
MPI_Finalize();
}
Which gives an MPI_ERR_TRUNCATE even when running on 1 rank only. Background is that I want to send multiple differently
sized objects, also with more complex types that to not map to MPI_*, for which I plan to use MPI_BYTES. I plan to pack
them into one stream and unpack them one after one.
I suspect I got somthig with the sizes wrong. The lines
int estimatedPackSize = 0;
MPI_Pack_size(vec.size(), MPI_INT, MPI_COMM_WORLD, &estimatedPackSize);
std::cout << "packSize = " << packSize << ", estimatedPackSize = " << estimatedPackSize << std::endl;
Return the same number, that is 12, the packSize from get_cont is also 12.
Could you give a hint, what the is problem is here?
Thanks,
Florian
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Florian Lindner
2018-02-14 13:10:45 UTC
Permalink
Hi Gilles,
Post by Gilles Gouaillardet
Florian,
You send position=0 MPI_PACKED instead of estimatedPackSize, so it is very odd you see get_count = 12
Can you please double check that part ?
https://gist.github.com/floli/310980790d5d76caac0b19a937e2a502

You mean in line 22:

MPI_Isend(packSendBuf.data(), position, MPI_PACKED, 0, 0, MPI_COMM_WORLD, &req2);

but position was incremented (to 12, see output below) by the preceding MPI_Pack call.
Post by Gilles Gouaillardet
Also, who returns MPI_ERR_TRUNCATE ? MPI_Recv ? MPI_Unpack ?
Sorry, forgot to include that crucial part:

% mpirun -n 1 ./a.out
packSize = 12, estimatedPackSize = 12
position after pack = 12
packSize from get_count = 12
[asaru:30337] *** An error occurred in MPI_Unpack
[asaru:30337] *** reported by process [4237492225,0]
[asaru:30337] *** on communicator MPI_COMM_WORLD
[asaru:30337] *** MPI_ERR_TRUNCATE: message truncated
[asaru:30337] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[asaru:30337] *** and potentially your MPI job)

Best Thanks,
Florian
Post by Gilles Gouaillardet
Cheers,
Gilles
Post by Florian Lindner
Hello,
#include <vector>
#include <mpi.h>
int main(int argc, char *argv[])
{
MPI_Init(&argc, &argv);
{
MPI_Request req1, req2;
std::vector<int> vec = {1, 2, 3};
int packSize = sizeof(int) * vec.size();
int position = 0;
std::vector<char> packSendBuf(packSize);
int vecSize = vec.size();
MPI_Pack(vec.data(), vec.size(), MPI_INT, packSendBuf.data(), packSize, &position, MPI_COMM_WORLD);
int estimatedPackSize = 0;
MPI_Pack_size(vec.size(), MPI_INT, MPI_COMM_WORLD, &estimatedPackSize);
std::cout << "packSize = " << packSize << ", estimatedPackSize = " << estimatedPackSize << std::endl;
MPI_Isend(&vecSize, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &req1);
MPI_Isend(packSendBuf.data(), position, MPI_PACKED, 0, 0, MPI_COMM_WORLD, &req2);
}
{
int vecSize, msgSize;
int packSize = 0, position = 0;
MPI_Recv(&vecSize, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Status status;
MPI_Probe(0, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
MPI_Get_count(&status, MPI_PACKED, &packSize);
char packBuffer[packSize];
std::cout << "packSize from get_count = " << packSize << std::endl;
std::vector<int> vec(vecSize);
MPI_Recv(packBuffer, packSize, MPI_PACKED, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Unpack(packBuffer, vecSize, &position, vec.data(), vecSize, MPI_INT, MPI_COMM_WORLD);
}
MPI_Finalize();
}
Which gives an MPI_ERR_TRUNCATE even when running on 1 rank only. Background is that I want to send multiple differently
sized objects, also with more complex types that to not map to MPI_*, for which I plan to use MPI_BYTES. I plan to pack
them into one stream and unpack them one after one.
I suspect I got somthig with the sizes wrong. The lines
int estimatedPackSize = 0;
MPI_Pack_size(vec.size(), MPI_INT, MPI_COMM_WORLD, &estimatedPackSize);
std::cout << "packSize = " << packSize << ", estimatedPackSize = " << estimatedPackSize << std::endl;
Return the same number, that is 12, the packSize from get_cont is also 12.
Could you give a hint, what the is problem is here?
Thanks,
Florian
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Gilles Gouaillardet
2018-02-14 13:26:47 UTC
Permalink
Florian,

My bad, I overlooked that.

Per the man page, the second parameter of MPI_Unpack() is the input buffer size in *bytes*.
In your case, it should be packSize instead of vecSize.

Makes sense ?

FWIW, I never hesitate to run small tests with mpich (or its derivative).
If both implementations fail, the odds are the problem is in the code and not in the library.


Gilles
Post by Florian Lindner
Hi Gilles,
Post by Gilles Gouaillardet
Florian,
You send position=0 MPI_PACKED instead of estimatedPackSize, so it is very odd you see get_count = 12
Can you please double check that part ?
https://gist.github.com/floli/310980790d5d76caac0b19a937e2a502
MPI_Isend(packSendBuf.data(), position, MPI_PACKED, 0, 0, MPI_COMM_WORLD, &req2);
but position was incremented (to 12, see output below) by the preceding MPI_Pack call.
Post by Gilles Gouaillardet
Also, who returns MPI_ERR_TRUNCATE ? MPI_Recv ? MPI_Unpack ?
% mpirun -n 1 ./a.out
packSize = 12, estimatedPackSize = 12
position after pack = 12
packSize from get_count = 12
[asaru:30337] *** An error occurred in MPI_Unpack
[asaru:30337] *** reported by process [4237492225,0]
[asaru:30337] *** on communicator MPI_COMM_WORLD
[asaru:30337] *** MPI_ERR_TRUNCATE: message truncated
[asaru:30337] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[asaru:30337] *** and potentially your MPI job)
Best Thanks,
Florian
Post by Gilles Gouaillardet
Cheers,
Gilles
Post by Florian Lindner
Hello,
#include <vector>
#include <mpi.h>
int main(int argc, char *argv[])
{
MPI_Init(&argc, &argv);
{
MPI_Request req1, req2;
std::vector<int> vec = {1, 2, 3};
int packSize = sizeof(int) * vec.size();
int position = 0;
std::vector<char> packSendBuf(packSize);
int vecSize = vec.size();
MPI_Pack(vec.data(), vec.size(), MPI_INT, packSendBuf.data(), packSize, &position, MPI_COMM_WORLD);
int estimatedPackSize = 0;
MPI_Pack_size(vec.size(), MPI_INT, MPI_COMM_WORLD, &estimatedPackSize);
std::cout << "packSize = " << packSize << ", estimatedPackSize = " << estimatedPackSize << std::endl;
MPI_Isend(&vecSize, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &req1);
MPI_Isend(packSendBuf.data(), position, MPI_PACKED, 0, 0, MPI_COMM_WORLD, &req2);
}
{
int vecSize, msgSize;
int packSize = 0, position = 0;
MPI_Recv(&vecSize, 1, MPI_INT, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Status status;
MPI_Probe(0, MPI_ANY_TAG, MPI_COMM_WORLD, &status);
MPI_Get_count(&status, MPI_PACKED, &packSize);
char packBuffer[packSize];
std::cout << "packSize from get_count = " << packSize << std::endl;
std::vector<int> vec(vecSize);
MPI_Recv(packBuffer, packSize, MPI_PACKED, 0, MPI_ANY_TAG, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Unpack(packBuffer, vecSize, &position, vec.data(), vecSize, MPI_INT, MPI_COMM_WORLD);
}
MPI_Finalize();
}
Which gives an MPI_ERR_TRUNCATE even when running on 1 rank only. Background is that I want to send multiple differently
sized objects, also with more complex types that to not map to MPI_*, for which I plan to use MPI_BYTES. I plan to pack
them into one stream and unpack them one after one.
I suspect I got somthig with the sizes wrong. The lines
int estimatedPackSize = 0;
MPI_Pack_size(vec.size(), MPI_INT, MPI_COMM_WORLD, &estimatedPackSize);
std::cout << "packSize = " << packSize << ", estimatedPackSize = " << estimatedPackSize << std::endl;
Return the same number, that is 12, the packSize from get_cont is also 12.
Could you give a hint, what the is problem is here?
Thanks,
Florian
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Thomas Jahns
2018-02-15 09:18:00 UTC
Permalink
Post by Florian Lindner
Hello,
#include <vector>
#include <mpi.h>
int main(int argc, char *argv[])
{
MPI_Init(&argc, &argv);
{
MPI_Request req1, req2;
std::vector<int> vec = {1, 2, 3};
int packSize = sizeof(int) * vec.size();
Why don't you use MPI_Pack_size for sizing of packSendBuf here? The way you
write it is not guaranteed to work.
Post by Florian Lindner
int position = 0;
std::vector<char> packSendBuf(packSize);
int vecSize = vec.size();
MPI_Pack(vec.data(), vec.size(), MPI_INT, packSendBuf.data(), packSize, &position, MPI_COMM_WORLD);
int estimatedPackSize = 0;
MPI_Pack_size(vec.size(), MPI_INT, MPI_COMM_WORLD, &estimatedPackSize);
std::cout << "packSize = " << packSize << ", estimatedPackSize = " << estimatedPackSize << std::endl;
MPI_Isend(&vecSize, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &req1);
MPI_Isend(packSendBuf.data(), position, MPI_PACKED, 0, 0, MPI_COMM_WORLD, &req2);
}
At the above you asynchronously send data which then goes out of scope. That's
pretty much guaranteed to not work, have you tried to run your program with
valgrind?

[...]
Post by Florian Lindner
Which gives an MPI_ERR_TRUNCATE even when running on 1 rank only. Background is that I want to send multiple differently
Which routine gives MPI_ERR_TRUNCATE?
Post by Florian Lindner
sized objects, also with more complex types that to not map to MPI_*, for which I plan to use MPI_BYTES. I plan to pack
them into one stream and unpack them one after one.
Which elemental type does not map to an MPI datatype? You are aware that for all
derived types except pointer components there are ways to build corresponding
data types?
Post by Florian Lindner
I suspect I got somthig with the sizes wrong. The lines
int estimatedPackSize = 0;
MPI_Pack_size(vec.size(), MPI_INT, MPI_COMM_WORLD, &estimatedPackSize);
std::cout << "packSize = " << packSize << ", estimatedPackSize = " << estimatedPackSize << std::endl;
Return the same number, that is 12, the packSize from get_cont is also 12.
But see above: a pack size must be computed with MPI_Pack_size.

Regards, Thomas

Loading...