[OMPI users] difference between OPENMPI e Intel MPI (DATATYPE)

Discussion:

Diego Avesani

2015-09-02 12:39:03 UTC

Dear all,

I have notice small difference between OPEN-MPI and intel MPI.
For example in MPI_ALLREDUCE in intel MPI is not allowed to use the same
variable in send and receiving Buff.

I have written my code in OPEN-MPI, but unfortunately I have to run in on a
intel-MPI cluster.
Now I have the following error:

*atal error in MPI_Isend: Invalid communicator, error stack:*
*MPI_Isend(158): MPI_Isend(buf=0x1dd27b0, count=1, INVALID DATATYPE,
dest=0, tag=0, comm=0x0, request=0x7fff9d7dd9f0) failed*

This is ho I create my type:

* CALL MPI_TYPE_VECTOR(1, Ncoeff_MLS, Ncoeff_MLS, MPI_DOUBLE_PRECISION,
coltype, MPIdata%iErr) *
* CALL MPI_TYPE_COMMIT(coltype, MPIdata%iErr)*
* !*
* CALL MPI_TYPE_VECTOR(1, nVar, nVar, coltype, MPI_WENO_TYPE,
MPIdata%iErr) *
* CALL MPI_TYPE_COMMIT(MPI_WENO_TYPE, MPIdata%iErr)*

do you believe that is here the problem?
Is also this the way how intel MPI create a datatype?

maybe I could also ask to intel MPI users
What do you think?

Diego

Gilles Gouaillardet

2015-09-02 12:52:12 UTC

Permalink

Diego,

about MPI_Allreduce, you should use MPI_IN_PLACE if you want the same
buffer in send and recv

about the stack, I notice comm is NULL which is a bit surprising...
at first glance, type creation looks good.
that being said, you do not check MPIdata%iErr is MPI_SUCCESS after each
MPI call.
I recommend you first do this, so you can catch the error as soon it
happens, and hopefully understand why it occursïŒ

Cheers,

Gilles

Post by Diego Avesani
Dear all,
I have notice small difference between OPEN-MPI and intel MPI.
For example in MPI_ALLREDUCE in intel MPI is not allowed to use the same
variable in send and receiving Buff.
I have written my code in OPEN-MPI, but unfortunately I have to run in on
a intel-MPI cluster.
*atal error in MPI_Isend: Invalid communicator, error stack:*
*MPI_Isend(158): MPI_Isend(buf=0x1dd27b0, count=1, INVALID DATATYPE,
dest=0, tag=0, comm=0x0, request=0x7fff9d7dd9f0) failed*
* CALL MPI_TYPE_VECTOR(1, Ncoeff_MLS, Ncoeff_MLS, MPI_DOUBLE_PRECISION,
coltype, MPIdata%iErr) *
* CALL MPI_TYPE_COMMIT(coltype, MPIdata%iErr)*
* !*
* CALL MPI_TYPE_VECTOR(1, nVar, nVar, coltype, MPI_WENO_TYPE,
MPIdata%iErr) *
* CALL MPI_TYPE_COMMIT(MPI_WENO_TYPE, MPIdata%iErr)*
do you believe that is here the problem?
Is also this the way how intel MPI create a datatype?
maybe I could also ask to intel MPI users
What do you think?
Diego

Diego Avesani

2015-09-02 16:13:06 UTC

Permalink

Dear Gilles, Dear all,
I have found the error. Some CPU has no element to share. It was a my error.

Now I have another one:

*Fatal error in MPI_Isend: Invalid communicator, error stack:*
*MPI_Isend(158): MPI_Isend(buf=0x137b7b4, count=1, INVALID DATATYPE,
dest=0, tag=0, comm=0x0, request=0x7fffe8726fc0) failed*

In this case with MPI does not work, with openMPI it works.

Could you see some particular information from the error message?

Diego

Diego

On 2 September 2015 at 14:52, Gilles Gouaillardet <

Post by Gilles Gouaillardet
Diego,
about MPI_Allreduce, you should use MPI_IN_PLACE if you want the same
buffer in send and recv
about the stack, I notice comm is NULL which is a bit surprising...
at first glance, type creation looks good.
that being said, you do not check MPIdata%iErr is MPI_SUCCESS after each
MPI call.
I recommend you first do this, so you can catch the error as soon it
happens, and hopefully understand why it occursïŒ
Cheers,
Gilles

_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27523.php

Jeff Squyres (jsquyres)

2015-09-02 17:07:16 UTC

Permalink

Can you reproduce the error in a small example?

Also, try using "use mpi" instead of "include 'mpif.h'", and see if that turns up any errors.

Post by Diego Avesani
Dear Gilles, Dear all,
I have found the error. Some CPU has no element to share. It was a my error.
MPI_Isend(158): MPI_Isend(buf=0x137b7b4, count=1, INVALID DATATYPE, dest=0, tag=0, comm=0x0, request=0x7fffe8726fc0) failed
In this case with MPI does not work, with openMPI it works.
Could you see some particular information from the error message?
Diego
Diego
Diego,
about MPI_Allreduce, you should use MPI_IN_PLACE if you want the same buffer in send and recv
about the stack, I notice comm is NULL which is a bit surprising...
at first glance, type creation looks good.
that being said, you do not check MPIdata%iErr is MPI_SUCCESS after each MPI call.
I recommend you first do this, so you can catch the error as soon it happens, and hopefully understand why it occurs．
Cheers,
Gilles
Dear all,
I have notice small difference between OPEN-MPI and intel MPI.
For example in MPI_ALLREDUCE in intel MPI is not allowed to use the same variable in send and receiving Buff.
I have written my code in OPEN-MPI, but unfortunately I have to run in on a intel-MPI cluster.
MPI_Isend(158): MPI_Isend(buf=0x1dd27b0, count=1, INVALID DATATYPE, dest=0, tag=0, comm=0x0, request=0x7fff9d7dd9f0) failed
CALL MPI_TYPE_VECTOR(1, Ncoeff_MLS, Ncoeff_MLS, MPI_DOUBLE_PRECISION, coltype, MPIdata%iErr)
CALL MPI_TYPE_COMMIT(coltype, MPIdata%iErr)
!
CALL MPI_TYPE_VECTOR(1, nVar, nVar, coltype, MPI_WENO_TYPE, MPIdata%iErr)
CALL MPI_TYPE_COMMIT(MPI_WENO_TYPE, MPIdata%iErr)
do you believe that is here the problem?
Is also this the way how intel MPI create a datatype?
maybe I could also ask to intel MPI users
What do you think?
Diego
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: http://www.open-mpi.org/community/lists/users/2015/09/27523.php
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: http://www.open-mpi.org/community/lists/users/2015/09/27524.php

--
Jeff Squyres
***@cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

Diego Avesani

2015-09-03 14:43:49 UTC

Permalink

Dear Jeff, Dear all,
I normaly use "USE MPI"

This is the answar fro intel HPC forum:

*If you are switching between intel and openmpi you must remember not to
mix environment. You might use modules to manage this. As the data types
encodings differ, you must take care that all objects are built against the
same headers.*

Could someone explain me what are these modules and how I can use them?

Thanks

Diego

Diego

Post by Jeff Squyres (jsquyres)
Can you reproduce the error in a small example?
Also, try using "use mpi" instead of "include 'mpif.h'", and see if that
turns up any errors.

Post by Diego Avesani
Dear Gilles, Dear all,
I have found the error. Some CPU has no element to share. It was a my

error.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x137b7b4, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fffe8726fc0) failed

Post by Diego Avesani
In this case with MPI does not work, with openMPI it works.
Could you see some particular information from the error message?
Diego
Diego
On 2 September 2015 at 14:52, Gilles Gouaillardet <
Diego,
about MPI_Allreduce, you should use MPI_IN_PLACE if you want the same

buffer in send and recv

Post by Diego Avesani
about the stack, I notice comm is NULL which is a bit surprising...
at first glance, type creation looks good.
that being said, you do not check MPIdata%iErr is MPI_SUCCESS after each

MPI call.

Post by Diego Avesani
I recommend you first do this, so you can catch the error as soon it

happens, and hopefully understand why it occursïŒ

Post by Diego Avesani
Cheers,
Gilles
Dear all,
I have notice small difference between OPEN-MPI and intel MPI.
For example in MPI_ALLREDUCE in intel MPI is not allowed to use the same

variable in send and receiving Buff.

Post by Diego Avesani
I have written my code in OPEN-MPI, but unfortunately I have to run in

on a intel-MPI cluster.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x1dd27b0, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fff9d7dd9f0) failed

Post by Diego Avesani
CALL MPI_TYPE_VECTOR(1, Ncoeff_MLS, Ncoeff_MLS, MPI_DOUBLE_PRECISION,

coltype, MPIdata%iErr)

Post by Diego Avesani
CALL MPI_TYPE_COMMIT(coltype, MPIdata%iErr)
!
CALL MPI_TYPE_VECTOR(1, nVar, nVar, coltype, MPI_WENO_TYPE,

MPIdata%iErr)

Post by Diego Avesani
CALL MPI_TYPE_COMMIT(MPI_WENO_TYPE, MPIdata%iErr)
do you believe that is here the problem?
Is also this the way how intel MPI create a datatype?
maybe I could also ask to intel MPI users
What do you think?
Diego
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users

http://www.open-mpi.org/community/lists/users/2015/09/27523.php

Post by Diego Avesani
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users

http://www.open-mpi.org/community/lists/users/2015/09/27524.php
--
Jeff Squyres
http://www.cisco.com/web/about/doing_business/legal/cri/
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27525.php

Nick Papior

2015-09-03 14:52:23 UTC

Permalink

When you change environment, that is change between OpenMPI and Intel MPI,
or compiler, it is recommended that you recompile everything.

use mpi

is a module, you cannot mix these between compilers/environments, sadly the
Fortran specification does not enforce a strict module format which is why
this is necessary.

Post by Diego Avesani
Dear Jeff, Dear all,
I normaly use "USE MPI"
*If you are switching between intel and openmpi you must remember not to
mix environment. You might use modules to manage this. As the data types
encodings differ, you must take care that all objects are built against the
same headers.*
Could someone explain me what are these modules and how I can use them?
Thanks
Diego
Diego

Post by Jeff Squyres (jsquyres)
Can you reproduce the error in a small example?
Also, try using "use mpi" instead of "include 'mpif.h'", and see if that
turns up any errors.

Post by Diego Avesani
Dear Gilles, Dear all,
I have found the error. Some CPU has no element to share. It was a my

error.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x137b7b4, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fffe8726fc0) failed

buffer in send and recv

each MPI call.

Post by Diego Avesani
I recommend you first do this, so you can catch the error as soon it

happens, and hopefully understand why it occursïŒ

Post by Diego Avesani
Cheers,
Gilles
Dear all,
I have notice small difference between OPEN-MPI and intel MPI.
For example in MPI_ALLREDUCE in intel MPI is not allowed to use the

same variable in send and receiving Buff.

Post by Diego Avesani
I have written my code in OPEN-MPI, but unfortunately I have to run in

on a intel-MPI cluster.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x1dd27b0, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fff9d7dd9f0) failed

Post by Diego Avesani
CALL MPI_TYPE_VECTOR(1, Ncoeff_MLS, Ncoeff_MLS,

MPI_DOUBLE_PRECISION, coltype, MPIdata%iErr)

Post by Diego Avesani
CALL MPI_TYPE_COMMIT(coltype, MPIdata%iErr)
!
CALL MPI_TYPE_VECTOR(1, nVar, nVar, coltype, MPI_WENO_TYPE,

MPIdata%iErr)

http://www.open-mpi.org/community/lists/users/2015/09/27523.php

Post by Diego Avesani
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27527.php

--
Kind regards Nick

Diego Avesani

2015-09-03 14:59:07 UTC

Permalink

Dear Nick, Dear all,

I use mpi.

I recompile everything, every time.

I do not understand what I shall do.

Thanks again

Diego

Diego

Post by Nick Papior
When you change environment, that is change between OpenMPI and Intel MPI,
or compiler, it is recommended that you recompile everything.
use mpi
is a module, you cannot mix these between compilers/environments, sadly
the Fortran specification does not enforce a strict module format which is
why this is necessary.

Post by Jeff Squyres (jsquyres)
Can you reproduce the error in a small example?
Also, try using "use mpi" instead of "include 'mpif.h'", and see if that
turns up any errors.

Post by Diego Avesani
Dear Gilles, Dear all,
I have found the error. Some CPU has no element to share. It was a my

error.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x137b7b4, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fffe8726fc0) failed

buffer in send and recv

each MPI call.

Post by Diego Avesani
I recommend you first do this, so you can catch the error as soon it

happens, and hopefully understand why it occursïŒ

Post by Diego Avesani
Cheers,
Gilles
On Wednesday, September 2, 2015, Diego Avesani <
Dear all,
I have notice small difference between OPEN-MPI and intel MPI.
For example in MPI_ALLREDUCE in intel MPI is not allowed to use the

same variable in send and receiving Buff.

Post by Diego Avesani
I have written my code in OPEN-MPI, but unfortunately I have to run in

on a intel-MPI cluster.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x1dd27b0, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fff9d7dd9f0) failed

Post by Diego Avesani
CALL MPI_TYPE_VECTOR(1, Ncoeff_MLS, Ncoeff_MLS,

MPI_DOUBLE_PRECISION, coltype, MPIdata%iErr)

Post by Diego Avesani
CALL MPI_TYPE_COMMIT(coltype, MPIdata%iErr)
!
CALL MPI_TYPE_VECTOR(1, nVar, nVar, coltype, MPI_WENO_TYPE,

MPIdata%iErr)

http://www.open-mpi.org/community/lists/users/2015/09/27523.php

Post by Diego Avesani
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27527.php

--
Kind regards Nick
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27528.php

Nick Papior

2015-09-03 15:03:27 UTC

Permalink

You still havent shown us anything about what goes wrong, you just give us
the error statement and assume it is because of ill-defined type-creation,
it might as well be because you call allreduce erroneously.
Please give us more information...

Post by Diego Avesani
Dear Nick, Dear all,
I use mpi.
I recompile everything, every time.
I do not understand what I shall do.
Thanks again
Diego
Diego

Post by Nick Papior
When you change environment, that is change between OpenMPI and Intel
MPI, or compiler, it is recommended that you recompile everything.
use mpi
is a module, you cannot mix these between compilers/environments, sadly
the Fortran specification does not enforce a strict module format which is
why this is necessary.

Post by Jeff Squyres (jsquyres)
Can you reproduce the error in a small example?
Also, try using "use mpi" instead of "include 'mpif.h'", and see if
that turns up any errors.

Post by Diego Avesani
Dear Gilles, Dear all,
I have found the error. Some CPU has no element to share. It was a my

error.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x137b7b4, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fffe8726fc0) failed

buffer in send and recv

each MPI call.

Post by Diego Avesani
I recommend you first do this, so you can catch the error as soon it

happens, and hopefully understand why it occursïŒ

same variable in send and receiving Buff.

Post by Diego Avesani
I have written my code in OPEN-MPI, but unfortunately I have to run

in on a intel-MPI cluster.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x1dd27b0, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fff9d7dd9f0) failed

Post by Diego Avesani
CALL MPI_TYPE_VECTOR(1, Ncoeff_MLS, Ncoeff_MLS,

MPI_DOUBLE_PRECISION, coltype, MPIdata%iErr)

Post by Diego Avesani
CALL MPI_TYPE_COMMIT(coltype, MPIdata%iErr)
!
CALL MPI_TYPE_VECTOR(1, nVar, nVar, coltype, MPI_WENO_TYPE,

MPIdata%iErr)

http://www.open-mpi.org/community/lists/users/2015/09/27523.php

Post by Diego Avesani
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27527.php

_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27531.php

--
Kind regards Nick

Diego Avesani

2015-09-03 15:08:49 UTC

Permalink

Dear all, Dear Nick,
you are right.

now will try to erase every time all *.mod file and *.o file. After thar
recompile all *.f90 files.

If I get another error I will tell you also the message.

Thanks again
Diego

Diego

Post by Nick Papior
You still havent shown us anything about what goes wrong, you just give us
the error statement and assume it is because of ill-defined type-creation,
it might as well be because you call allreduce erroneously.
Please give us more information...

Post by Diego Avesani
Dear Nick, Dear all,
I use mpi.
I recompile everything, every time.
I do not understand what I shall do.
Thanks again
Diego
Diego

Post by Nick Papior
When you change environment, that is change between OpenMPI and Intel
MPI, or compiler, it is recommended that you recompile everything.
use mpi
is a module, you cannot mix these between compilers/environments, sadly
the Fortran specification does not enforce a strict module format which is
why this is necessary.

Post by Diego Avesani
Dear Jeff, Dear all,
I normaly use "USE MPI"
*If you are switching between intel and openmpi you must remember not
to mix environment. You might use modules to manage this. As the data
types encodings differ, you must take care that all objects are built
against the same headers.*
Could someone explain me what are these modules and how I can use them?
Thanks
Diego
Diego
On 2 September 2015 at 19:07, Jeff Squyres (jsquyres) <

Post by Jeff Squyres (jsquyres)
Can you reproduce the error in a small example?
Also, try using "use mpi" instead of "include 'mpif.h'", and see if
that turns up any errors.

Post by Diego Avesani
Dear Gilles, Dear all,
I have found the error. Some CPU has no element to share. It was a

my error.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x137b7b4, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fffe8726fc0) failed

same buffer in send and recv

each MPI call.

Post by Diego Avesani
I recommend you first do this, so you can catch the error as soon it

happens, and hopefully understand why it occursïŒ

same variable in send and receiving Buff.

Post by Diego Avesani
I have written my code in OPEN-MPI, but unfortunately I have to run

in on a intel-MPI cluster.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x1dd27b0, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fff9d7dd9f0) failed

Post by Diego Avesani
CALL MPI_TYPE_VECTOR(1, Ncoeff_MLS, Ncoeff_MLS,

MPI_DOUBLE_PRECISION, coltype, MPIdata%iErr)

Post by Diego Avesani
CALL MPI_TYPE_COMMIT(coltype, MPIdata%iErr)
!
CALL MPI_TYPE_VECTOR(1, nVar, nVar, coltype, MPI_WENO_TYPE,

MPIdata%iErr)

http://www.open-mpi.org/community/lists/users/2015/09/27523.php

Post by Diego Avesani
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27527.php

_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27531.php

Gilles Gouaillardet

2015-09-03 15:05:09 UTC

Permalink

Diego,

did you update your code to check all MPI calls are successful ?
(e.g. test ierr is MPI_SUCCES after each MPI call)

can you write a short program that reproduce the same issue ?
if not, is your program and input data public ally available ?

Cheers,

Gilles

Post by Diego Avesani
Dear Nick, Dear all,
I use mpi.
I recompile everything, every time.
I do not understand what I shall do.
Thanks again
Diego
Diego

Post by Nick Papior
When you change environment, that is change between OpenMPI and Intel
MPI, or compiler, it is recommended that you recompile everything.
use mpi
is a module, you cannot mix these between compilers/environments, sadly
the Fortran specification does not enforce a strict module format which is
why this is necessary.

Post by Jeff Squyres (jsquyres)
Can you reproduce the error in a small example?
Also, try using "use mpi" instead of "include 'mpif.h'", and see if
that turns up any errors.

Post by Diego Avesani
Dear Gilles, Dear all,
I have found the error. Some CPU has no element to share. It was a my

error.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x137b7b4, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fffe8726fc0) failed

buffer in send and recv

each MPI call.

Post by Diego Avesani
I recommend you first do this, so you can catch the error as soon it

happens, and hopefully understand why it occursïŒ

same variable in send and receiving Buff.

Post by Diego Avesani
I have written my code in OPEN-MPI, but unfortunately I have to run

in on a intel-MPI cluster.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x1dd27b0, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fff9d7dd9f0) failed

Post by Diego Avesani
CALL MPI_TYPE_VECTOR(1, Ncoeff_MLS, Ncoeff_MLS,

MPI_DOUBLE_PRECISION, coltype, MPIdata%iErr)

Post by Diego Avesani
CALL MPI_TYPE_COMMIT(coltype, MPIdata%iErr)
!
CALL MPI_TYPE_VECTOR(1, nVar, nVar, coltype, MPI_WENO_TYPE,

MPIdata%iErr)

http://www.open-mpi.org/community/lists/users/2015/09/27523.php

Post by Diego Avesani
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27527.php

Thomas Jahns

2015-09-03 15:04:59 UTC

Permalink

Hello,

When you change environment, that is change between OpenMPI and Intel MPI, or
compiler, it is recommended that you recompile everything.
use mpi
is a module, you cannot mix these between compilers/environments, sadly the
Fortran specification does not enforce a strict module format which is why this
is necessary.

this is sensible because the ISO Fortran standard also does not enforce the
memory layout of descriptors that get passed around in a Fortran program, so
having a module from one compiler work in another cannot work unless a common
ABI for arguments is agreed upon (like it was for Fortran 77, where the SysV ABI
is pretty consistently used).

Because Fortran modules (introduced in Fortran 90) supply more semantics than
the Fortran 77 mpif.h it's often useful to 'use mpi' than 'include "mpif.h"'.

Dear Jeff, Dear all,
I normaly use "USE MPI"
/If you are switching between intel and openmpi you must remember not to mix
environment. You might use modules to manage this. As the data types
encodings differ, you must take care that all objects are built against the
same headers./
Could someone explain me what are these modules and how I can use them?

This refers to the 'modules' software package[1] (different from Fortran
modules) which simplifies having multiple versions of the same and different
software packages with the same commands around.

It's typically used in server environments where an upgrade for all
users/dependent software packages is non-trivial.

Regards, Thomas

[1] http://modules.sourceforge.net/

Gilles Gouaillardet

2015-09-03 14:56:45 UTC

Permalink

Diego,

basically that means "do not build with openmpi and run with intelmpi, or
the other way around" and/or "do not build a part of your app with openmpi
and an other part with intelmpi"
"part" can be your app or the use of third party libraries.
if you use intel scalapack, make you use the lib built for openmpi

that can happen by inadvertence if environment ($PATH) is messed up.
a convenient way to keep its environment clean is to use modules
http://modules.sourceforge.net

if module files are correctly written, there should be virtually no way to
mix intelmpi and openmpi, or use openmpi with a lib built with intelmpi.

Cheers,

Gilles

Post by Jeff Squyres (jsquyres)
Can you reproduce the error in a small example?
Also, try using "use mpi" instead of "include 'mpif.h'", and see if that
turns up any errors.

Post by Diego Avesani
Dear Gilles, Dear all,
I have found the error. Some CPU has no element to share. It was a my

error.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x137b7b4, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fffe8726fc0) failed

buffer in send and recv

each MPI call.

Post by Diego Avesani
I recommend you first do this, so you can catch the error as soon it

happens, and hopefully understand why it occursïŒ

Post by Diego Avesani
Cheers,
Gilles
Dear all,
I have notice small difference between OPEN-MPI and intel MPI.
For example in MPI_ALLREDUCE in intel MPI is not allowed to use the

same variable in send and receiving Buff.

Post by Diego Avesani
I have written my code in OPEN-MPI, but unfortunately I have to run in

on a intel-MPI cluster.

Post by Diego Avesani
MPI_Isend(158): MPI_Isend(buf=0x1dd27b0, count=1, INVALID DATATYPE,

dest=0, tag=0, comm=0x0, request=0x7fff9d7dd9f0) failed

Post by Diego Avesani
CALL MPI_TYPE_VECTOR(1, Ncoeff_MLS, Ncoeff_MLS,

MPI_DOUBLE_PRECISION, coltype, MPIdata%iErr)

Post by Diego Avesani
CALL MPI_TYPE_COMMIT(coltype, MPIdata%iErr)
!
CALL MPI_TYPE_VECTOR(1, nVar, nVar, coltype, MPI_WENO_TYPE,

MPIdata%iErr)

http://www.open-mpi.org/community/lists/users/2015/09/27523.php

Post by Diego Avesani
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users

Timothy Brown

2015-09-03 15:06:53 UTC

Permalink

Hi Diego,

I think the Intel HPC forum comment is about using environment modules to manage your environment (PATH, LD_LIBRARY_PATH variables).

Most HPC systems use environment modules:
- Tcl ( http://modules.cvs.sourceforge.net/viewvc/modules/modules/tcl/ )
- C/Tcl ( http://sourceforge.net/project/showfiles.php?group_id=15538 )
- Lmod ( https://www.tacc.utexas.edu/research-development/tacc-projects/lmod )

If your system has environment modules, you'd typically.
- load a compiler (Intel, GCC, PGI, etc).
- load a MPI built with that compiler (Intel MPI, OpenMPI).

The most important thing here is to have a software stack that is consistent. That is built with the same compiler. For example GCC with OpenMPI to build and execute your program. While not GCC and OpenMPI to build then Intel and Intel MPI to execute.

Regards

Post by Diego Avesani
Dear Jeff, Dear all,
I normaly use "USE MPI"
If you are switching between intel and openmpi you must remember not to mix environment. You might use modules to manage this. As the data types encodings differ, you must take care that all objects are built against the same headers.
Could someone explain me what are these modules and how I can use them?
Thanks
Diego
Diego
Can you reproduce the error in a small example?
Also, try using "use mpi" instead of "include 'mpif.h'", and see if that turns up any errors.

--
Jeff Squyres
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: http://www.open-mpi.org/community/lists/users/2015/09/27525.php
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: http://www.open-mpi.org/community/lists/users/2015/09/27527.php

Jeff Squyres (jsquyres)

2015-09-03 15:13:21 UTC

Permalink

Post by Diego Avesani
Dear Jeff, Dear all,
I normaly use "USE MPI"
If you are switching between intel and openmpi you must remember not to mix environment. You might use modules to manage this.

I think the source of the confusion here might well be an overload of the word "modules".

I think the word "module" in the phrase "You might use modules to manage this" is referring to *environment modules*, not *Fortran modules*. I.e.: http://modules.sourceforge.net/

Where you can do stuff like this:

-----
# Use Open MPI
$ module load openmpi
$ mpicc my_program.c
$ mpirun -np 4 a.out

# Use __some_other_MPI__
$ module load othermpi
$ mpicc my_program.c
$ mpirun -np 4 a.out
-----

Environment modules are typically used to set things like PATH, LD_LIBRARY_PATH, and MANPATH.

I think the poster on the Intel HPC forum was probably referring to you using environment modules to switch your PATH / LD_LIBRARY_PATH / MANPATH between Open MPI and Intel MPI.

Post by Diego Avesani
As the data types encodings differ, you must take care that all objects are built against the same headers.

--
Jeff Squyres
***@cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

Bennet Fauber

2015-09-03 15:23:43 UTC

Permalink

There is also the package Lmod, which provides similar functionality
to environment modules. It is maintained by TACC.

https://www.tacc.utexas.edu/research-development/tacc-projects/lmod

but I think the current source code is at

https://github.com/TACC/Lmod

-- bennet

On Thu, Sep 3, 2015 at 11:13 AM, Jeff Squyres (jsquyres)

Post by Jeff Squyres (jsquyres)

Post by Diego Avesani
Dear Jeff, Dear all,
I normaly use "USE MPI"
If you are switching between intel and openmpi you must remember not to mix environment. You might use modules to manage this.

I think the source of the confusion here might well be an overload of the word "modules".
I think the word "module" in the phrase "You might use modules to manage this" is referring to *environment modules*, not *Fortran modules*. I.e.: http://modules.sourceforge.net/
-----
# Use Open MPI
$ module load openmpi
$ mpicc my_program.c
$ mpirun -np 4 a.out
# Use __some_other_MPI__
$ module load othermpi
$ mpicc my_program.c
$ mpirun -np 4 a.out
-----
Environment modules are typically used to set things like PATH, LD_LIBRARY_PATH, and MANPATH.
I think the poster on the Intel HPC forum was probably referring to you using environment modules to switch your PATH / LD_LIBRARY_PATH / MANPATH between Open MPI and Intel MPI.

Post by Diego Avesani
As the data types encodings differ, you must take care that all objects are built against the same headers.

Here, the poster is essentially saying that if you want to use Open MPI, you have to compile and mpirun with Open MPI. And if you want to use Open MPI, you have to (re)compile and mpirun with Intel MPI.
In short: Open MPI and Intel MPI are not binary compatible, and their mpirun's are not compatible, either.
(note that this is an Open MPI mailing list; we can't answer questions about Intel MPI here)
My point with "use mpi" was that you should try replacing "include 'mpif.h'" with "use mpi" in your Fortran blocks. Open MPI's "use mpi" implementation will do a lot of compile-time type checking that "include 'mpif.h'" will not. Hence, it help determine if you're passing an incorrect parameter to an MPI subroutine, for example.
--
Jeff Squyres
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: http://www.open-mpi.org/community/lists/users/2015/09/27537.php

Diego Avesani

2015-09-04 08:23:46 UTC

Permalink

Dear all,
let's me check all your mail, because there are a lot of thing that I can
not understand.

As soon as possible I will reply, hopefully.

Diego

Post by Bennet Fauber
There is also the package Lmod, which provides similar functionality
to environment modules. It is maintained by TACC.
https://www.tacc.utexas.edu/research-development/tacc-projects/lmod
but I think the current source code is at
https://github.com/TACC/Lmod
-- bennet
On Thu, Sep 3, 2015 at 11:13 AM, Jeff Squyres (jsquyres)

Post by Jeff Squyres (jsquyres)

Post by Diego Avesani
Dear Jeff, Dear all,
I normaly use "USE MPI"
If you are switching between intel and openmpi you must remember not to

mix environment. You might use modules to manage this.

Post by Jeff Squyres (jsquyres)
I think the source of the confusion here might well be an overload of

the word "modules".

Post by Jeff Squyres (jsquyres)
I think the word "module" in the phrase "You might use modules to manage

http://modules.sourceforge.net/

Post by Jeff Squyres (jsquyres)
-----
# Use Open MPI
$ module load openmpi
$ mpicc my_program.c
$ mpirun -np 4 a.out
# Use __some_other_MPI__
$ module load othermpi
$ mpicc my_program.c
$ mpirun -np 4 a.out
-----
Environment modules are typically used to set things like PATH,

LD_LIBRARY_PATH, and MANPATH.

Post by Jeff Squyres (jsquyres)
I think the poster on the Intel HPC forum was probably referring to you

using environment modules to switch your PATH / LD_LIBRARY_PATH / MANPATH
between Open MPI and Intel MPI.

Post by Jeff Squyres (jsquyres)

Post by Diego Avesani
As the data types encodings differ, you must take care that all objects

are built against the same headers.

Post by Jeff Squyres (jsquyres)
Here, the poster is essentially saying that if you want to use Open MPI,

you have to compile and mpirun with Open MPI. And if you want to use Open
MPI, you have to (re)compile and mpirun with Intel MPI.

Post by Jeff Squyres (jsquyres)
In short: Open MPI and Intel MPI are not binary compatible, and their

mpirun's are not compatible, either.

Post by Jeff Squyres (jsquyres)
(note that this is an Open MPI mailing list; we can't answer questions

about Intel MPI here)

Post by Jeff Squyres (jsquyres)
My point with "use mpi" was that you should try replacing "include

'mpif.h'" with "use mpi" in your Fortran blocks. Open MPI's "use mpi"
implementation will do a lot of compile-time type checking that "include
'mpif.h'" will not. Hence, it help determine if you're passing an
incorrect parameter to an MPI subroutine, for example.

Post by Jeff Squyres (jsquyres)
--
Jeff Squyres

http://www.cisco.com/web/about/doing_business/legal/cri/

Post by Jeff Squyres (jsquyres)
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users

http://www.open-mpi.org/community/lists/users/2015/09/27537.php
_______________________________________________
users mailing list
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
http://www.open-mpi.org/community/lists/users/2015/09/27538.php