[OMPI users] Problem with MPI_Comm_spawn using openmpi 2.0.x + sbatch

[Anastasia Kruchinina]

Hi,

I am trying to use MPI_Comm_spawn function in my code. I am having trouble
with openmpi 2.0.x + sbatch (batch system Slurm).
My test program is located here:
http://user.it.uu.se/~anakr367/files/MPI_test/

When I am running my code I am getting an error:

OPAL ERROR: Timeout in file
../../../../openmpi-2.0.1/opal/mca/pmix/base/pmix_base_fns.c at line 193
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

ompi_dpm_dyn_init() failed
--> Returned "Timeout" (-15) instead of "Success" (0)
--------------------------------------------------------------------------

The interesting thing is that there is no error when I am firstly
allocating nodes with salloc and then run my program. So, I noticed that
the program works fine using openmpi 1.x+sbach/salloc or openmpi
2.0.x+salloc but not openmpi 2.0.x+sbatch.

The error was reproduced on three different computer clusters.

Best regards,
Anastasia

[r***@open-mpi.org]

Nothing immediate comes to mind - all sbatch does is create an allocation and then run your script in it. Perhaps your script is using a different “mpirun” command than when you type it interactively?

Hi,
I am trying to use MPI_Comm_spawn function in my code. I am having trouble with openmpi 2.0.x + sbatch (batch system Slurm).
My test program is located here: http://user.it.uu.se/~anakr367/files/MPI_test/ <http://user.it.uu.se/%7Eanakr367/files/MPI_test/>
OPAL ERROR: Timeout in file
../../../../openmpi-2.0.1/opal/mca/pmix/base/pmix_base_fns.c at line 193
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
ompi_dpm_dyn_init() failed
--> Returned "Timeout" (-15) instead of "Success" (0)
--------------------------------------------------------------------------
The interesting thing is that there is no error when I am firstly allocating nodes with salloc and then run my program. So, I noticed that the program works fine using openmpi 1.x+sbach/salloc or openmpi 2.0.x+salloc but not openmpi 2.0.x+sbatch.
The error was reproduced on three different computer clusters.
Best regards,
Anastasia
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Howard Pritchard]

Hi Anastasia,

Definitely check the mpirun when in batch environment but you may also want
to upgrade to Open MPI 2.0.2.

Howard

Post by r***@open-mpi.org
Nothing immediate comes to mind - all sbatch does is create an allocation
and then run your script in it. Perhaps your script is using a different
“mpirun” command than when you type it interactively?
On Feb 14, 2017, at 5:11 AM, Anastasia Kruchinina <
Hi,
I am trying to use MPI_Comm_spawn function in my code. I am having trouble
with openmpi 2.0.x + sbatch (batch system Slurm).
http://user.it.uu.se/~anakr367/files/MPI_test/
OPAL ERROR: Timeout in file
../../../../openmpi-2.0.1/opal/mca/pmix/base/pmix_base_fns.c at line 193
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
ompi_dpm_dyn_init() failed
--> Returned "Timeout" (-15) instead of "Success" (0)
--------------------------------------------------------------------------
The interesting thing is that there is no error when I am firstly
allocating nodes with salloc and then run my program. So, I noticed that
the program works fine using openmpi 1.x+sbach/salloc or openmpi
2.0.x+salloc but not openmpi 2.0.x+sbatch.
The error was reproduced on three different computer clusters.
Best regards,
Anastasia
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Anastasia Kruchinina]

Hi,

I am running like this:
mpirun -np 1 ./manager

Should I do it differently?

I also thought that all sbatch does is create an allocation and then run my
script in it. But it seems it is not since I am getting these results...

I would like to upgrade to OpenMPI, but no clusters near me have it yet :(
So I even cannot check if it works with OpenMPI 2.0.2.

Post by Howard Pritchard
Hi Anastasia,
Definitely check the mpirun when in batch environment but you may also
want to upgrade to Open MPI 2.0.2.
Howard

Post by r***@open-mpi.org
Nothing immediate comes to mind - all sbatch does is create an allocation
and then run your script in it. Perhaps your script is using a different
“mpirun” command than when you type it interactively?
On Feb 14, 2017, at 5:11 AM, Anastasia Kruchinina <
Hi,
I am trying to use MPI_Comm_spawn function in my code. I am having
trouble with openmpi 2.0.x + sbatch (batch system Slurm).
My test program is located here: http://user.it.uu.se/~
anakr367/files/MPI_test/
OPAL ERROR: Timeout in file
../../../../openmpi-2.0.1/opal/mca/pmix/base/pmix_base_fns.c at line 193
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
ompi_dpm_dyn_init() failed
--> Returned "Timeout" (-15) instead of "Success" (0)
--------------------------------------------------------------------------
The interesting thing is that there is no error when I am firstly
allocating nodes with salloc and then run my program. So, I noticed that
the program works fine using openmpi 1.x+sbach/salloc or openmpi
2.0.x+salloc but not openmpi 2.0.x+sbatch.
The error was reproduced on three different computer clusters.
Best regards,
Anastasia
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[r***@open-mpi.org]

The cmd line looks fine - when you do your “sbatch” request, what is in the shell script you give it? Or are you saying you just “sbatch” the mpirun cmd directly?

Post by Anastasia Kruchinina
Hi,
mpirun -np 1 ./manager
Should I do it differently?
I also thought that all sbatch does is create an allocation and then run my script in it. But it seems it is not since I am getting these results...
I would like to upgrade to OpenMPI, but no clusters near me have it yet :( So I even cannot check if it works with OpenMPI 2.0.2.
Hi Anastasia,
Definitely check the mpirun when in batch environment but you may also want to upgrade to Open MPI 2.0.2.
Howard
Nothing immediate comes to mind - all sbatch does is create an allocation and then run your script in it. Perhaps your script is using a different “mpirun” command than when you type it interactively?

Hi,
I am trying to use MPI_Comm_spawn function in my code. I am having trouble with openmpi 2.0.x + sbatch (batch system Slurm).
My test program is located here: http://user.it.uu.se/~anakr367/files/MPI_test/ <http://user.it.uu.se/%7Eanakr367/files/MPI_test/>
OPAL ERROR: Timeout in file
../../../../openmpi-2.0.1/opal/mca/pmix/base/pmix_base_fns.c at line 193
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
ompi_dpm_dyn_init() failed
--> Returned "Timeout" (-15) instead of "Success" (0)
--------------------------------------------------------------------------
The interesting thing is that there is no error when I am firstly allocating nodes with salloc and then run my program. So, I noticed that the program works fine using openmpi 1.x+sbach/salloc or openmpi 2.0.x+salloc but not openmpi 2.0.x+sbatch.
The error was reproduced on three different computer clusters.
Best regards,
Anastasia
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Anastasia Kruchinina]

Hi!

I am doing like this:

sbatch -N 2 -n 5 ./job.sh

where job.sh is:

#!/bin/bash -l
module load openmpi/2.0.1-icc
mpirun -np 1 ./manager 4

Post by r***@open-mpi.org
The cmd line looks fine - when you do your “sbatch” request, what is in
the shell script you give it? Or are you saying you just “sbatch” the
mpirun cmd directly?
On Feb 15, 2017, at 8:07 AM, Anastasia Kruchinina <
Hi,
mpirun -np 1 ./manager
Should I do it differently?
I also thought that all sbatch does is create an allocation and then run
my script in it. But it seems it is not since I am getting these results...
I would like to upgrade to OpenMPI, but no clusters near me have it yet :(
So I even cannot check if it works with OpenMPI 2.0.2.

Post by Howard Pritchard
Hi Anastasia,
Definitely check the mpirun when in batch environment but you may also
want to upgrade to Open MPI 2.0.2.
Howard

Post by r***@open-mpi.org
Nothing immediate comes to mind - all sbatch does is create an
allocation and then run your script in it. Perhaps your script is using a
different “mpirun” command than when you type it interactively?
On Feb 14, 2017, at 5:11 AM, Anastasia Kruchinina <
Hi,
I am trying to use MPI_Comm_spawn function in my code. I am having
trouble with openmpi 2.0.x + sbatch (batch system Slurm).
My test program is located here: http://user.it.uu.se/~anakr367
/files/MPI_test/
OPAL ERROR: Timeout in file
../../../../openmpi-2.0.1/opal/mca/pmix/base/pmix_base_fns.c at line 193
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
ompi_dpm_dyn_init() failed
--> Returned "Timeout" (-15) instead of "Success" (0)
--------------------------------------------------------------------------
The interesting thing is that there is no error when I am firstly
allocating nodes with salloc and then run my program. So, I noticed that
the program works fine using openmpi 1.x+sbach/salloc or openmpi
2.0.x+salloc but not openmpi 2.0.x+sbatch.
The error was reproduced on three different computer clusters.
Best regards,
Anastasia
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Jason Maldonis]

Just to throw this out there -- to me, that doesn't seem to be just a
problem with SLURM. I'm guessing the exact same error would be thrown
interactively (unless I didn't read the above messages carefully enough).
I had a lot of problems running spawned jobs on 2.0.x a few months ago, so
I switched back to 1.10.2 and everything worked. Just in case that helps
someone.

Jason

On Wed, Feb 15, 2017 at 1:09 PM, Anastasia Kruchinina <

Post by Anastasia Kruchinina
Hi!
sbatch -N 2 -n 5 ./job.sh
#!/bin/bash -l
module load openmpi/2.0.1-icc
mpirun -np 1 ./manager 4

Post by r***@open-mpi.org
The cmd line looks fine - when you do your “sbatch” request, what is in
the shell script you give it? Or are you saying you just “sbatch” the
mpirun cmd directly?
On Feb 15, 2017, at 8:07 AM, Anastasia Kruchinina <
Hi,
mpirun -np 1 ./manager
Should I do it differently?
I also thought that all sbatch does is create an allocation and then run
my script in it. But it seems it is not since I am getting these results...
I would like to upgrade to OpenMPI, but no clusters near me have it yet
:( So I even cannot check if it works with OpenMPI 2.0.2.

Post by Howard Pritchard
Hi Anastasia,
Definitely check the mpirun when in batch environment but you may also
want to upgrade to Open MPI 2.0.2.
Howard

Post by r***@open-mpi.org
Nothing immediate comes to mind - all sbatch does is create an
allocation and then run your script in it. Perhaps your script is using a
different “mpirun” command than when you type it interactively?
On Feb 14, 2017, at 5:11 AM, Anastasia Kruchinina <
Hi,
I am trying to use MPI_Comm_spawn function in my code. I am having
trouble with openmpi 2.0.x + sbatch (batch system Slurm).
My test program is located here: http://user.it.uu.se/~anakr367
/files/MPI_test/
OPAL ERROR: Timeout in file
../../../../openmpi-2.0.1/opal/mca/pmix/base/pmix_base_fns.c at line 193
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
ompi_dpm_dyn_init() failed
--> Returned "Timeout" (-15) instead of "Success" (0)
--------------------------------------------------------------------------
The interesting thing is that there is no error when I am firstly
allocating nodes with salloc and then run my program. So, I noticed that
the program works fine using openmpi 1.x+sbach/salloc or openmpi
2.0.x+salloc but not openmpi 2.0.x+sbatch.
The error was reproduced on three different computer clusters.
Best regards,
Anastasia
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[r***@open-mpi.org]

Yes, 2.0.1 has a spawn issue. We believe that 2.0.2 is okay if you want to give it a try

Sent from my iPad

Just to throw this out there -- to me, that doesn't seem to be just a problem with SLURM. I'm guessing the exact same error would be thrown interactively (unless I didn't read the above messages carefully enough). I had a lot of problems running spawned jobs on 2.0.x a few months ago, so I switched back to 1.10.2 and everything worked. Just in case that helps someone.
Jason

Post by Anastasia Kruchinina
Hi!
sbatch -N 2 -n 5 ./job.sh
#!/bin/bash -l
module load openmpi/2.0.1-icc
mpirun -np 1 ./manager 4

Post by r***@open-mpi.org
The cmd line looks fine - when you do your “sbatch” request, what is in the shell script you give it? Or are you saying you just “sbatch” the mpirun cmd directly?

Post by Anastasia Kruchinina
Hi,
mpirun -np 1 ./manager
Should I do it differently?
I also thought that all sbatch does is create an allocation and then run my script in it. But it seems it is not since I am getting these results...
I would like to upgrade to OpenMPI, but no clusters near me have it yet :( So I even cannot check if it works with OpenMPI 2.0.2.

Post by Howard Pritchard
Hi Anastasia,
Definitely check the mpirun when in batch environment but you may also want to upgrade to Open MPI 2.0.2.
Howard

Post by r***@open-mpi.org
Nothing immediate comes to mind - all sbatch does is create an allocation and then run your script in it. Perhaps your script is using a different “mpirun” command than when you type it interactively?

Hi,
I am trying to use MPI_Comm_spawn function in my code. I am having trouble with openmpi 2.0.x + sbatch (batch system Slurm).
My test program is located here: http://user.it.uu.se/~anakr367/files/MPI_test/
OPAL ERROR: Timeout in file
../../../../openmpi-2.0.1/opal/mca/pmix/base/pmix_base_fns.c at line 193
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
ompi_dpm_dyn_init() failed
--> Returned "Timeout" (-15) instead of "Success" (0)
--------------------------------------------------------------------------
The interesting thing is that there is no error when I am firstly allocating nodes with salloc and then run my program. So, I noticed that the program works fine using openmpi 1.x+sbach/salloc or openmpi 2.0.x+salloc but not openmpi 2.0.x+sbatch.
The error was reproduced on three different computer clusters.
Best regards,
Anastasia
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Anastasia Kruchinina]

Ok, thanks for your answers! I was not aware that it is a known issue.

I guess I will just try to find a machine with OpenMPI/2.0.2 and try there.

Post by r***@open-mpi.org
Yes, 2.0.1 has a spawn issue. We believe that 2.0.2 is okay if you want to give it a try
Sent from my iPad
Just to throw this out there -- to me, that doesn't seem to be just a
problem with SLURM. I'm guessing the exact same error would be thrown
interactively (unless I didn't read the above messages carefully enough).
I had a lot of problems running spawned jobs on 2.0.x a few months ago, so
I switched back to 1.10.2 and everything worked. Just in case that helps
someone.
Jason
On Wed, Feb 15, 2017 at 1:09 PM, Anastasia Kruchinina <

Post by Anastasia Kruchinina
Hi!
sbatch -N 2 -n 5 ./job.sh
#!/bin/bash -l
module load openmpi/2.0.1-icc
mpirun -np 1 ./manager 4

Post by r***@open-mpi.org
The cmd line looks fine - when you do your “sbatch” request, what is in
the shell script you give it? Or are you saying you just “sbatch” the
mpirun cmd directly?
On Feb 15, 2017, at 8:07 AM, Anastasia Kruchinina <
Hi,
mpirun -np 1 ./manager
Should I do it differently?
I also thought that all sbatch does is create an allocation and then run
my script in it. But it seems it is not since I am getting these results...
I would like to upgrade to OpenMPI, but no clusters near me have it yet
:( So I even cannot check if it works with OpenMPI 2.0.2.

Post by Howard Pritchard
Hi Anastasia,
Definitely check the mpirun when in batch environment but you may also
want to upgrade to Open MPI 2.0.2.
Howard

Post by r***@open-mpi.org
Nothing immediate comes to mind - all sbatch does is create an
allocation and then run your script in it. Perhaps your script is using a
different “mpirun” command than when you type it interactively?
On Feb 14, 2017, at 5:11 AM, Anastasia Kruchinina <
Hi,
I am trying to use MPI_Comm_spawn function in my code. I am having
trouble with openmpi 2.0.x + sbatch (batch system Slurm).
My test program is located here: http://user.it.uu.se/~anakr367
/files/MPI_test/
OPAL ERROR: Timeout in file
../../../../openmpi-2.0.1/opal/mca/pmix/base/pmix_base_fns.c at line 193
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
ompi_dpm_dyn_init() failed
--> Returned "Timeout" (-15) instead of "Success" (0)
--------------------------------------------------------------------------
The interesting thing is that there is no error when I am firstly
allocating nodes with salloc and then run my program. So, I noticed that
the program works fine using openmpi 1.x+sbach/salloc or openmpi
2.0.x+salloc but not openmpi 2.0.x+sbatch.
The error was reproduced on three different computer clusters.
Best regards,
Anastasia
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users