[OMPI users] OMPI users] upgraded mpi and R and now cannot find slots

Discussion:

Gilles Gouaillardet

2017-10-03 14:07:25 UTC

Thanks Gilles, relative noob here at this level, apologies if nonsensical!
I removed previous versions of open mpi which were compiled from source using sudo make uninstall ...
downloaded new open-mpi 3.0.0 in tar.gz
configure --disable-dlopen
sudo make install
then ran sudo ldconfig
updated R, downloaded R-3.4.2.tar.gz
./configure
sudo make install
Then run R from sudo
sudo R
once runningÂ
install.packages("Rmpi")
install.packages("doMPI")
both of these load and test fine during install
Then from R run
rm(list=ls(all=TRUE))
library(doMPI)
## load MPI cluster
cl <- startMPIcluster(count=6)
At this point it throws the error, doesn't find any of the slots.
There is a precompiled version of Rmpi that installs an older version of open-mpi directly from Ubuntu, but I think the mpi version is an older one so I wanted to try using the new version.
I use this 6 core (12) asÂ test bed before uploading to a cluster.Â It is Ubuntu 16.04 Linux, lstopo pdf is attached.
Thanks,
J
Hi Jim,
can you please provide minimal instructions on how to reproduce the issue ?
we know Open MPI, but i am afraid few or none of us know about Rmpi nor doMPI.
once you explain how to download and build these, and how to run the
failing test,
we ll be able to investigate that.
also, can you describe your environment ?
i assume one ubuntu machine, can you please run
lstopo
on and post the output ?
did you use to have some specific settings in the system-wide conf
file (e.g. /.../etc/openmpi-mca-params.conf) ?
if yes, can you post these, the syntax might have changed in 3.0.0
Cheers,
Gilles

I've used this for years, just updated open-mpi to 3.0.0 and reloaded R,
have reinstalled doMPI and thus Rmpi but when I try to use startMPICluster,
asking for 6 slots (there are 12 on this machine) I get this error.Â Where
can I start to debug it?
Thanks
J
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 6 slots
Â Â /usr/lib/R/bin/Rscript
Either request fewer slots for your application, or make more slots
available
for use.
--------------------------------------------------------------------------
Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves = count,
Â Â MPI_ERR_SPAWN: could not spawn processes
--
Jim Maas
jimmaasukÂ at gmail.com
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
--
Jim Maas
74 Turner Road
Norwich, Norfolk, UK.
NR2 4HB
jimmaasukÂ at gmail.com
http://www.jamaas.com
+ 44 (0)771 985 8698

Jim Maas

2017-10-03 14:22:52 UTC

Permalink

Previously it worked fine if I asked for 12, I'm sure you are correct, it
is only 6 physical cores, but with hyperthreading or whatever, it looks
like 12. The system monitor shows 12.

Thanks
J

On 3 October 2017 at 15:07, Gilles Gouaillardet <

Post by Gilles Gouaillardet
Thanks, i will have a look at it.
By default, a slot is a core, so there are 6 slots on your system.
Could your app spawn 6 procs on top of the initial proc ? That would be 7
slots and there are only 6.
What if you ask 5 slots only ?
With some parameters i do not know off hand, you could either
oversubscribe or use hyperthreads as slots. In both cases, 7 slots would be
available.
Cheers,
Gilles
Thanks Gilles, relative noob here at this level, apologies if nonsensical!
I removed previous versions of open mpi which were compiled from source
using sudo make uninstall ...
downloaded new open-mpi 3.0.0 in tar.gz
configure --disable-dlopen
sudo make install
then ran sudo ldconfig
updated R, downloaded R-3.4.2.tar.gz
./configure
sudo make install
Then run R from sudo
sudo R
once running
install.packages("Rmpi")
install.packages("doMPI")
both of these load and test fine during install
Then from R run
rm(list=ls(all=TRUE))
library(doMPI)
## load MPI cluster
cl <- startMPIcluster(count=6)
At this point it throws the error, doesn't find any of the slots.
There is a precompiled version of Rmpi that installs an older version of
open-mpi directly from Ubuntu, but I think the mpi version is an older one
so I wanted to try using the new version.
I use this 6 core (12) as test bed before uploading to a cluster. It is
Ubuntu 16.04 Linux, lstopo pdf is attached.
Thanks,
J
On 3 October 2017 at 14:06, Gilles Gouaillardet <

Hi Jim,
can you please provide minimal instructions on how to reproduce the issue ?
we know Open MPI, but i am afraid few or none of us know about Rmpi nor doMPI.
once you explain how to download and build these, and how to run the
failing test,
we ll be able to investigate that.
also, can you describe your environment ?
i assume one ubuntu machine, can you please run
lstopo
on and post the output ?
did you use to have some specific settings in the system-wide conf
file (e.g. /.../etc/openmpi-mca-params.conf) ?
if yes, can you post these, the syntax might have changed in 3.0.0
Cheers,
Gilles

I've used this for years, just updated open-mpi to 3.0.0 and reloaded R,
have reinstalled doMPI and thus Rmpi but when I try to use

startMPICluster,

asking for 6 slots (there are 12 on this machine) I get this error.

Where

can I start to debug it?
Thanks
J
------------------------------------------------------------

--------------

There are not enough slots available in the system to satisfy the 6

slots

/usr/lib/R/bin/Rscript
Either request fewer slots for your application, or make more slots
available
for use.
------------------------------------------------------------

--------------

Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves =

count,

MPI_ERR_SPAWN: could not spawn processes
--
Jim Maas
jimmaasuk at gmail.com
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
Jim Maas
74 Turner Road
Norwich, Norfolk, UK.
NR2 4HB
jimmaasuk at gmail.com
http://www.jamaas.com
+ 44 (0)771 985 8698 <07719%20858698>
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
Jim Maas
74 Turner Road
Norwich, Norfolk, UK.
NR2 4HB

jimmaasuk at gmail.com
http://www.jamaas.com
+ 44 (0)771 985 8698

Jim Maas

2017-10-03 14:27:33 UTC

Permalink

Tried this and got this error, and slots are available, nothing else is
running.

cl <- startMPIcluster(count=7)

--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 7 slots
that were requested by the application:
/usr/local/lib/R/bin/Rscript

Either request fewer slots for your application, or make more slots
available
for use.
--------------------------------------------------------------------------
Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves = count,
:
MPI_ERR_SPAWN: could not spawn processes
On 3 October 2017 at 15:07, Gilles Gouaillardet <

Thanks, i will have a look at it.
By default, a slot is a core, so there are 6 slots on your system.
Could your app spawn 6 procs on top of the initial proc ? That would be 7
slots and there are only 6.
What if you ask 5 slots only ?
With some parameters i do not know off hand, you could either
oversubscribe or use hyperthreads as slots. In both cases, 7 slots would be
available.
Cheers,
Gilles
Thanks Gilles, relative noob here at this level, apologies if nonsensical!
I removed previous versions of open mpi which were compiled from source
using sudo make uninstall ...
downloaded new open-mpi 3.0.0 in tar.gz
configure --disable-dlopen
sudo make install
then ran sudo ldconfig
updated R, downloaded R-3.4.2.tar.gz
./configure
sudo make install
Then run R from sudo
sudo R
once running
install.packages("Rmpi")
install.packages("doMPI")
both of these load and test fine during install
Then from R run
rm(list=ls(all=TRUE))
library(doMPI)
## load MPI cluster
cl <- startMPIcluster(count=6)
At this point it throws the error, doesn't find any of the slots.
There is a precompiled version of Rmpi that installs an older version of
open-mpi directly from Ubuntu, but I think the mpi version is an older one
so I wanted to try using the new version.
I use this 6 core (12) as test bed before uploading to a cluster. It is
Ubuntu 16.04 Linux, lstopo pdf is attached.
Thanks,
J
On 3 October 2017 at 14:06, Gilles Gouaillardet <

I've used this for years, just updated open-mpi to 3.0.0 and reloaded R,
have reinstalled doMPI and thus Rmpi but when I try to use

startMPICluster,

asking for 6 slots (there are 12 on this machine) I get this error.

Where

can I start to debug it?
Thanks
J
------------------------------------------------------------

--------------

There are not enough slots available in the system to satisfy the 6

slots

/usr/lib/R/bin/Rscript
Either request fewer slots for your application, or make more slots
available
for use.
------------------------------------------------------------

--------------

Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves =

count,

MPI_ERR_SPAWN: could not spawn processes
--
Jim Maas
jimmaasuk at gmail.com
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

r***@open-mpi.org

2017-10-03 15:01:03 UTC

Permalink

As Gilles said, we default to slots = cores, not HTs. If you want to treat HTs as independent cpus, then you need to add OMPI_MCA_hwloc_base_use_hwthreads_as_cpus=1 in your environment.

Tried this and got this error, and slots are available, nothing else is running.

cl <- startMPIcluster(count=7)

--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 7 slots
/usr/local/lib/R/bin/Rscript
Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
MPI_ERR_SPAWN: could not spawn processes
Thanks, i will have a look at it.
By default, a slot is a core, so there are 6 slots on your system.
Could your app spawn 6 procs on top of the initial proc ? That would be 7 slots and there are only 6.
What if you ask 5 slots only ?
With some parameters i do not know off hand, you could either oversubscribe or use hyperthreads as slots. In both cases, 7 slots would be available.
Cheers,
Gilles
Thanks Gilles, relative noob here at this level, apologies if nonsensical!
I removed previous versions of open mpi which were compiled from source using sudo make uninstall ...
downloaded new open-mpi 3.0.0 in tar.gz
configure --disable-dlopen
sudo make install
then ran sudo ldconfig
updated R, downloaded R-3.4.2.tar.gz
./configure
sudo make install
Then run R from sudo
sudo R
once running
install.packages("Rmpi")
install.packages("doMPI")
both of these load and test fine during install
Then from R run
rm(list=ls(all=TRUE))
library(doMPI)
## load MPI cluster
cl <- startMPIcluster(count=6)
At this point it throws the error, doesn't find any of the slots.
There is a precompiled version of Rmpi that installs an older version of open-mpi directly from Ubuntu, but I think the mpi version is an older one so I wanted to try using the new version.
I use this 6 core (12) as test bed before uploading to a cluster. It is Ubuntu 16.04 Linux, lstopo pdf is attached.
Thanks,
J
Hi Jim,
can you please provide minimal instructions on how to reproduce the issue ?
we know Open MPI, but i am afraid few or none of us know about Rmpi nor doMPI.
once you explain how to download and build these, and how to run the
failing test,
we ll be able to investigate that.
also, can you describe your environment ?
i assume one ubuntu machine, can you please run
lstopo
on and post the output ?
did you use to have some specific settings in the system-wide conf
file (e.g. /.../etc/openmpi-mca-params.co <http://openmpi-mca-params.co/>nf) ?
if yes, can you post these, the syntax might have changed in 3.0.0
Cheers,
Gilles

I've used this for years, just updated open-mpi to 3.0.0 and reloaded R,
have reinstalled doMPI and thus Rmpi but when I try to use startMPICluster,
asking for 6 slots (there are 12 on this machine) I get this error. Where
can I start to debug it?
Thanks
J
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 6 slots
/usr/lib/R/bin/Rscript
Either request fewer slots for your application, or make more slots
available
for use.
--------------------------------------------------------------------------
Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves = count,
MPI_ERR_SPAWN: could not spawn processes
--
Jim Maas
jimmaasuk at gmail.com <http://gmail.com/>
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users <https://lists.open-mpi.org/mailman/listinfo/users>

_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users <https://lists.open-mpi.org/mailman/listinfo/users>
--
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users <https://lists.open-mpi.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users <https://lists.open-mpi.org/mailman/listinfo/users>

Jim Maas

2017-10-03 19:44:01 UTC

Permalink

Thanks RHC .... where do I put that so it will be in the environment?

J

Post by r***@open-mpi.org
As Gilles said, we default to slots = cores, not HTs. If you want to treat
HTs as independent cpus, then you need to add OMPI_MCA_hwloc_base_use_hwthreads_as_cpus=1
in your environment.
Tried this and got this error, and slots are available, nothing else is running.

cl <- startMPIcluster(count=7)

I've used this for years, just updated open-mpi to 3.0.0 and reloaded

have reinstalled doMPI and thus Rmpi but when I try to use

startMPICluster,

asking for 6 slots (there are 12 on this machine) I get this error.

Where

can I start to debug it?
Thanks
J
------------------------------------------------------------

--------------

There are not enough slots available in the system to satisfy the 6

slots

/usr/lib/R/bin/Rscript
Either request fewer slots for your application, or make more slots available
for use.
------------------------------------------------------------

--------------

Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves =

count,

MPI_ERR_SPAWN: could not spawn processes
--
Jim Maas
jimmaasuk at gmail.com
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

r***@open-mpi.org

2017-10-03 20:06:04 UTC

Permalink

You can add it to the default MCA param file, if you want - <prefix>/etc/openmpi-mca-params.conf

Post by Jim Maas
Thanks RHC .... where do I put that so it will be in the environment?
J
As Gilles said, we default to slots = cores, not HTs. If you want to treat HTs as independent cpus, then you need to add OMPI_MCA_hwloc_base_use_hwthreads_as_cpus=1 in your environment.

Tried this and got this error, and slots are available, nothing else is running.

cl <- startMPIcluster(count=7)

I've used this for years, just updated open-mpi to 3.0.0 and reloaded R,
have reinstalled doMPI and thus Rmpi but when I try to use startMPICluster,
asking for 6 slots (there are 12 on this machine) I get this error. Where
can I start to debug it?
Thanks
J
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 6 slots
/usr/lib/R/bin/Rscript
Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves = count,
MPI_ERR_SPAWN: could not spawn processes
--
Jim Maas
jimmaasuk at gmail.com <http://gmail.com/>
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users <https://lists.open-mpi.org/mailman/listinfo/users>

Jim Maas

2017-10-03 20:54:12 UTC

Permalink

Not sure exactly how but back up and running, with all cores. Thanks to
all. Hopefully the new binary for Ubuntu with open-mpi 3.0.0 won't be too
far away.

Very Best,
J

Post by r***@open-mpi.org
You can add it to the default MCA param file, if you want -
<prefix>/etc/openmpi-mca-params.conf
Thanks RHC .... where do I put that so it will be in the environment?
J

Post by r***@open-mpi.org
As Gilles said, we default to slots = cores, not HTs. If you want to
treat HTs as independent cpus, then you need to add
OMPI_MCA_hwloc_base_use_hwthreads_as_cpus=1 in your environment.
Tried this and got this error, and slots are available, nothing else is running.

cl <- startMPIcluster(count=7)

Thanks, i will have a look at it.
By default, a slot is a core, so there are 6 slots on your system.
Could your app spawn 6 procs on top of the initial proc ? That would be
7 slots and there are only 6.
What if you ask 5 slots only ?
With some parameters i do not know off hand, you could either
oversubscribe or use hyperthreads as slots. In both cases, 7 slots would be
available.
Cheers,
Gilles
Thanks Gilles, relative noob here at this level, apologies if nonsensical!
I removed previous versions of open mpi which were compiled from source
using sudo make uninstall ...
downloaded new open-mpi 3.0.0 in tar.gz
configure --disable-dlopen
sudo make install
then ran sudo ldconfig
updated R, downloaded R-3.4.2.tar.gz
./configure
sudo make install
Then run R from sudo
sudo R
once running
install.packages("Rmpi")
install.packages("doMPI")
both of these load and test fine during install
Then from R run
rm(list=ls(all=TRUE))
library(doMPI)
## load MPI cluster
cl <- startMPIcluster(count=6)
At this point it throws the error, doesn't find any of the slots.
There is a precompiled version of Rmpi that installs an older version of
open-mpi directly from Ubuntu, but I think the mpi version is an older one
so I wanted to try using the new version.
I use this 6 core (12) as test bed before uploading to a cluster. It
is Ubuntu 16.04 Linux, lstopo pdf is attached.
Thanks,
J
On 3 October 2017 at 14:06, Gilles Gouaillardet <gilles.gouaillar

I've used this for years, just updated open-mpi to 3.0.0 and reloaded

have reinstalled doMPI and thus Rmpi but when I try to use

startMPICluster,

asking for 6 slots (there are 12 on this machine) I get this error.

Where

can I start to debug it?
Thanks
J
------------------------------------------------------------

--------------

There are not enough slots available in the system to satisfy the 6

slots

/usr/lib/R/bin/Rscript
Either request fewer slots for your application, or make more slots available
for use.
------------------------------------------------------------

--------------

Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves =

count,

MPI_ERR_SPAWN: could not spawn processes
--
Jim Maas
jimmaasuk at gmail.com
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users