[OMPI users] IBM Spectrum MPI problem

Discussion:

Gabriele Fatigati

2017-05-18 12:02:08 UTC

Dear OpenMPI users and developers, I'm using IBM Spectrum MPI 10.1.0 based
on OpenMPI, so I hope there are some MPI expert can help me to solve the
problem.

When I run a simple Hello World MPI program, I get the follow error message:

A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded). Note that
Open MPI stopped checking at the first component that it did not find.

Host: openpower
Framework: pml
Component: pami
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)

My sysadmin used official IBM Spectrum packages to install MPI, so It's
quite strange that there are some components missing (pami). Any help?
Thanks
--
Ing. Gabriele Fatigati

HPC specialist

SuperComputing Applications and Innovation Department

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.it Tel: +39 051 6171722

g.fatigati [AT] cineca.it

Reuti

2017-05-18 12:10:44 UTC

Permalink

Hi,

Post by Gabriele Fatigati
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI 10.1.0

I noticed this on IBM's website too. Is this freely available? Up to now I was always bounced back to their former Platform MPI when trying to download the community edition (even the evaluation link on the Spectrum MPI page does the same).

-- Reuti

Post by Gabriele Fatigati
based on OpenMPI, so I hope there are some MPI expert can help me to solve the problem.
A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded). Note that
Open MPI stopped checking at the first component that it did not find.
Host: openpower
Framework: pml
Component: pami
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum packages to install MPI, so It's quite strange that there are some components missing (pami). Any help? Thanks
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati [AT] cineca.it
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

John Hearns via users

2017-05-18 12:13:51 UTC

Permalink

Gabriele, as this is based on OpenMPI can you run ompi_info
then look for the btl which are available and the mtl which are available?

Post by Reuti
Hi,

Post by Gabriele Fatigati
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI 10.1.0

I noticed this on IBM's website too. Is this freely available? Up to now I
was always bounced back to their former Platform MPI when trying to
download the community edition (even the evaluation link on the Spectrum
MPI page does the same).
-- Reuti

Post by Gabriele Fatigati
based on OpenMPI, so I hope there are some MPI expert can help me to

solve the problem.

Post by Gabriele Fatigati
When I run a simple Hello World MPI program, I get the follow error
A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded). Note that
Open MPI stopped checking at the first component that it did not find.
Host: openpower
Framework: pml
Component: pami
------------------------------------------------------------

--------------

Post by Gabriele Fatigati
------------------------------------------------------------

--------------

Post by Gabriele Fatigati
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or

environment

Post by Gabriele Fatigati
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of "Success" (0)
------------------------------------------------------------

--------------

Post by Gabriele Fatigati
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum packages to install MPI, so It's

quite strange that there are some components missing (pami). Any help?
Thanks

Post by Gabriele Fatigati
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati [AT] cineca.it
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-18 12:20:08 UTC

Permalink

Hi John, about btl this is the output of ompi_info:

MCA btl: self (MCA v2.1.0, API v3.0.0, Component v10.1.0)
MCA btl: sm (MCA v2.1.0, API v3.0.0, Component v10.1.0)
MCA btl: tcp (MCA v2.1.0, API v3.0.0, Component v10.1.0)
MCA btl: vader (MCA v2.1.0, API v3.0.0, Component v10.1.0)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v10.1.0)

about mtl no information retrieve ompi_info

Post by John Hearns via users
Gabriele, as this is based on OpenMPI can you run ompi_info
then look for the btl which are available and the mtl which are available?

Post by Reuti
Hi,

Post by Gabriele Fatigati
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI 10.1.0

I noticed this on IBM's website too. Is this freely available? Up to now
I was always bounced back to their former Platform MPI when trying to
download the community edition (even the evaluation link on the Spectrum
MPI page does the same).
-- Reuti

Post by Gabriele Fatigati
based on OpenMPI, so I hope there are some MPI expert can help me to

solve the problem.

--------------

Post by Gabriele Fatigati
------------------------------------------------------------

--------------

environment

--------------

quite strange that there are some components missing (pami). Any help?
Thanks

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

John Hearns via users

2017-05-18 12:32:16 UTC

Permalink

One very stupid question... what does 'ibv_devinfo' say when you run
it on the compute nodes?

ps. I know nothing about IBM MPI or pami but I think this is as you say
some simple library being missing etc.

Post by Gabriele Fatigati
MCA btl: self (MCA v2.1.0, API v3.0.0, Component v10.1.0)
MCA btl: sm (MCA v2.1.0, API v3.0.0, Component v10.1.0)
MCA btl: tcp (MCA v2.1.0, API v3.0.0, Component v10.1.0)
MCA btl: vader (MCA v2.1.0, API v3.0.0, Component v10.1.0)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v10.1.0)
about mtl no information retrieve ompi_info

Post by John Hearns via users
Gabriele, as this is based on OpenMPI can you run ompi_info
then look for the btl which are available and the mtl which are available?

Post by Reuti
Hi,

Post by Gabriele Fatigati
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI 10.1.0

I noticed this on IBM's website too. Is this freely available? Up to now
I was always bounced back to their former Platform MPI when trying to
download the community edition (even the evaluation link on the Spectrum
MPI page does the same).
-- Reuti

Post by Gabriele Fatigati
based on OpenMPI, so I hope there are some MPI expert can help me to

solve the problem.

--------------

Post by Gabriele Fatigati
------------------------------------------------------------

--------------

environment

--------------

Post by Gabriele Fatigati
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now

abort,

Post by Gabriele Fatigati
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum packages to install MPI, so

It's quite strange that there are some components missing (pami). Any help?
Thanks

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
<+39%20051%20617%201722>
g.fatigati [AT] cineca.it
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-18 12:18:51 UTC

Permalink

Hi Reuti, I think is it freely available. I posted also on IBM Spectrum
forum, I'm waiting some reply.

Post by Reuti
Hi,

Post by Gabriele Fatigati
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI 10.1.0

I noticed this on IBM's website too. Is this freely available? Up to now I
was always bounced back to their former Platform MPI when trying to
download the community edition (even the evaluation link on the Spectrum
MPI page does the same).
-- Reuti

Post by Gabriele Fatigati
based on OpenMPI, so I hope there are some MPI expert can help me to

solve the problem.

--------------

Post by Gabriele Fatigati
------------------------------------------------------------

--------------

environment

--------------

quite strange that there are some components missing (pami). Any help?
Thanks

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gilles Gouaillardet

2017-05-18 13:04:55 UTC

Permalink

Gabriele,

can you
ompi_info --all | grep pml

also, make sure there is nothing in your environment pointing to an other
Open MPI install
for example
ldd a.out
should only point to IBM libraries

Cheers,

Gilles

Post by Gabriele Fatigati
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI 10.1.0 based
on OpenMPI, so I hope there are some MPI expert can help me to solve the
problem.
A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded). Note that
Open MPI stopped checking at the first component that it did not find.
Host: openpower
Framework: pml
Component: pami
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum packages to install MPI, so It's
quite strange that there are some components missing (pami). Any help?
Thanks
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati [AT] cineca.it

Chuck Schneider

2017-05-18 13:32:51 UTC

Permalink

Another thing Gabriele might want to check is whether the Spectrum MPI
launcher is being used. In Spectrum MPI, "mpirun" is a script that, among
other things, sets up the environment for Spectrum-specific libraries to be found.

In my experience, trying to "short circuit" the use of that script - for
example, by using the orterun binary - can lead to problems similar to what
Gabriele is experiencing.

My suggestion would be to run file(1) on the launcher to make sure it is the
Spectrum script. For example, if the command used to launch is simply
"mpirun", then a suitable check would be:

file `which mpirun`

Regards,

Chuck Schneider | Software Engineer
Rogue Wave Software, Inc.
Innovate with Confidence
P 508.652.7718 | F 508.652.7701

Post by John Hearns via users
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your environment pointing to an other Open
MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI 10.1.0 based
on OpenMPI, so I hope there are some MPI expert can help me to solve the
problem.
A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded). Note that
Open MPI stopped checking at the first component that it did not find.
Host: openpower
Framework: pml
Component: pami
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum packages to install MPI, so It's
quite strange that there are some components missing (pami). Any help? Thanks
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-18 13:41:17 UTC

Permalink

Hi Gilles, attached the requested info

2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet <

Post by Gabriele Fatigati
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI 10.1.0 based
on OpenMPI, so I hope there are some MPI expert can help me to solve the
problem.
A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded). Note that
Open MPI stopped checking at the first component that it did not find.
Host: openpower
Framework: pml
Component: pami
------------------------------------------------------------
--------------
------------------------------------------------------------
--------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of "Success" (0)
------------------------------------------------------------
--------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum packages to install MPI, so It's
quite strange that there are some components missing (pami). Any help?
Thanks
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
<051%20617%201722>
g.fatigati [AT] cineca.it

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-19 06:41:42 UTC

Permalink

Hi Gilles, using your command:

[openpower:88536] mca: base: components_register: registering framework pml
components
[openpower:88536] mca: base: components_register: found loaded component
pami
[openpower:88536] mca: base: components_register: component pami register
function successful
[openpower:88536] mca: base: components_open: opening pml components
[openpower:88536] mca: base: components_open: found loaded component pami
[openpower:88536] mca: base: components_open: component pami open function
successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure for component pami
[openpower:88536] PML pami cannot be selected
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed componnets can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

Host: openpower
Framework: pml
--------------------------------------------------------------------------

Post by John Hearns via users
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles

Post by Gabriele Fatigati
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet <
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope there are some MPI expert
can help me to solve the problem.
When I run a simple Hello World MPI program, I get the follow
A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).
Note that
Open MPI stopped checking at the first component that it did not find.
Host: openpower
Framework: pml
Component: pami
------------------------------------------------------------
--------------
------------------------------------------------------------
--------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel
process can
fail during MPI_INIT; some of which are due to configuration
or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of "Success" (0)
------------------------------------------------------------
--------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum packages to install
MPI, so It's quite strange that there are some components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39
051 6171722 <tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

John Hearns via users

2017-05-19 06:50:28 UTC

Permalink

Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical interface cards in these
systems, but you do not have the correct drivers or libraries loaded.

I have had similar messages when using Infiniband on x86 systems - which
did not have libibverbs installed.

Post by Gabriele Fatigati
[openpower:88536] mca: base: components_register: registering framework
pml components
[openpower:88536] mca: base: components_register: found loaded component
pami
[openpower:88536] mca: base: components_register: component pami register
function successful
[openpower:88536] mca: base: components_open: opening pml components
[openpower:88536] mca: base: components_open: found loaded component pami
[openpower:88536] mca: base: components_open: component pami open function
successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure for component pami
[openpower:88536] PML pami cannot be selected
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were
installed, or none of the installed componnets can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
--------------------------------------------------------------------------

Post by Gabriele Fatigati
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet <
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope there are some MPI expert
can help me to solve the problem.
When I run a simple Hello World MPI program, I get the follow
A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).
Note that
Open MPI stopped checking at the first component that it did not find.
Host: openpower
Framework: pml
Component: pami
------------------------------------------------------------
--------------
------------------------------------------------------------
--------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel
process can
fail during MPI_INIT; some of which are due to configuration
or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of "Success" (0)
------------------------------------------------------------
--------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum packages to install
MPI, so It's quite strange that there are some components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39
051 6171722 <tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-19 07:01:06 UTC

Permalink

Hi John,
Infiniband is not used, there is a single node on this machine.

Post by John Hearns via users
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical interface cards in these
systems, but you do not have the correct drivers or libraries loaded.
I have had similar messages when using Infiniband on x86 systems - which
did not have libibverbs installed.

Post by Gabriele Fatigati
[openpower:88536] mca: base: components_register: registering framework
pml components
[openpower:88536] mca: base: components_register: found loaded component
pami
[openpower:88536] mca: base: components_register: component pami register
function successful
[openpower:88536] mca: base: components_open: opening pml components
[openpower:88536] mca: base: components_open: found loaded component pami
[openpower:88536] mca: base: components_open: component pami open
function successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure for component pami
[openpower:88536] PML pami cannot be selected
------------------------------------------------------------
--------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were
installed, or none of the installed componnets can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
------------------------------------------------------------
--------------

Post by Gabriele Fatigati
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet <
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
Dear OpenMPI users and developers, I'm using IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope there are some MPI expert
can help me to solve the problem.
When I run a simple Hello World MPI program, I get the follow
A requested component was not found, or was unable to be
opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded).
Note that
Open MPI stopped checking at the first component that it did
not find.
Host: openpower
Framework: pml
Component: pami
------------------------------------------------------------
--------------
------------------------------------------------------------
--------------
It looks like MPI_INIT failed for some reason; your parallel
process is
likely to abort. There are many reasons that a parallel
process can
fail during MPI_INIT; some of which are due to configuration
or environment
problems. This failure appears to be an internal failure;
here's some
additional information (which may only be relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of "Success" (0)
------------------------------------------------------------
--------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum packages to install
MPI, so It's quite strange that there are some components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39
051 6171722 <tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-19 07:07:57 UTC

Permalink

---------- Forwarded message ----------
From: Gabriele Fatigati <***@cineca.it>
Date: 2017-05-19 9:07 GMT+02:00
Subject: Re: [OMPI users] IBM Spectrum MPI problem
To: John Hearns <***@googlemail.com>

If I understand well, when I launch mpirun by default try to use
Infiniband, but because there are no infiniband module the run fails?
--
Ing. Gabriele Fatigati

HPC specialist

SuperComputing Applications and Innovation Department

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.it Tel: +39 051 6171722 <051%20617%201722>

g.fatigati [AT] cineca.it
--
Ing. Gabriele Fatigati

HPC specialist

SuperComputing Applications and Innovation Department

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.it Tel: +39 051 6171722

g.fatigati [AT] cineca.it

Gilles Gouaillardet

2017-05-19 07:10:30 UTC

Permalink

Gabriele,

so it seems pml/pami assumes there is an infiniband card available (!)

i guess IBM folks will comment on that shortly.

meanwhile, you do not need pami since you are running on a single node

mpirun --mca pml ^pami ...

should do the trick

(if it does not work, can run and post the logs)

mpirun --mca pml ^pami --mca pml_base_verbose 100 ...

Cheers,

Gilles

Post by Gabriele Fatigati
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on x86 systems -
which did not have libibverbs installed.
[openpower:88536] mca: base: components_register: registering
framework pml components
[openpower:88536] mca: base: components_register: found loaded
component pami
[openpower:88536] mca: base: components_register: component
pami register function successful
[openpower:88536] mca: base: components_open: opening pml components
[openpower:88536] mca: base: components_open: found loaded component pami
[openpower:88536] mca: base: components_open: component pami
open function successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure for component pami
[openpower:88536] PML pami cannot be selected
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were
installed, or none of the installed componnets can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
--------------------------------------------------------------------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers, I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World MPI program, I
get the follow
A requested component was not found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g., sometimes this
means that shared
libraries
that the component requires are unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason;
your parallel
process is
likely to abort. There are many reasons that a
parallel
process can
fail during MPI_INIT; some of which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of
"Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%206171722> <tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
+39 051 6171722 <tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

John Hearns via users

2017-05-19 07:16:52 UTC

Permalink

Gabriele,
as Gilles says if you are running within a single host system, you don not
need the pami layer.
Usually you would use the btls sm,self though I guess 'vader' is the
more up to date choice

Post by John Hearns via users
Gabriele,
so it seems pml/pami assumes there is an infiniband card available (!)
i guess IBM folks will comment on that shortly.
meanwhile, you do not need pami since you are running on a single node
mpirun --mca pml ^pami ...
should do the trick
(if it does not work, can run and post the logs)
mpirun --mca pml ^pami --mca pml_base_verbose 100 ...
Cheers,
Gilles

Post by Gabriele Fatigati
Hi John,
Infiniband is not used, there is a single node on this machine.
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on x86 systems -
which did not have libibverbs installed.
[openpower:88536] mca: base: components_register: registering
framework pml components
[openpower:88536] mca: base: components_register: found loaded
component pami
[openpower:88536] mca: base: components_register: component
pami register function successful
[openpower:88536] mca: base: components_open: opening pml components
[openpower:88536] mca: base: components_open: found loaded component pami
[openpower:88536] mca: base: components_open: component pami
open function successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure for component pami
[openpower:88536] PML pami cannot be selected
------------------------------------------------------------
--------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were
installed, or none of the installed componnets can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
------------------------------------------------------------
--------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers, I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World MPI program, I
get the follow
A requested component was not found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g., sometimes this
means that shared
libraries
that the component requires are unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
------------------------------
--------------------------------------------
------------------------------
--------------------------------------------
It looks like MPI_INIT failed for some reason;
your parallel
process is
likely to abort. There are many reasons that a
parallel
process can
fail during MPI_INIT; some of which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of
"Success" (0)
------------------------------
--------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%206171722> <tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortiu
m.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortiu
m.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
+39 051 6171722 <tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-19 07:28:32 UTC

Permalink

Using:

mpirun --mca pml ^pami --mca pml_base_verbose 100 -n 2 ./prova_mpi

I attach the output

Post by John Hearns via users
Gabriele,
as Gilles says if you are running within a single host system, you don not
need the pami layer.
Usually you would use the btls sm,self though I guess 'vader' is the
more up to date choice

Post by Gabriele Fatigati
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users <
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on x86 systems -
which did not have libibverbs installed.
[openpower:88536] mca: base: components_register: registering
framework pml components
[openpower:88536] mca: base: components_register: found loaded
component pami
[openpower:88536] mca: base: components_register: component
pami register function successful
[openpower:88536] mca: base: components_open: opening pml components
[openpower:88536] mca: base: components_open: found loaded
component pami
[openpower:88536] mca: base: components_open: component pami
open function successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure for component pami
[openpower:88536] PML pami cannot be selected
------------------------------------------------------------
--------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were
installed, or none of the installed componnets can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
------------------------------------------------------------
--------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers, I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World MPI program, I
get the follow
A requested component was not found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g., sometimes this
means that shared
libraries
that the component requires are unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
------------------------------
--------------------------------------------
------------------------------
--------------------------------------------
It looks like MPI_INIT failed for some reason;
your parallel
process is
likely to abort. There are many reasons that a
parallel
process can
fail during MPI_INIT; some of which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of
"Success" (0)
------------------------------
--------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <051%20617%201722> <tel:051%206171722>
<tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortiu
m.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
rg>
https://rfd.newmexicoconsortiu
m.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
+39 051 6171722 <tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gilles Gouaillardet

2017-05-19 08:05:09 UTC

Permalink

Gabriele,

i am sorry, i really meant

mpirun --mca pml ^pami --mca btl_base_verbose 100 ...

Cheers,

Gilles

Post by Gabriele Fatigati
mpirun --mca pml ^pami --mca pml_base_verbose 100 -n 2 ./prova_mpi
I attach the output
2017-05-19 9:16 GMT+02:00 John Hearns via users
Gabriele,
as Gilles says if you are running within a single host system, you
don not need the pami layer.
Usually you would use the btls sm,self though I guess 'vader'
is the more up to date choice
Gabriele,
so it seems pml/pami assumes there is an infiniband card available (!)
i guess IBM folks will comment on that shortly.
meanwhile, you do not need pami since you are running on a single node
mpirun --mca pml ^pami ...
should do the trick
(if it does not work, can run and post the logs)
mpirun --mca pml ^pami --mca pml_base_verbose 100 ...
Cheers,
Gilles
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical
interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on
x86 systems -
which did not have libibverbs installed.
On 19 May 2017 at 08:41, Gabriele Fatigati
registering
framework pml components
found loaded
component pami
[openpower:88536] mca: base: components_register: component
pami register function successful
opening pml
components
[openpower:88536] mca: base: components_open: found loaded
component pami
component pami
open function successful
[openpower:88536] select: initializing pml
component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure
for component pami
[openpower:88536] PML pami cannot be selected
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.
This typically means that either no components of
this type were
installed, or none of the installed componnets can
be loaded.
Sometimes this means that shared libraries
required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
--------------------------------------------------------------------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers,
I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope
there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World
MPI program, I
get the follow
A requested component was not found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g.,
sometimes this
means that shared
libraries
that the component requires are
unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for
some reason;
your parallel
process is
likely to abort. There are many
reasons that a
parallel
process can
fail during MPI_INIT; some of
which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of
"Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL
(processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that
there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di
Reno (BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%20617%201722> <tel:051%206171722>
<tel:051%20617%201722>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051
6171722 <tel:%2B39%20051%206171722>
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
+39 051 6171722 <tel:%2B39%20051%206171722>
<tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-19 09:09:10 UTC

Permalink

Ok Gilles, the output of

mpirun --mca pml ^pami --mca btl_base_verbose 100

is in attached

Post by John Hearns via users
Gabriele,
i am sorry, i really meant
mpirun --mca pml ^pami --mca btl_base_verbose 100 ...
Cheers,
Gilles

Post by Gabriele Fatigati
mpirun --mca pml ^pami --mca pml_base_verbose 100 -n 2 ./prova_mpi
I attach the output
Gabriele,
as Gilles says if you are running within a single host system, you
don not need the pami layer.
Usually you would use the btls sm,self though I guess 'vader'
is the more up to date choice
Gabriele,
so it seems pml/pami assumes there is an infiniband card available (!)
i guess IBM folks will comment on that shortly.
meanwhile, you do not need pami since you are running on a single node
mpirun --mca pml ^pami ...
should do the trick
(if it does not work, can run and post the logs)
mpirun --mca pml ^pami --mca pml_base_verbose 100 ...
Cheers,
Gilles
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical
interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on
x86 systems -
which did not have libibverbs installed.
On 19 May 2017 at 08:41, Gabriele Fatigati
registering
framework pml components
found loaded
component pami
component
pami register function successful
opening pml
components
found loaded
component pami
component pami
open function successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure
for component pami
[openpower:88536] PML pami cannot be selected
-----------------------------
---------------------------------------------
No components were able to be opened in the pml
framework.
This typically means that either no components of
this type were
installed, or none of the installed componnets can
be loaded.
Sometimes this means that shared libraries
required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
-----------------------------
---------------------------------------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers,
I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope
there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World
MPI program, I
get the follow
A requested component was not
found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g., sometimes this
means that shared
libraries
that the component requires are
unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
------------------------------------------------------------
--------------
------------------------------------------------------------
--------------
It looks like MPI_INIT failed for
some reason;
your parallel
process is
likely to abort. There are many
reasons that a
parallel
process can
fail during MPI_INIT; some of
which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13)
instead of
"Success" (0)
------------------------------------------------------------
--------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL
(processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that
there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and
Innovation
Department
Via Magnanelli 6/3, Casalecchio di
Reno (BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%20617%201722> <tel:051%206171722>
<tel:051%20617%201722>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
<
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno
(BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051
6171722 <tel:%2B39%20051%206171722>
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_____________________________
__________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
+39 051 6171722 <tel:%2B39%20051%206171722>
<tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-19 07:22:17 UTC

Permalink

Hi GIlles,

using your command with one MPI procs I get:

findActiveDevices Error
We found no active IB device ports
Hello world from rank 0 out of 1 processors

So it seems to work apart the error message.

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-19 07:23:27 UTC

Permalink

Oh no, by using two procs:

findActiveDevices Error
We found no active IB device ports
findActiveDevices Error
We found no active IB device ports
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.

Process 1 ([[12380,1],0]) is on host: openpower
Process 2 ([[12380,1],1]) is on host: openpower
BTLs attempted: self

Your MPI job is now going to abort; sorry.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
MPI_INIT has failed because at least one MPI process is unreachable
from another. This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used. Your MPI job will now abort.

You may wish to try to narrow down the problem;
* Check the output of ompi_info to see which BTL/MTL plugins are
available.
* Run your application with MPI_THREAD_SINGLE.
* Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
if using MTL-based communications) to see exactly which
communication plugins were considered and/or discarded.
--------------------------------------------------------------------------
[openpower:88867] 1 more process has sent help message help-mca-bml-r2.txt
/ unreachable proc
[openpower:88867] Set MCA parameter "orte_base_help_aggregate" to 0 to see
all help / error messages
[openpower:88867] 1 more process has sent help message help-mpi-runtime.txt
/ mpi_init:startup:pml-add-procs-fail

Post by Gabriele Fatigati
Hi GIlles,
findActiveDevices Error
We found no active IB device ports
Hello world from rank 0 out of 1 processors
So it seems to work apart the error message.

Post by Gabriele Fatigati
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users <
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on x86 systems -
which did not have libibverbs installed.
[openpower:88536] mca: base: components_register: registering
framework pml components
[openpower:88536] mca: base: components_register: found loaded
component pami
[openpower:88536] mca: base: components_register: component
pami register function successful
[openpower:88536] mca: base: components_open: opening pml components
[openpower:88536] mca: base: components_open: found loaded
component pami
[openpower:88536] mca: base: components_open: component pami
open function successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure for component pami
[openpower:88536] PML pami cannot be selected
------------------------------------------------------------
--------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were
installed, or none of the installed componnets can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
------------------------------------------------------------
--------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers, I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World MPI program, I
get the follow
A requested component was not found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g., sometimes this
means that shared
libraries
that the component requires are unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
------------------------------
--------------------------------------------
------------------------------
--------------------------------------------
It looks like MPI_INIT failed for some reason;
your parallel
process is
likely to abort. There are many reasons that a
parallel
process can
fail during MPI_INIT; some of which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of
"Success" (0)
------------------------------
--------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%206171722> <tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortiu
m.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
rg>
https://rfd.newmexicoconsortiu
m.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
+39 051 6171722 <tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

John Hearns via users

2017-05-19 07:41:24 UTC

Permalink

BTLs attempted: self

That should only allow a single process to communicate with its self

Post by Gabriele Fatigati
Hi GIlles,
findActiveDevices Error
We found no active IB device ports
Hello world from rank 0 out of 1 processors
So it seems to work apart the error message.

Post by Gabriele Fatigati
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users <
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on x86 systems -
which did not have libibverbs installed.
[openpower:88536] mca: base: components_register: registering
framework pml components
[openpower:88536] mca: base: components_register: found loaded
component pami
[openpower:88536] mca: base: components_register: component
pami register function successful
[openpower:88536] mca: base: components_open: opening pml components
[openpower:88536] mca: base: components_open: found loaded
component pami
[openpower:88536] mca: base: components_open: component pami
open function successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure for component pami
[openpower:88536] PML pami cannot be selected
------------------------------------------------------------
--------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were
installed, or none of the installed componnets can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
------------------------------------------------------------
--------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers, I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World MPI program, I
get the follow
A requested component was not found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g., sometimes this
means that shared
libraries
that the component requires are unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
------------------------------
--------------------------------------------
------------------------------
--------------------------------------------
It looks like MPI_INIT failed for some reason;
your parallel
process is
likely to abort. There are many reasons that a
parallel
process can
fail during MPI_INIT; some of which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of
"Success" (0)
------------------------------
--------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%206171722> <tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortiu
m.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
rg>
https://rfd.newmexicoconsortiu
m.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
+39 051 6171722 <tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gilles Gouaillardet

2017-05-19 07:43:13 UTC

Permalink

Gabriele,

can you

mpirun --mca btl_base_verbose 100 -np 2 ...

so we can figure out why nor sm nor vader is used ?

Cheers,

Gilles

Post by Gabriele Fatigati
findActiveDevices Error
We found no active IB device ports
findActiveDevices Error
We found no active IB device ports
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.
Process 1 ([[12380,1],0]) is on host: openpower
Process 2 ([[12380,1],1]) is on host: openpower
BTLs attempted: self
Your MPI job is now going to abort; sorry.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
MPI_INIT has failed because at least one MPI process is unreachable
from another. This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used. Your MPI job will now abort.
You may wish to try to narrow down the problem;
* Check the output of ompi_info to see which BTL/MTL plugins are
available.
* Run your application with MPI_THREAD_SINGLE.
* Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
if using MTL-based communications) to see exactly which
communication plugins were considered and/or discarded.
--------------------------------------------------------------------------
[openpower:88867] 1 more process has sent help message
help-mca-bml-r2.txt / unreachable proc
[openpower:88867] Set MCA parameter "orte_base_help_aggregate" to 0 to
see all help / error messages
[openpower:88867] 1 more process has sent help message
help-mpi-runtime.txt / mpi_init:startup:pml-add-procs-fail
Hi GIlles,
findActiveDevices Error
We found no active IB device ports
Hello world from rank 0 out of 1 processors
So it seems to work apart the error message.
Gabriele,
so it seems pml/pami assumes there is an infiniband card available (!)
i guess IBM folks will comment on that shortly.
meanwhile, you do not need pami since you are running on a single node
mpirun --mca pml ^pami ...
should do the trick
(if it does not work, can run and post the logs)
mpirun --mca pml ^pami --mca pml_base_verbose 100 ...
Cheers,
Gilles
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical
interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on
x86 systems -
which did not have libibverbs installed.
On 19 May 2017 at 08:41, Gabriele Fatigati
registering
framework pml components
found loaded
component pami
[openpower:88536] mca: base: components_register: component
pami register function successful
opening pml
components
[openpower:88536] mca: base: components_open: found loaded
component pami
component pami
open function successful
[openpower:88536] select: initializing pml
component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure
for component pami
[openpower:88536] PML pami cannot be selected
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.
This typically means that either no components of
this type were
installed, or none of the installed componnets can
be loaded.
Sometimes this means that shared libraries
required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
--------------------------------------------------------------------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers,
I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope
there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World
MPI program, I
get the follow
A requested component was not found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g.,
sometimes this
means that shared
libraries
that the component requires are
unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for
some reason;
your parallel
process is
likely to abort. There are many
reasons that a
parallel
process can
fail during MPI_INIT; some of
which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of
"Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL
(processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that
there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di
Reno (BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%206171722> <tel:051%206171722>
<tel:051%20617%201722>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051
6171722 <tel:%2B39%20051%206171722>
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
+39 051 6171722 <tel:%2B39%20051%206171722>
<tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
<tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Gabriele Fatigati

2017-05-19 08:16:17 UTC

Permalink

Hi Gilles, in attach the outpuf of:

mpirun --mca btl_base_verbose 100 -np 2 ...

Post by John Hearns via users
Gabriele,
can you
mpirun --mca btl_base_verbose 100 -np 2 ...
so we can figure out why nor sm nor vader is used ?
Cheers,
Gilles

Post by Gabriele Fatigati
findActiveDevices Error
We found no active IB device ports
findActiveDevices Error
We found no active IB device ports
------------------------------------------------------------
--------------
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.
Process 1 ([[12380,1],0]) is on host: openpower
Process 2 ([[12380,1],1]) is on host: openpower
BTLs attempted: self
Your MPI job is now going to abort; sorry.
------------------------------------------------------------
--------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
------------------------------------------------------------
--------------
MPI_INIT has failed because at least one MPI process is unreachable
from another. This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used. Your MPI job will now abort.
You may wish to try to narrow down the problem;
* Check the output of ompi_info to see which BTL/MTL plugins are
available.
* Run your application with MPI_THREAD_SINGLE.
* Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
if using MTL-based communications) to see exactly which
communication plugins were considered and/or discarded.
------------------------------------------------------------
--------------
[openpower:88867] 1 more process has sent help message
help-mca-bml-r2.txt / unreachable proc
[openpower:88867] Set MCA parameter "orte_base_help_aggregate" to 0 to
see all help / error messages
[openpower:88867] 1 more process has sent help message
help-mpi-runtime.txt / mpi_init:startup:pml-add-procs-fail
Hi GIlles,
findActiveDevices Error
We found no active IB device ports
Hello world from rank 0 out of 1 processors
So it seems to work apart the error message.
Gabriele,
so it seems pml/pami assumes there is an infiniband card available (!)
i guess IBM folks will comment on that shortly.
meanwhile, you do not need pami since you are running on a single node
mpirun --mca pml ^pami ...
should do the trick
(if it does not work, can run and post the logs)
mpirun --mca pml ^pami --mca pml_base_verbose 100 ...
Cheers,
Gilles
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical
interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on
x86 systems -
which did not have libibverbs installed.
On 19 May 2017 at 08:41, Gabriele Fatigati
registering
framework pml components
found loaded
component pami
component
pami register function successful
opening pml
components
found loaded
component pami
component pami
open function successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure
for component pami
[openpower:88536] PML pami cannot be selected
-----------------------------
---------------------------------------------
No components were able to be opened in the pml
framework.
This typically means that either no components of
this type were
installed, or none of the installed componnets can
be loaded.
Sometimes this means that shared libraries
required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
-----------------------------
---------------------------------------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers,
I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope
there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World
MPI program, I
get the follow
A requested component was not
found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g., sometimes this
means that shared
libraries
that the component requires are
unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
------------------------------------------------------------
--------------
------------------------------------------------------------
--------------
It looks like MPI_INIT failed for
some reason;
your parallel
process is
likely to abort. There are many
reasons that a
parallel
process can
fail during MPI_INIT; some of
which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13)
instead of
"Success" (0)
------------------------------------------------------------
--------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL
(processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that
there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and
Innovation
Department
Via Magnanelli 6/3, Casalecchio di
Reno (BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%206171722> <tel:051%206171722>
<tel:051%20617%201722>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
<
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno
(BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051
6171722 <tel:%2B39%20051%206171722>
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_____________________________
__________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
+39 051 6171722 <tel:%2B39%20051%206171722>
<tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
<tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

r***@open-mpi.org

2017-05-19 09:01:02 UTC

Permalink

If I might interject here before lots of time is wasted. Spectrum MPI is an IBM -product- and is not free. What you are likely running into is that their license manager is blocking you from running, albeit without a really nice error message. Iâm sure thatâs something they are working on.

If you really want to use Spectrum MPI, I suggest you contact them about purchasing it.

Post by Gilles Gouaillardet
mpirun --mca btl_base_verbose 100 -np 2 ...
Gabriele,
can you
mpirun --mca btl_base_verbose 100 -np 2 ...
so we can figure out why nor sm nor vader is used ?
Cheers,
Gilles
findActiveDevices Error
We found no active IB device ports
findActiveDevices Error
We found no active IB device ports
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.
Process 1 ([[12380,1],0]) is on host: openpower
Process 2 ([[12380,1],1]) is on host: openpower
BTLs attempted: self
Your MPI job is now going to abort; sorry.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
MPI_INIT has failed because at least one MPI process is unreachable
from another. This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used. Your MPI job will now abort.
You may wish to try to narrow down the problem;
* Check the output of ompi_info to see which BTL/MTL plugins are
available.
* Run your application with MPI_THREAD_SINGLE.
* Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
if using MTL-based communications) to see exactly which
communication plugins were considered and/or discarded.
--------------------------------------------------------------------------
[openpower:88867] 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc
[openpower:88867] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[openpower:88867] 1 more process has sent help message help-mpi-runtime.txt / mpi_init:startup:pml-add-procs-fail
Hi GIlles,
findActiveDevices Error
We found no active IB device ports
Hello world from rank 0 out of 1 processors
So it seems to work apart the error message.
Gabriele,
so it seems pml/pami assumes there is an infiniband card available (!)
i guess IBM folks will comment on that shortly.
meanwhile, you do not need pami since you are running on a single node
mpirun --mca pml ^pami ...
should do the trick
(if it does not work, can run and post the logs)
mpirun --mca pml ^pami --mca pml_base_verbose 100 ...
Cheers,
Gilles
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical
interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on
x86 systems -
which did not have libibverbs installed.
On 19 May 2017 at 08:41, Gabriele Fatigati
registering
framework pml components
found loaded
component pami
[openpower:88536] mca: base: components_register: component
pami register function successful
opening pml
components
[openpower:88536] mca: base: components_open: found loaded
component pami
component pami
open function successful
[openpower:88536] select: initializing pml
component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure
for component pami
[openpower:88536] PML pami cannot be selected
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.
This typically means that either no components of
this type were
installed, or none of the installed componnets can
be loaded.
Sometimes this means that shared libraries
required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
--------------------------------------------------------------------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers,
I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope
there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World
MPI program, I
get the follow
A requested component was not found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g.,
sometimes this
means that shared
libraries
that the component requires are
unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for
some reason;
your parallel
process is
likely to abort. There are many
reasons that a
parallel
process can
fail during MPI_INIT; some of
which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of
"Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL
(processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that
there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di
Reno (BO) Italy
www.cineca.it <http://www.cineca.it/> <http://www.cineca.it <http://www.cineca.it/>> <http://www.cineca.it <http://www.cineca.it/>>
<http://www.cineca.it <http://www.cineca.it/>> Tel: +39
051 6171722 <tel:051%206171722> <tel:051%206171722> <tel:051%206171722>
<tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it/>
<http://cineca.it <http://cineca.it/>> <http://cineca.it <http://cineca.it/>>
<http://cineca.it <http://cineca.it/>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it/> <http://www.cineca.it <http://www.cineca.it/>> <http://www.cineca.it <http://www.cineca.it/>>
<http://www.cineca.it <http://www.cineca.it/>> Tel: +39 051
6171722 <tel:%2B39%20051%206171722>
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it/>
<http://cineca.it <http://cineca.it/>> <http://cineca.it <http://cineca.it/>>
<http://cineca.it <http://cineca.it/>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it/> <http://www.cineca.it <http://www.cineca.it/>>
+39 051 6171722 <tel:%2B39%20051%206171722> <tel:%2B39%20051%206171722>
<tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it/> <http://cineca.it <http://cineca.it/>>
<http://cineca.it <http://cineca.it/>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it/> <http://www.cineca.it <http://www.cineca.it/>>
<http://www.cineca.it <http://www.cineca.it/>> Tel: +39 051 6171722 <tel:%2B39%20051%206171722>
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it/> <http://cineca.it <http://cineca.it/>>
<http://cineca.it <http://cineca.it/>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it/> <http://www.cineca.it <http://www.cineca.it/>> Tel: +39 051 6171722 <tel:%2B39%20051%206171722>
<tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it/> <http://cineca.it <http://cineca.it/>>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it/> <http://www.cineca.it <http://www.cineca.it/>> Tel: +39 051 6171722 <tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it/> <http://cineca.it <http://cineca.it/>>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it/> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it/>
<output_mpirun>_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

John Hearns via users

2017-05-19 09:12:06 UTC

Permalink

I am not sure I agree with that.
(a) the original error message from Gabriele was quite clear - the MPI
could not find an interface card which was up, so it would not run.
(b) Nysal actually pointed out the solution which looks good - after
reaidng the documentation.. use pami_noib
(c) Having discussions like this helps us all to learn. I have made many
stupid replies on this list, and looking at problems like this has helped
me to learn.

Post by r***@open-mpi.org
If I might interject here before lots of time is wasted. Spectrum MPI is
an IBM -product- and is not free. What you are likely running into is that
their license manager is blocking you from running, albeit without a really
nice error message. Iâm sure thatâs something they are working on.
If you really want to use Spectrum MPI, I suggest you contact them about purchasing it.
mpirun --mca btl_base_verbose 100 -np 2 ...

Post by John Hearns via users
Gabriele,
can you
mpirun --mca btl_base_verbose 100 -np 2 ...
so we can figure out why nor sm nor vader is used ?
Cheers,
Gilles

Post by Gabriele Fatigati
findActiveDevices Error
We found no active IB device ports
findActiveDevices Error
We found no active IB device ports
------------------------------------------------------------
--------------
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.
Process 1 ([[12380,1],0]) is on host: openpower
Process 2 ([[12380,1],1]) is on host: openpower
BTLs attempted: self
Your MPI job is now going to abort; sorry.
------------------------------------------------------------
--------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
------------------------------------------------------------
--------------
MPI_INIT has failed because at least one MPI process is unreachable
from another. This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used. Your MPI job will now abort.
You may wish to try to narrow down the problem;
* Check the output of ompi_info to see which BTL/MTL plugins are
available.
* Run your application with MPI_THREAD_SINGLE.
* Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
if using MTL-based communications) to see exactly which
communication plugins were considered and/or discarded.
------------------------------------------------------------
--------------
[openpower:88867] 1 more process has sent help message
help-mca-bml-r2.txt / unreachable proc
[openpower:88867] Set MCA parameter "orte_base_help_aggregate" to 0 to
see all help / error messages
[openpower:88867] 1 more process has sent help message
help-mpi-runtime.txt / mpi_init:startup:pml-add-procs-fail
Hi GIlles,
findActiveDevices Error
We found no active IB device ports
Hello world from rank 0 out of 1 processors
So it seems to work apart the error message.
Gabriele,
so it seems pml/pami assumes there is an infiniband card available (!)
i guess IBM folks will comment on that shortly.
meanwhile, you do not need pami since you are running on a single node
mpirun --mca pml ^pami ...
should do the trick
(if it does not work, can run and post the logs)
mpirun --mca pml ^pami --mca pml_base_verbose 100 ...
Cheers,
Gilles
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical
interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on
x86 systems -
which did not have libibverbs installed.
On 19 May 2017 at 08:41, Gabriele Fatigati
registering
framework pml components
found loaded
component pami
component
pami register function successful
opening pml
components
found loaded
component pami
component pami
open function successful
[openpower:88536] select: initializing pml
component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure
for component pami
[openpower:88536] PML pami cannot be selected
-----------------------------
---------------------------------------------
No components were able to be opened in the pml
framework.
This typically means that either no components of
this type were
installed, or none of the installed componnets can
be loaded.
Sometimes this means that shared libraries
required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
-----------------------------
---------------------------------------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele
Fatigati
Dear OpenMPI users and developers,
I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope
there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World
MPI program, I
get the follow
A requested component was not
found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g.,
sometimes this
means that shared
libraries
that the component requires are
unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
------------------------------------------------------------
--------------
------------------------------------------------------------
--------------
It looks like MPI_INIT failed for
some reason;
your parallel
process is
likely to abort. There are many
reasons that a
parallel
process can
fail during MPI_INIT; some of
which are due to
configuration
or environment
problems. This failure appears to
be an
internal failure;
here's some
additional information (which may
only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13)
instead of
"Success" (0)
------------------------------------------------------------
--------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL
(processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that
there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and
Innovation
Department
Via Magnanelli 6/3, Casalecchio di
Reno (BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%206171722 <051%206171722>> <
tel:051%206171722 <051%206171722>>
<tel:051%20617%201722 <051%20617%201722>>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno
(BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051
6171722 <tel:%2B39%20051%206171722 <%2B39%20051%206171722>>
<tel:%2B39%20051%206171722 <%2B39%20051%206171722>>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_____________________________
__________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
+39 051 6171722 <tel:%2B39%20051%206171722
<%2B39%20051%206171722>>
<tel:+39%20051%20617%201722 <+39%20051%20617%201722>>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722 <%2B39%20051%206171722>>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
<tel:051%20617%201722 <051%20617%201722>>
g.fatigati [AT] cineca.it <http://cineca.it>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
<+39%20051%20617%201722>
g.fatigati [AT] cineca.it
<output_mpirun>_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

John Hearns via users

2017-05-19 09:17:21 UTC

Permalink

This post might be inappropriate. Click to display it.

Gabriele Fatigati

2017-05-19 09:40:31 UTC

Permalink

Yes, using " -pami_noib" solve the problem, I lost the previous message.
Thanks you so much for the support.

Post by John Hearns via users
I am not sure I agree with that.
(a) the original error message from Gabriele was quite clear - the MPI
could not find an interface card which was up, so it would not run.
(b) Nysal actually pointed out the solution which looks good - after
reaidng the documentation.. use pami_noib
(c) Having discussions like this helps us all to learn. I have made many
stupid replies on this list, and looking at problems like this has helped
me to learn.

Post by John Hearns via users
Gabriele,
can you
mpirun --mca btl_base_verbose 100 -np 2 ...
so we can figure out why nor sm nor vader is used ?
Cheers,
Gilles

Post by Gabriele Fatigati
findActiveDevices Error
We found no active IB device ports
findActiveDevices Error
We found no active IB device ports
------------------------------------------------------------
--------------
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.
Process 1 ([[12380,1],0]) is on host: openpower
Process 2 ([[12380,1],1]) is on host: openpower
BTLs attempted: self
Your MPI job is now going to abort; sorry.
------------------------------------------------------------
--------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
------------------------------------------------------------
--------------
MPI_INIT has failed because at least one MPI process is unreachable
from another. This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used. Your MPI job will now abort.
You may wish to try to narrow down the problem;
* Check the output of ompi_info to see which BTL/MTL plugins are
available.
* Run your application with MPI_THREAD_SINGLE.
* Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
if using MTL-based communications) to see exactly which
communication plugins were considered and/or discarded.
------------------------------------------------------------
--------------
[openpower:88867] 1 more process has sent help message
help-mca-bml-r2.txt / unreachable proc
[openpower:88867] Set MCA parameter "orte_base_help_aggregate" to 0 to
see all help / error messages
[openpower:88867] 1 more process has sent help message
help-mpi-runtime.txt / mpi_init:startup:pml-add-procs-fail
Hi GIlles,
findActiveDevices Error
We found no active IB device ports
Hello world from rank 0 out of 1 processors
So it seems to work apart the error message.
Gabriele,
so it seems pml/pami assumes there is an infiniband card
available (!)
i guess IBM folks will comment on that shortly.
meanwhile, you do not need pami since you are running on a
single node
mpirun --mca pml ^pami ...
should do the trick
(if it does not work, can run and post the logs)
mpirun --mca pml ^pami --mca pml_base_verbose 100 ...
Cheers,
Gilles
Hi John,
Infiniband is not used, there is a single node on this
machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical
interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on
x86 systems -
which did not have libibverbs installed.
On 19 May 2017 at 08:41, Gabriele Fatigati
registering
framework pml components
found loaded
component pami
component
pami register function successful
opening pml
components
found loaded
component pami
component pami
open function successful
[openpower:88536] select: initializing pml
component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure
for component pami
[openpower:88536] PML pami cannot be selected
-----------------------------
---------------------------------------------
No components were able to be opened in the pml
framework.
This typically means that either no components of
this type were
installed, or none of the installed componnets can
be loaded.
Sometimes this means that shared libraries
required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
-----------------------------
---------------------------------------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele
Fatigati
Dear OpenMPI users and developers,
I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope
there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World
MPI program, I
get the follow
A requested component was not
found, or was
unable to be
opened. This
means that this component is
either not
installed or is unable
to be
used on your system (e.g.,
sometimes this
means that shared
libraries
that the component requires are
unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
------------------------------
--------------------------------------------
------------------------------
--------------------------------------------
It looks like MPI_INIT failed for
some reason;
your parallel
process is
likely to abort. There are many
reasons that a
parallel
process can
fail during MPI_INIT; some of
which are due to
configuration
or environment
problems. This failure appears to
be an
internal failure;
here's some
additional information (which may
only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13)
instead of
"Success" (0)
------------------------------
--------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL
(processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that
there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and
Innovation
Department
Via Magnanelli 6/3, Casalecchio di
Reno (BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%206171722 <051%206171722>> <
tel:051%206171722 <051%206171722>>
<tel:051%20617%201722 <051%20617%201722>>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno
(BO) Italy
www.cineca.it <http://www.cineca.it> <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051
6171722 <tel:%2B39%20051%206171722 <%2B39%20051%206171722>>
<tel:%2B39%20051%206171722 <%2B39%20051%206171722>>
g.fatigati [AT] cineca.it
<http://cineca.it> <http://cineca.it>
<http://cineca.it>
_____________________________
__________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
+39 051 6171722 <tel:%2B39%20051%206171722
<%2B39%20051%206171722>>
<tel:+39%20051%20617%201722 <+39%20051%20617%201722>>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722 <%2B39%20051%206171722>>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
<tel:051%20617%201722 <051%20617%201722>>
g.fatigati [AT] cineca.it <http://cineca.it>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
<+39%20051%20617%201722>
g.fatigati [AT] cineca.it
<output_mpirun>_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Nathan Hjelm

2017-05-19 07:52:02 UTC

Permalink

Add —mca btl self,vader

-Nathan

Post by Gabriele Fatigati
findActiveDevices Error
We found no active IB device ports
findActiveDevices Error
We found no active IB device ports
--------------------------------------------------------------------------
At least one pair of MPI processes are unable to reach each other for
MPI communications. This means that no Open MPI device has indicated
that it can be used to communicate between these processes. This is
an error; Open MPI requires that all MPI processes be able to reach
each other. This error can sometimes be the result of forgetting to
specify the "self" BTL.
Process 1 ([[12380,1],0]) is on host: openpower
Process 2 ([[12380,1],1]) is on host: openpower
BTLs attempted: self
Your MPI job is now going to abort; sorry.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
--------------------------------------------------------------------------
MPI_INIT has failed because at least one MPI process is unreachable
from another. This *usually* means that an underlying communication
plugin -- such as a BTL or an MTL -- has either not loaded or not
allowed itself to be used. Your MPI job will now abort.
You may wish to try to narrow down the problem;
* Check the output of ompi_info to see which BTL/MTL plugins are
available.
* Run your application with MPI_THREAD_SINGLE.
* Set the MCA parameter btl_base_verbose to 100 (or mtl_base_verbose,
if using MTL-based communications) to see exactly which
communication plugins were considered and/or discarded.
--------------------------------------------------------------------------
[openpower:88867] 1 more process has sent help message help-mca-bml-r2.txt / unreachable proc
[openpower:88867] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[openpower:88867] 1 more process has sent help message help-mpi-runtime.txt / mpi_init:startup:pml-add-procs-fail
Hi GIlles,
findActiveDevices Error
We found no active IB device ports
Hello world from rank 0 out of 1 processors
So it seems to work apart the error message.
Gabriele,
so it seems pml/pami assumes there is an infiniband card available (!)
i guess IBM folks will comment on that shortly.
meanwhile, you do not need pami since you are running on a single node
mpirun --mca pml ^pami ...
should do the trick
(if it does not work, can run and post the logs)
mpirun --mca pml ^pami --mca pml_base_verbose 100 ...
Cheers,
Gilles
Hi John,
Infiniband is not used, there is a single node on this machine.
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on x86 systems -
which did not have libibverbs installed.
[openpower:88536] mca: base: components_register: registering
framework pml components
[openpower:88536] mca: base: components_register: found loaded
component pami
[openpower:88536] mca: base: components_register: component
pami register function successful
[openpower:88536] mca: base: components_open: opening pml components
[openpower:88536] mca: base: components_open: found loaded component pami
[openpower:88536] mca: base: components_open: component pami
open function successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure for component pami
[openpower:88536] PML pami cannot be selected
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were
installed, or none of the installed componnets can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
--------------------------------------------------------------------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers, I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World MPI program, I
get the follow
A requested component was not found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g., sometimes this
means that shared
libraries
that the component requires are unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason;
your parallel
process is
likely to abort. There are many reasons that a
parallel
process can
fail during MPI_INIT; some of which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of
"Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%206171722> <tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
+39 051 6171722 <tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati [AT] cineca.it
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati [AT] cineca.it
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Nysal Jan K A

2017-05-19 07:58:04 UTC

Permalink

hi Gabriele,
You can check some of the available options here -
https://www.ibm.com/support/knowledgecenter/en/SSZTET_10.1.0/smpi02/smpi02_interconnect.html
The "-pami_noib" option might be of help in this scenario. Alternatively,
on a single node, the vader BTL can also be used.

Regards
--Nysal

Post by Gabriele Fatigati
Hi GIlles,
findActiveDevices Error
We found no active IB device ports
Hello world from rank 0 out of 1 processors
So it seems to work apart the error message.

Post by Gabriele Fatigati
Hi John,
Infiniband is not used, there is a single node on this machine.
2017-05-19 8:50 GMT+02:00 John Hearns via users <
Gabriele, pleae run 'ibv_devinfo'
It looks to me like you may have the physical interface cards in
these systems, but you do not have the correct drivers or
libraries loaded.
I have had similar messages when using Infiniband on x86 systems -
which did not have libibverbs installed.
[openpower:88536] mca: base: components_register: registering
framework pml components
[openpower:88536] mca: base: components_register: found loaded
component pami
[openpower:88536] mca: base: components_register: component
pami register function successful
[openpower:88536] mca: base: components_open: opening pml components
[openpower:88536] mca: base: components_open: found loaded
component pami
[openpower:88536] mca: base: components_open: component pami
open function successful
[openpower:88536] select: initializing pml component pami
findActiveDevices Error
We found no active IB device ports
[openpower:88536] select: init returned failure for component pami
[openpower:88536] PML pami cannot be selected
------------------------------------------------------------
--------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were
installed, or none of the installed componnets can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: openpower
Framework: pml
------------------------------------------------------------
--------------
2017-05-19 7:03 GMT+02:00 Gilles Gouaillardet
Gabriele,
pml/pami is here, at least according to ompi_info
can you update your mpirun command like this
mpirun --mca pml_base_verbose 100 ..
and post the output ?
Cheers,
Gilles
Hi Gilles, attached the requested info
2017-05-18 15:04 GMT+02:00 Gilles Gouaillardet
Gabriele,
can you
ompi_info --all | grep pml
also, make sure there is nothing in your
environment pointing to
an other Open MPI install
for example
ldd a.out
should only point to IBM libraries
Cheers,
Gilles
On Thursday, May 18, 2017, Gabriele Fatigati
Dear OpenMPI users and developers, I'm using
IBM Spectrum MPI
10.1.0 based on OpenMPI, so I hope there are
some MPI expert
can help me to solve the problem.
When I run a simple Hello World MPI program, I
get the follow
A requested component was not found, or was
unable to be
opened. This
means that this component is either not
installed or is unable
to be
used on your system (e.g., sometimes this
means that shared
libraries
that the component requires are unable to be
found/loaded). Note that
Open MPI stopped checking at the first
component that it did
not find.
Host: openpower
Framework: pml
Component: pami
------------------------------
--------------------------------------------
------------------------------
--------------------------------------------
It looks like MPI_INIT failed for some reason;
your parallel
process is
likely to abort. There are many reasons that a
parallel
process can
fail during MPI_INIT; some of which are due to
configuration
or environment
problems. This failure appears to be an
internal failure;
here's some
additional information (which may only be
relevant to an Open MPI
mca_pml_base_open() failed
--> Returned "Not found" (-13) instead of
"Success" (0)
------------------------------
--------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this
communicator will
now abort,
*** and potentially your MPI job)
My sysadmin used official IBM Spectrum
packages to install
MPI, so It's quite strange that there are some
components
missing (pami). Any help? Thanks
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation
Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39
051 6171722 <tel:051%206171722> <tel:051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortiu
m.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>
<https://rfd.newmexicoconsort
ium.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it>
<http://www.cineca.it> Tel: +39 051 6171722
<tel:%2B39%20051%206171722>
g.fatigati [AT] cineca.it <http://cineca.it>
<http://cineca.it>
_______________________________________________
users mailing list
rg>
https://rfd.newmexicoconsortiu
m.org/mailman/listinfo/users
<https://rfd.newmexicoconsorti
um.org/mailman/listinfo/users>
-- Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
+39 051 6171722 <tel:+39%20051%20617%201722>
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Ing. Gabriele Fatigati
HPC specialist
SuperComputing Applications and Innovation Department
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it <http://www.cineca.it> Tel: +39 051 6171722
g.fatigati [AT] cineca.it <http://cineca.it>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users