Discussion:
[OMPI users] PMIx + OpenMPI
Charles A Taylor
2017-08-06 10:53:57 UTC
Permalink
Anyone successfully using PMIx with OpenMPI and SLURM? I have,

1. Installed an “external” version (1.1.5) of PMIx.
2. Patched SLURM 15.08.13 with the SchedMD-provided PMIx patch (results in an mpi_pmix plugin along the lines of mpi_pmi2).
3. Built OpenMPI 2.0.1 (tried 2.0.3 as well).

However, when attempting to launch MPI apps (LAMMPS in this case), I get

[c9a-s2.ufhpc:08914] PMIX ERROR: UNREACHABLE in file src/client/pmix_client.c at line 199

This comes from,

if (PMIX_SUCCESS != (ret=usock_connect((struct sockaddr *)address, &sd))) {
PMIX_ERROR_LOG(ret);
return ret;
}
I’ve googled and looked at the archives and don’t see any other references to this error. Don’t really see much about using OpenMPI with pmix at all. I assumed the “server” side was embedded in orted or some such but maybe not.

What am I missing? Is there some server that needs to be started separately as with mpd?

Thanks,

Charlie Taylor
Research Computing
University of Florida
Charles A Taylor
2017-08-06 10:59:58 UTC
Permalink
Post by Charles A Taylor
Anyone successfully using PMIx with OpenMPI and SLURM? I have,
1. Installed an “external” version (1.1.5) of PMIx.
2. Patched SLURM 15.08.13 with the SchedMD-provided PMIx patch (results in an mpi_pmix plugin along the lines of mpi_pmi2).
3. Built OpenMPI 2.0.1 (tried 2.0.3 as well).
However, when attempting to launch MPI apps (LAMMPS in this case), I get
[c9a-s2.ufhpc:08914] PMIX ERROR: UNREACHABLE in file src/client/pmix_client.c at line 199
I should have mentioned that I’m launching with

srun —mpi=pmix …

If I launch with

srun —mpi=pmi2 ...

the app starts and runs without issue.
Gilles Gouaillardet
2017-08-06 11:43:12 UTC
Permalink
Charles,

did you build Open MPI with the external PMIx ?
iirc, Open MPI 2.0.x does not support cross version PMIx

Cheers,

Gilles
Post by Charles A Taylor
Post by Charles A Taylor
Anyone successfully using PMIx with OpenMPI and SLURM? I have,
1. Installed an “external” version (1.1.5) of PMIx.
2. Patched SLURM 15.08.13 with the SchedMD-provided PMIx patch (results in an mpi_pmix plugin along the lines of mpi_pmi2).
3. Built OpenMPI 2.0.1 (tried 2.0.3 as well).
However, when attempting to launch MPI apps (LAMMPS in this case), I get
[c9a-s2.ufhpc:08914] PMIX ERROR: UNREACHABLE in file src/client/pmix_client.c at line 199
I should have mentioned that I’m launching with
srun —mpi=pmix …
If I launch with
srun —mpi=pmi2 ...
the app starts and runs without issue.
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Charles A Taylor
2017-08-06 13:14:44 UTC
Permalink
HI Gilles,

I tried both “—with-pmix=/opt/pmix” and “—with-pmix=internal” and got the same “UNREACHABLE” error both ways. I tried the “external” first since that is what SLURM was built against.

I’m missing something simple/basic - just not sure what it is.

Thanks,

Charlie
Post by Gilles Gouaillardet
Charles,
did you build Open MPI with the external PMIx ?
iirc, Open MPI 2.0.x does not support cross version PMIx
Cheers,
Gilles
Post by Charles A Taylor
Post by Charles A Taylor
Anyone successfully using PMIx with OpenMPI and SLURM? I have,
1. Installed an “external” version (1.1.5) of PMIx.
2. Patched SLURM 15.08.13 with the SchedMD-provided PMIx patch (results in an mpi_pmix plugin along the lines of mpi_pmi2).
3. Built OpenMPI 2.0.1 (tried 2.0.3 as well).
However, when attempting to launch MPI apps (LAMMPS in this case), I get
[c9a-s2.ufhpc:08914] PMIX ERROR: UNREACHABLE in file src/client/pmix_client.c at line 199
I should have mentioned that I’m launching with
srun —mpi=pmix …
If I launch with
srun —mpi=pmi2 ...
the app starts and runs without issue.
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
dani
2017-08-06 20:07:40 UTC
Permalink
_______________________________________________
users mailing list
***@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Charles A Taylor
2017-08-07 18:09:50 UTC
Permalink
Many thanks to all who replied and especially to Artem Polyakov of Mellanox who provided a slurm-15.08.13 specific pmix patch. That patch applied and built cleanly against the 15.08.13 tarball and better yet, it works.

Regards,

Charles A. Taylor
UF Research Computing
Post by Charles A Taylor
HI Gilles,
I tried both “—with-pmix=/opt/pmix” and “—with-pmix=internal” and got the same “UNREACHABLE” error both ways. I tried the “external” first since that is what SLURM was built against.
I’m missing something simple/basic - just not sure what it is.
Thanks,
Charlie
Post by Gilles Gouaillardet
Charles,
did you build Open MPI with the external PMIx ?
iirc, Open MPI 2.0.x does not support cross version PMIx
Cheers,
Gilles
Post by Charles A Taylor
Post by Charles A Taylor
Anyone successfully using PMIx with OpenMPI and SLURM? I have,
1. Installed an “external” version (1.1.5) of PMIx.
2. Patched SLURM 15.08.13 with the SchedMD-provided PMIx patch (results in an mpi_pmix plugin along the lines of mpi_pmi2).
3. Built OpenMPI 2.0.1 (tried 2.0.3 as well).
However, when attempting to launch MPI apps (LAMMPS in this case), I get
[c9a-s2.ufhpc:08914] PMIX ERROR: UNREACHABLE in file src/client/pmix_client.c at line 199
I should have mentioned that I’m launching with
srun —mpi=pmix …
If I launch with
srun —mpi=pmi2 ...
the app starts and runs without issue.
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Loading...