I -think- that is correct, but you may need the verbs library as well - I honestly donât remember if the configury checks for functions in the library or not. If so, then youâll need that wherever you build OMPI, but everything else is accurate
Post by Bennet FauberRalph.
I will be building from the Master branch at github.com for testing
purposes. We are not 'supporting' Singularity container creation, but
we do hope to be able to offer some guidance, so I think we can
finesse the PMIx version, yes?
That is good to know about the verbs headers being the only thing
needed; thanks for that detail. Sometimes the library also needs to
be present.
Also very good to know that the host mpirun will start processes, as
we are using cgroups, and if the processes get started by a
non-tm-supporting MPI, they will be outside the proper cgroup.
So, just to recap, if I install from the current master at
http://github.com/open-mpi/ompi.git on the host system and within the
container, I copy the verbs headers into the container, then configure
and build OMPI within the container and ignore TM support, I should be
able to copy the container to the cluster and run it with verbs and
the system OMPI using tm.
If a user were to build without the verbs support, it would still run,
but it would fall back to non-verbs communication, so it would just be
commensurately slower.
Let me know if I've garbled things. Otherwise, wish me luck, and have
a good weekend!
Thanks, -- bennet
The embedded Singularity support hasnât made it into the OMPI 2.x release series yet, though OMPI will still work within a Singularity container anyway.
Compatibility across the container boundary is always a problem, as your examples illustrate. If the system is using one OMPI version and the container is using another, then the only concern is compatibility across the container boundary of the process-to-ORTE daemon communication. In the OMPI 2.x series and beyond, this is done with PMIx. OMPI v2.0 is based on PMIx v1.x, and so will OMPI v2.1. Thus, there is no compatibility issue there. However, that statement is _not_ true for OMPI v1.10 and earlier series.
Future OMPI versions will utilize PMIx v2 and above, which include a cross-version compatibility layer. Thus, you shouldnât have any issues mixing and matching OMPI versions from this regard.
However, your second example is a perfect illustration of where containerization can break down. If you build your container on a system that doesnât have (for example) tm and verbs installed on it, then those OMPI components will not be built. The tm component wonât matter as the system version of mpirun will be executing, and it presumably knows how to interact with Torque.
However, if you run that container on a system that has verbs, your application wonât be able to utilize the verbs support because those components were never compiled. Note that the converse is not true: if you build your container on a system that has verbs installed, you can then run it on a system that doesnât have verbs support and those components will dynamically disqualify themselves.
Remember, you only need the verbs headers to be installed - you donât have to build on a machine that actually has a verbs-supporting NIC installed (this is how the distributions get around the problem). Thus, it isnât hard to avoid this portability problem - you just need to think ahead a bit.
HTH
Ralph
Post by Bennet FauberI am wishing to follow the instructions on the Singularity web site,
http://singularity.lbl.gov/docs-hpc
to test Singularity and OMPI on our cluster. My previously normal
configure for the 1.x series looked like this.
./configure --prefix=/usr/local \
--mandir=${PREFIX}/share/man \
--with-tm --with-verbs \
--disable-dlopen --enable-shared
CC=gcc CXX=g++ FC=gfortran
I have a couple of wonderments.
First, I presume it will be best to have the same version of OMPI
inside the container as out, but how sensitive will it be to minor
versions? All 2.1.x version should be fine, but not mix 2.1.x outside
with 2.2.x inside or vice-versa (might be backward compatible but not
forward)?
Second, if someone builds OMPI inside their container on an external
system, without tm and verbs, then brings the container to our system,
will the tm and verbs be handled by the calling mpirun from the host
system, and the OMPI inside the container won't care? Will not having
those inside the container cause them to be suppressed outside?
Thanks in advance, -- bennet
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users