[OMPI users] MCA compilation later

[Sean Ahern]

There's been discussion on the OpenMPI list recently about static linking
of OpenMPI with all of the desired MCAs in it. I've got the opposite
question. I'd like to add MCAs later on to an already-compiled version of
OpenMPI and am not quite sure how to do it.

Let me summarize. We've got a commercial code that we deploy on customer
machines in binary form. We're working to integrate OpenMPI into the
installer, and things seem to be progressing well. (Note: because we're a
commercial code, making the customer compile something doesn't work for us
like it can for open source or research codes.)

Now, we want to take advantage of OpenMPI's ability to find MCAs at
runtime, pointing to the various plugins that might apply to a deployed
system. I've configured and compiled OpenMPI on one of our build machines,
one that doesn't have any special interconnect hardware or software
installed. We take this compiled version of OpenMPI and use it on all of
our machines. (Yes, I've read Building FAQ #39
<https://www.open-mpi.org/faq/?category=building#installdirs> about
relocating OpenMPI. Useful, that.) I'd like to take our pre-compiled
version of OpenMPI and add MCA libraries to it, giving OpenMPI the ability
to communicate via transport mechanisms that weren't available on the
original build machine. Things like InfiniBand, OmniPath, or one of Cray's
interconnects.

How would I go about doing this? And what are the limitations?

I'm guessing that I need to go configure and compile the same version of
OpenMPI on a machine that has the desired interconnect installation
(headers and libraries), then go grab the corresponding
lib/openmpi/mca_*{la,so} files. Take those files and drop them in our
pre-built OpenMPI from our build machine in the same relative plugin
location (lib/openmpi). If I stick with the same compiler (gcc, in this
case), I'm hoping that symbols will all resolve themselves at runtime. (I
probably will have to do some LD_LIBRARY_PATH games to be sure to find the
appropriate underlying libraries unless OpenMPI's process for building MCAs
links them in statically somehow.)

Am I even on the right track here? (The various system-level FAQs (here
<https://www.open-mpi.org/faq/?category=supported-systems>, here
<https://www.open-mpi.org/faq/?category=developers>, and especially here
<https://www.open-mpi.org/faq/?category=sysadmin>) seem to suggest that I
am.)

Our first test platform will be getting OpenMPI via IB working on our
cluster, where we have IB (and TCP/IP) functional and not OpenMPI. This
will be a great stand-in for a customer that has an IB cluster and wants to
just run our binary installation.

Thanks.

-Sean

--
Sean Ahern
Computational Engineering International
919-363-0883

[r***@open-mpi.org]

You don’t need any of the hardware - you just need the headers. Things like libfabric and libibverbs are all publicly available, and so you can build all that support even if you cannot run it on your machine.

Once your customer installs the binary, the various plugins will check for their required library and hardware and disqualify themselves if it isn’t found.

There's been discussion on the OpenMPI list recently about static linking of OpenMPI with all of the desired MCAs in it. I've got the opposite question. I'd like to add MCAs later on to an already-compiled version of OpenMPI and am not quite sure how to do it.
Let me summarize. We've got a commercial code that we deploy on customer machines in binary form. We're working to integrate OpenMPI into the installer, and things seem to be progressing well. (Note: because we're a commercial code, making the customer compile something doesn't work for us like it can for open source or research codes.)
Now, we want to take advantage of OpenMPI's ability to find MCAs at runtime, pointing to the various plugins that might apply to a deployed system. I've configured and compiled OpenMPI on one of our build machines, one that doesn't have any special interconnect hardware or software installed. We take this compiled version of OpenMPI and use it on all of our machines. (Yes, I've read Building FAQ #39 <https://www.open-mpi.org/faq/?category=building#installdirs> about relocating OpenMPI. Useful, that.) I'd like to take our pre-compiled version of OpenMPI and add MCA libraries to it, giving OpenMPI the ability to communicate via transport mechanisms that weren't available on the original build machine. Things like InfiniBand, OmniPath, or one of Cray's interconnects.
How would I go about doing this? And what are the limitations?
I'm guessing that I need to go configure and compile the same version of OpenMPI on a machine that has the desired interconnect installation (headers and libraries), then go grab the corresponding lib/openmpi/mca_*{la,so} files. Take those files and drop them in our pre-built OpenMPI from our build machine in the same relative plugin location (lib/openmpi). If I stick with the same compiler (gcc, in this case), I'm hoping that symbols will all resolve themselves at runtime. (I probably will have to do some LD_LIBRARY_PATH games to be sure to find the appropriate underlying libraries unless OpenMPI's process for building MCAs links them in statically somehow.)
Am I even on the right track here? (The various system-level FAQs (here <https://www.open-mpi.org/faq/?category=supported-systems>, here <https://www.open-mpi.org/faq/?category=developers>, and especially here <https://www.open-mpi.org/faq/?category=sysadmin>) seem to suggest that I am.)
Our first test platform will be getting OpenMPI via IB working on our cluster, where we have IB (and TCP/IP) functional and not OpenMPI. This will be a great stand-in for a customer that has an IB cluster and wants to just run our binary installation.
Thanks.
-Sean
--
Sean Ahern
Computational Engineering International
919-363-0883
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Sean Ahern]

Thanks. That's what I expected and hoped. But is there a pointer about how
to get started? If I've got an existing OpenMPI build, what's the process
to get a new MCA plugin built with a new set of header files?

(I'm a bit surprised only header files are necessary. Shouldn't the plugin
require at least runtime linking with a low-level transport library?)

-Sean

--
Sean Ahern
Computational Engineering International
919-363-0883

Post by r***@open-mpi.org
You don’t need any of the hardware - you just need the headers. Things
like libfabric and libibverbs are all publicly available, and so you can
build all that support even if you cannot run it on your machine.
Once your customer installs the binary, the various plugins will check for
their required library and hardware and disqualify themselves if it isn’t
found.
There's been discussion on the OpenMPI list recently about static linking
of OpenMPI with all of the desired MCAs in it. I've got the opposite
question. I'd like to add MCAs later on to an already-compiled version of
OpenMPI and am not quite sure how to do it.
Let me summarize. We've got a commercial code that we deploy on customer
machines in binary form. We're working to integrate OpenMPI into the
installer, and things seem to be progressing well. (Note: because we're a
commercial code, making the customer compile something doesn't work for us
like it can for open source or research codes.)
Now, we want to take advantage of OpenMPI's ability to find MCAs at
runtime, pointing to the various plugins that might apply to a deployed
system. I've configured and compiled OpenMPI on one of our build machines,
one that doesn't have any special interconnect hardware or software
installed. We take this compiled version of OpenMPI and use it on all of
our machines. (Yes, I've read Building FAQ #39
<https://www.open-mpi.org/faq/?category=building#installdirs> about
relocating OpenMPI. Useful, that.) I'd like to take our pre-compiled
version of OpenMPI and add MCA libraries to it, giving OpenMPI the ability
to communicate via transport mechanisms that weren't available on the
original build machine. Things like InfiniBand, OmniPath, or one of Cray's
interconnects.
How would I go about doing this? And what are the limitations?
I'm guessing that I need to go configure and compile the same version of
OpenMPI on a machine that has the desired interconnect installation
(headers and libraries), then go grab the corresponding
lib/openmpi/mca_*{la,so} files. Take those files and drop them in our
pre-built OpenMPI from our build machine in the same relative plugin
location (lib/openmpi). If I stick with the same compiler (gcc, in this
case), I'm hoping that symbols will all resolve themselves at runtime. (I
probably will have to do some LD_LIBRARY_PATH games to be sure to find the
appropriate underlying libraries unless OpenMPI's process for building MCAs
links them in statically somehow.)
Am I even on the right track here? (The various system-level FAQs (here
<https://www.open-mpi.org/faq/?category=supported-systems>, here
<https://www.open-mpi.org/faq/?category=developers>, and especially here
<https://www.open-mpi.org/faq/?category=sysadmin>) seem to suggest that I
am.)
Our first test platform will be getting OpenMPI via IB working on our
cluster, where we have IB (and TCP/IP) functional and not OpenMPI. This
will be a great stand-in for a customer that has an IB cluster and wants to
just run our binary installation.
Thanks.
-Sean
--
Sean Ahern
Computational Engineering International
919-363-0883
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[r***@open-mpi.org]

Here’s a link on how to create components:

https://github.com/open-mpi/ompi/wiki/devel-CreateComponent

and if you want to create a completely new framework:

https://github.com/open-mpi/ompi/wiki/devel-CreateFramework

If you want to distribute a proprietary plugin, you first develop and build it within the OMPI code base on your own machines. Then, just take the dll for your plugin from the <prefix>/lib/openmpi directory and distribute that “blob”.

I’ll correct my comment: you need the headers and the libraries. You just don’t need the hardware, though it means you cannot test those features.

Thanks. That's what I expected and hoped. But is there a pointer about how to get started? If I've got an existing OpenMPI build, what's the process to get a new MCA plugin built with a new set of header files?
(I'm a bit surprised only header files are necessary. Shouldn't the plugin require at least runtime linking with a low-level transport library?)
-Sean
--
Sean Ahern
Computational Engineering International
919-363-0883
You don’t need any of the hardware - you just need the headers. Things like libfabric and libibverbs are all publicly available, and so you can build all that support even if you cannot run it on your machine.
Once your customer installs the binary, the various plugins will check for their required library and hardware and disqualify themselves if it isn’t found.

There's been discussion on the OpenMPI list recently about static linking of OpenMPI with all of the desired MCAs in it. I've got the opposite question. I'd like to add MCAs later on to an already-compiled version of OpenMPI and am not quite sure how to do it.
Let me summarize. We've got a commercial code that we deploy on customer machines in binary form. We're working to integrate OpenMPI into the installer, and things seem to be progressing well. (Note: because we're a commercial code, making the customer compile something doesn't work for us like it can for open source or research codes.)
Now, we want to take advantage of OpenMPI's ability to find MCAs at runtime, pointing to the various plugins that might apply to a deployed system. I've configured and compiled OpenMPI on one of our build machines, one that doesn't have any special interconnect hardware or software installed. We take this compiled version of OpenMPI and use it on all of our machines. (Yes, I've read Building FAQ #39 <https://www.open-mpi.org/faq/?category=building#installdirs> about relocating OpenMPI. Useful, that.) I'd like to take our pre-compiled version of OpenMPI and add MCA libraries to it, giving OpenMPI the ability to communicate via transport mechanisms that weren't available on the original build machine. Things like InfiniBand, OmniPath, or one of Cray's interconnects.
How would I go about doing this? And what are the limitations?
I'm guessing that I need to go configure and compile the same version of OpenMPI on a machine that has the desired interconnect installation (headers and libraries), then go grab the corresponding lib/openmpi/mca_*{la,so} files. Take those files and drop them in our pre-built OpenMPI from our build machine in the same relative plugin location (lib/openmpi). If I stick with the same compiler (gcc, in this case), I'm hoping that symbols will all resolve themselves at runtime. (I probably will have to do some LD_LIBRARY_PATH games to be sure to find the appropriate underlying libraries unless OpenMPI's process for building MCAs links them in statically somehow.)
Am I even on the right track here? (The various system-level FAQs (here <https://www.open-mpi.org/faq/?category=supported-systems>, here <https://www.open-mpi.org/faq/?category=developers>, and especially here <https://www.open-mpi.org/faq/?category=sysadmin>) seem to suggest that I am.)
Our first test platform will be getting OpenMPI via IB working on our cluster, where we have IB (and TCP/IP) functional and not OpenMPI. This will be a great stand-in for a customer that has an IB cluster and wants to just run our binary installation.
Thanks.
-Sean
--
Sean Ahern
Computational Engineering International
919-363-0883
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Sean Ahern]

That's useful. Thank you.

It sounds like, as long as the component exists for OpenMPI already, it's
just a matter of compiling OpenMPI on a machine that has the headers and
libraries (with appropriate configure flags), and grabbing the individual
component from there.

-Sean

--
Sean Ahern
Computational Engineering International
919-363-0883

Post by r***@open-mpi.org
https://github.com/open-mpi/ompi/wiki/devel-CreateComponent
https://github.com/open-mpi/ompi/wiki/devel-CreateFramework
If you want to distribute a proprietary plugin, you first develop and
build it within the OMPI code base on your own machines. Then, just take
the dll for your plugin from the <prefix>/lib/openmpi directory and
distribute that “blob”.
I’ll correct my comment: you need the headers and the libraries. You just
don’t need the hardware, though it means you cannot test those features.
Thanks. That's what I expected and hoped. But is there a pointer about how
to get started? If I've got an existing OpenMPI build, what's the process
to get a new MCA plugin built with a new set of header files?
(I'm a bit surprised only header files are necessary. Shouldn't the plugin
require at least runtime linking with a low-level transport library?)
-Sean
--
Sean Ahern
Computational Engineering International
919-363-0883

Post by r***@open-mpi.org
You don’t need any of the hardware - you just need the headers. Things
like libfabric and libibverbs are all publicly available, and so you can
build all that support even if you cannot run it on your machine.
Once your customer installs the binary, the various plugins will check
for their required library and hardware and disqualify themselves if it
isn’t found.
There's been discussion on the OpenMPI list recently about static linking
of OpenMPI with all of the desired MCAs in it. I've got the opposite
question. I'd like to add MCAs later on to an already-compiled version of
OpenMPI and am not quite sure how to do it.
Let me summarize. We've got a commercial code that we deploy on customer
machines in binary form. We're working to integrate OpenMPI into the
installer, and things seem to be progressing well. (Note: because we're a
commercial code, making the customer compile something doesn't work for us
like it can for open source or research codes.)
Now, we want to take advantage of OpenMPI's ability to find MCAs at
runtime, pointing to the various plugins that might apply to a deployed
system. I've configured and compiled OpenMPI on one of our build machines,
one that doesn't have any special interconnect hardware or software
installed. We take this compiled version of OpenMPI and use it on all of
our machines. (Yes, I've read Building FAQ #39
<https://www.open-mpi.org/faq/?category=building#installdirs> about
relocating OpenMPI. Useful, that.) I'd like to take our pre-compiled
version of OpenMPI and add MCA libraries to it, giving OpenMPI the ability
to communicate via transport mechanisms that weren't available on the
original build machine. Things like InfiniBand, OmniPath, or one of Cray's
interconnects.
How would I go about doing this? And what are the limitations?
I'm guessing that I need to go configure and compile the same version of
OpenMPI on a machine that has the desired interconnect installation
(headers and libraries), then go grab the corresponding
lib/openmpi/mca_*{la,so} files. Take those files and drop them in our
pre-built OpenMPI from our build machine in the same relative plugin
location (lib/openmpi). If I stick with the same compiler (gcc, in this
case), I'm hoping that symbols will all resolve themselves at runtime. (I
probably will have to do some LD_LIBRARY_PATH games to be sure to find the
appropriate underlying libraries unless OpenMPI's process for building MCAs
links them in statically somehow.)
Am I even on the right track here? (The various system-level FAQs (here
<https://www.open-mpi.org/faq/?category=supported-systems>, here
<https://www.open-mpi.org/faq/?category=developers>, and especially here
<https://www.open-mpi.org/faq/?category=sysadmin>) seem to suggest that
I am.)
Our first test platform will be getting OpenMPI via IB working on our
cluster, where we have IB (and TCP/IP) functional and not OpenMPI. This
will be a great stand-in for a customer that has an IB cluster and wants to
just run our binary installation.
Thanks.
-Sean
--
Sean Ahern
Computational Engineering International
919-363-0883
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users