[OMPI users] Strange OpenMPI errors showing up in Caffe rc5 build

[Lane, William]

I know this could possibly be off-topic, but the errors are OpenMPI errors and if anyone could shed light on the nature of these errors I figure it would be this group:


CXX/LD -o .build_release/tools/upgrade_solver_proto_text.bin
g++ .build_release/tools/upgrade_solver_proto_text.o -o .build_release/tools/upgrade_solver_proto_text.bin -pthread -fPIC -DCAFFE_VERSION=1.0.0-rc5 -DNDEBUG -O2 -DUSE_OPENCV -DUSE_LEVELDB -DUSE_LMDB -DCPU_ONLY -DWITH_PYTHON_LAYER -I/hpc/apps/python27/include/python2.7 -I/hpc/apps/python27/externals/numpy/1.9.2/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/hpc/apps/hdf5/1.8.17/include -I.build_release/src -I./src -I./include -I/hpc/apps/atlas/3.10.2/include -Wall -Wno-sign-compare -lcaffe -L/hpc/apps/gflags/lib -L/hpc/apps/python27/lib -L/hpc/apps/python27/lib/python2.7 -L/hpc/apps/atlas/3.10.2/lib -L.build_release/lib -lglog -lgflags -lprotobuf -lboost_system -lboost_filesystem -lm -lhdf5_hl -lhdf5 -lleveldb -lsnappy -llmdb -lopencv_core -lopencv_highgui -lopencv_imgproc -lboost_thread -lstdc++ -lboost_python -lpython2.7 -lcblas -latlas \
-Wl,-rpath,\$ORIGIN/../lib
.build_release/lib/libcaffe.so: undefined reference to `ompi_mpi_cxx_op_intercept'
.build_release/lib/libcaffe.so: undefined reference to `MPI::Datatype::Free()'
.build_release/lib/libcaffe.so: undefined reference to `MPI::Comm::Comm()'
.build_release/lib/libcaffe.so: undefined reference to `MPI::Win::Free()'
collect2: error: ld returned 1 exit status


I've read this may be due to a dependency of Caffe that uses OpenMPI (since I've been told Caffe itself doesn't use OpenMPI).


Would adding -l directives to LIBRARIES line in the Makefile for Caffe that reference all OpenMPI libraries fix this problem?

For example, -l mpi.


Thank you in advance. Hopefully this isn't entirely OT.


William L.

IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is strictly prohibited. Thank you for your cooperation. IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is strictly prohibited. Thank you for your cooperation.

[g***@rist.or.jp]

William,

the link error clearly shows libcaffe.so does require C++ bindings.

did you build caffe from a fresh tree ?

what if you

ldd libcaffe.so

nm libcaffe.so | grep -i ompi

if libcaffe.so does require mpi c++ bindings, it should depend on it

(otherwise the way it was built is questionnable)

you might want to link with mpic++ instead of g++

note mpi C++ bindings are no more built by default since v2.0, so you
likely have to

configure --enable-mpi-cxx

last but not least, make sure caffe and openmpi were built with the same
c++ compiler

Cheers,

Gilles
----- Original Message -----

I know this could possibly be off-topic, but the errors are OpenMPI
errors and if anyone could shed light on the nature of these errors I
figure it would be this group:


CXX/LD -o .build_release/tools/upgrade_solver_proto_text.bin
g++ .build_release/tools/upgrade_solver_proto_text.o -o .build_
release/tools/upgrade_solver_proto_text.bin -pthread -fPIC -DCAFFE_
VERSION=1.0.0-rc5 -DNDEBUG -O2 -DUSE_OPENCV -DUSE_LEVELDB -DUSE_LMDB -
DCPU_ONLY -DWITH_PYTHON_LAYER -I/hpc/apps/python27/include/python2.7 -I/
hpc/apps/python27/externals/numpy/1.9.2/lib/python2.7/site-packages/
numpy/core/include -I/usr/local/include -I/hpc/apps/hdf5/1.8.17/include
-I.build_release/src -I./src -I./include -I/hpc/apps/atlas/3.10.2/
include -Wall -Wno-sign-compare -lcaffe -L/hpc/apps/gflags/lib -L/hpc/
apps/python27/lib -L/hpc/apps/python27/lib/python2.7 -L/hpc/apps/atlas/3.
10.2/lib -L.build_release/lib -lglog -lgflags -lprotobuf -lboost_system
-lboost_filesystem -lm -lhdf5_hl -lhdf5 -lleveldb -lsnappy -llmdb -
lopencv_core -lopencv_highgui -lopencv_imgproc -lboost_thread -lstdc++ -
lboost_python -lpython2.7 -lcblas -latlas \
-Wl,-rpath,\$ORIGIN/../lib
.build_release/lib/libcaffe.so: undefined reference to `ompi_mpi
_cxx_op_intercept'
.build_release/lib/libcaffe.so: undefined reference to `MPI::
Datatype::Free()'
.build_release/lib/libcaffe.so: undefined reference to `MPI::
Comm::Comm()'
.build_release/lib/libcaffe.so: undefined reference to `MPI::Win:
:Free()'
collect2: error: ld returned 1 exit status


I've read this may be due to a dependency of Caffe that uses OpenMPI
(since I've been told Caffe itself doesn't use OpenMPI).


Would adding -l directives to LIBRARIES line in the Makefile for
Caffe that reference all OpenMPI libraries fix this problem?

For example, -l mpi.


Thank you in advance. Hopefully this isn't entirely OT.


William L.
IMPORTANT WARNING: This message is intended for the use of the
person or entity to which it is addressed and may contain information
that is privileged and confidential, the disclosure of which is governed
by applicable law. If the reader of this message is not the intended
recipient, or the employee or agent responsible for delivering it to the
intended recipient, you are hereby notified that any dissemination,
distribution or copying of this information is strictly prohibited.
Thank you for your cooperation. IMPORTANT WARNING: This message is
intended for the use of the person or entity to which it is addressed
and may contain information that is privileged and confidential, the
disclosure of which is governed by applicable law. If the reader of this
message is not the intended recipient, or the employee or agent
responsible for delivering it to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
information is strictly prohibited. Thank you for your cooperation.

[Prentice Bisbal]

On 05/04/2017 09:08 PM, ***@rist.or.jp wrote:
>
> William,
>
> the link error clearly shows libcaffe.so does require C++ bindings.
>
> did you build caffe from a fresh tree ?
>
> what if you
>
> ldd libcaffe.so
>
> nm libcaffe.so | grep -i ompi
>
> if libcaffe.so does require mpi c++ bindings, it should depend on it
>
> (otherwise the way it was built is questionnable)
>
> you might want to link with mpic++ instead of g++
>

This is a great point I missed in my previous e-mail on this topic. When
compiling a program that uses MPI, you want to specify the MPI compiler
wrappers for your C, C++ and Fortran Compilers, and not your chosen
compiler directly, For example:

./configure --prefix=/usr/local/foo-1.2.3 CC=mpicc CXX=mpicxx FC=mpif90

or something similar. This guarantees that the actual compiler is
called with all the write flags for the C preprocessor, linker, etc.
This will almost always prevent those linking errors.
>
> note mpi C++ bindings are no more built by default since v2.0, so you
> likely have to
>
> configure --enable-mpi-cxx
>
> last but not least, make sure caffe and openmpi were built with the
> same c++ compiler
>
> Cheers,
>
> Gilles
>
> ----- Original Message -----
>
> I know this could possibly be off-topic, but the errors are
> OpenMPI errors and if anyone could shed light on the nature of
> these errors I figure it would be this group:
>
> CXX/LD -o .build_release/tools/upgrade_solver_proto_text.bin
> g++ .build_release/tools/upgrade_solver_proto_text.o -o
> .build_release/tools/upgrade_solver_proto_text.bin -pthread
> -fPIC -DCAFFE_VERSION=1.0.0-rc5 -DNDEBUG -O2 -DUSE_OPENCV
> -DUSE_LEVELDB -DUSE_LMDB -DCPU_ONLY -DWITH_PYTHON_LAYER
> -I/hpc/apps/python27/include/python2.7
> -I/hpc/apps/python27/externals/numpy/1.9.2/lib/python2.7/site-packages/numpy/core/include
> -I/usr/local/include -I/hpc/apps/hdf5/1.8.17/include
> -I.build_release/src -I./src -I./include
> -I/hpc/apps/atlas/3.10.2/include -Wall -Wno-sign-compare
> -lcaffe -L/hpc/apps/gflags/lib -L/hpc/apps/python27/lib
> -L/hpc/apps/python27/lib/python2.7
> -L/hpc/apps/atlas/3.10.2/lib -L.build_release/lib-lglog
> -lgflags -lprotobuf -lboost_system -lboost_filesystem -lm
> -lhdf5_hl -lhdf5 -lleveldb -lsnappy -llmdb -lopencv_core
> -lopencv_highgui -lopencv_imgproc -lboost_thread -lstdc++
> -lboost_python -lpython2.7 -lcblas -latlas \
> -Wl,-rpath,\$ORIGIN/../lib
> .build_release/lib/libcaffe.so: undefined reference to
> `ompi_mpi_cxx_op_intercept'
> .build_release/lib/libcaffe.so: undefined reference to
> `MPI::Datatype::Free()'
> .build_release/lib/libcaffe.so: undefined reference to
> `MPI::Comm::Comm()'
> .build_release/lib/libcaffe.so: undefined reference to
> `MPI::Win::Free()'
> collect2: error: ld returned 1 exit status
>
> I've read this may be due to a dependency of Caffe that uses
> OpenMPI (since I've been told Caffe itself doesn't use OpenMPI).
>
> Would adding -l directives to LIBRARIES line in the Makefile for
> Caffe that reference all OpenMPI libraries fix this problem?
>
> For example, -l mpi.
>
> Thank you in advance. Hopefully this isn't entirely OT.
>
> William L.
>
> IMPORTANT WARNING: This message is intended for the use of the
> person or entity to which it is addressed and may contain
> information that is privileged and confidential, the disclosure of
> which is governed by applicable law. If the reader of this message
> is not the intended recipient, or the employee or agent
> responsible for delivering it to the intended recipient, you are
> hereby notified that any dissemination, distribution or copying of
> this information is strictly prohibited. Thank you for your
> cooperation. IMPORTANT WARNING: This message is intended for the
> use of the person or entity to which it is addressed and may
> contain information that is privileged and confidential, the
> disclosure of which is governed by applicable law. If the reader
> of this message is not the intended recipient, or the employee or
> agent responsible for delivering it to the intended recipient, you
> are hereby notified that any dissemination, distribution or
> copying of this information is strictly prohibited. Thank you for
> your cooperation.
>
>
>
> _______________________________________________
> users mailing list
> ***@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Lane, William]

The strange thing is OpenMPI isn't mentioned anywhere as being a dependency for Caffe! I haven't read anything that suggests OpenMPI is supported in Caffe either. This is why I figure it must be a dependency of Caffe (of which there are 15) that relies on OpenMPI.


I tried setting the compiler to mpic++ in the Makefile.config file and the result was:

Makefile:314: *** Cannot static link with the mpic++ compiler. Stop.


I'm going to try explicitly enumerating all OpenMPI libraries in Makefile.config and see if that makes a difference.


Thanks for your help, the Caffe listserve group doesn't have any answers for this issue (except use the Docker image).


-William L.

________________________________
From: users <users-***@lists.open-mpi.org> on behalf of Prentice Bisbal <***@pppl.gov>
Sent: Friday, May 5, 2017 7:47:39 AM
To: ***@lists.open-mpi.org
Subject: Re: [OMPI users] Strange OpenMPI errors showing up in Caffe rc5 build


On 05/04/2017 09:08 PM, ***@rist.or.jp<mailto:***@rist.or.jp> wrote:

William,



the link error clearly shows libcaffe.so does require C++ bindings.

did you build caffe from a fresh tree ?

what if you

ldd libcaffe.so

nm libcaffe.so | grep -i ompi

if libcaffe.so does require mpi c++ bindings, it should depend on it

(otherwise the way it was built is questionnable)

you might want to link with mpic++ instead of g++

This is a great point I missed in my previous e-mail on this topic. When compiling a program that uses MPI, you want to specify the MPI compiler wrappers for your C, C++ and Fortran Compilers, and not your chosen compiler directly, For example:

./configure --prefix=/usr/local/foo-1.2.3 CC=mpicc CXX=mpicxx FC=mpif90

or something similar. This guarantees that the actual compiler is called with all the write flags for the C preprocessor, linker, etc. This will almost always prevent those linking errors.



note mpi C++ bindings are no more built by default since v2.0, so you likely have to

configure --enable-mpi-cxx

last but not least, make sure caffe and openmpi were built with the same c++ compiler



Cheers,



Gilles

----- Original Message -----

I know this could possibly be off-topic, but the errors are OpenMPI errors and if anyone could shed light on the nature of these errors I figure it would be this group:





CXX/LD -o .build_release/tools/upgrade_solver_proto_text.bin
g++ .build_release/tools/upgrade_solver_proto_text.o -o .build_release/tools/upgrade_solver_proto_text.bin -pthread -fPIC -DCAFFE_VERSION=1.0.0-rc5 -DNDEBUG -O2 -DUSE_OPENCV -DUSE_LEVELDB -DUSE_LMDB -DCPU_ONLY -DWITH_PYTHON_LAYER -I/hpc/apps/python27/include/python2.7 -I/hpc/apps/python27/externals/numpy/1.9.2/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/hpc/apps/hdf5/1.8.17/include -I.build_release/src -I./src -I./include -I/hpc/apps/atlas/3.10.2/include -Wall -Wno-sign-compare -lcaffe -L/hpc/apps/gflags/lib -L/hpc/apps/python27/lib -L/hpc/apps/python27/lib/python2.7 -L/hpc/apps/atlas/3.10.2/lib -L.build_release/lib -lglog -lgflags -lprotobuf -lboost_system -lboost_filesystem -lm -lhdf5_hl -lhdf5 -lleveldb -lsnappy -llmdb -lopencv_core -lopencv_highgui -lopencv_imgproc -lboost_thread -lstdc++ -lboost_python -lpython2.7 -lcblas -latlas \
-Wl,-rpath,\$ORIGIN/../lib
.build_release/lib/libcaffe.so: undefined reference to `ompi_mpi_cxx_op_intercept'
.build_release/lib/libcaffe.so: undefined reference to `MPI::Datatype::Free()'
.build_release/lib/libcaffe.so: undefined reference to `MPI::Comm::Comm()'
.build_release/lib/libcaffe.so: undefined reference to `MPI::Win::Free()'
collect2: error: ld returned 1 exit status





I've read this may be due to a dependency of Caffe that uses OpenMPI (since I've been told Caffe itself doesn't use OpenMPI).



Would adding -l directives to LIBRARIES line in the Makefile for Caffe that reference all OpenMPI libraries fix this problem?

For example, -l mpi.



Thank you in advance. Hopefully this isn't entirely OT.



William L.

IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is strictly prohibited. Thank you for your cooperation. IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is strictly prohibited. Thank you for your cooperation.



_______________________________________________
users mailing list
***@lists.open-mpi.org<mailto:***@lists.open-mpi.org>
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is strictly prohibited. Thank you for your cooperation.

[Jeff Squyres (jsquyres)]

On May 6, 2017, at 3:28 AM, Lane, William <***@cshs.org> wrote:
>
> The strange thing is OpenMPI isn't mentioned anywhere as being a dependency for Caffe! I haven't read anything that suggests OpenMPI is supported in Caffe either. This is why I figure it must be a dependency of Caffe (of which there are 15) that relies on OpenMPI.
>
> I tried setting the compiler to mpic++ in the Makefile.config file and the result was:
> Makefile:314: *** Cannot static link with the mpic++ compiler. Stop.
>
> I'm going to try explicitly enumerating all OpenMPI libraries in Makefile.config and see if that makes a difference.

Keep in mind that the list of libraries you need to link an Open MPI application can change between different environments, and may also change between versions of Open MPI. It's safer to use the wrapper compilers, or if you can't use them, then at least query to see what you should be using to compile and link:

https://www.open-mpi.org/faq/?category=mpi-apps#cant-use-wrappers

--
Jeff Squyres
***@cisco.com

[Prentice Bisbal]

On 05/06/2017 03:28 AM, Lane, William wrote:
>
> The strange thing is OpenMPI isn't mentioned anywhere as being a
> dependency for Caffe! I haven't read anything that suggests OpenMPI is
> supported in Caffe either. This is why I figure it must be a
> dependency of Caffe (of which there are 15) that relies on OpenMPI.
>

Are you sure you didn't donwload Caffe MPI by acident? Both versions are
open-source and available from Github:

http://www.inspursystems.com/dl/open-source-caffe-mpi-download/
>
>
> I tried setting the compiler to mpic++ in the Makefile.config file and
> the result was:
>
> Makefile:314: *** Cannot static link with the mpic++ compiler. Stop.
>
>
> I'm going to try explicitly enumerating all OpenMPI libraries in
> Makefile.config and see if that makes a difference.
>

I would not recommend that. It's always better to use the wrapper
scripts (mpicc, mpic++, mpif90, etc.). If that's not working, it would
be better for you to find out why and fix that problem I would start
with a simple MPI-enabled "Hello, world!" C++ program. See if you can
compile that, and go from there. Start simple and work your way up from
there.

I've seen errors similar to yours in the past that were caused by the
wrong switches being passed to the compiler. I've also seen similar
errors when the compiler command was screwed up, too (typo, etc.) Also,
check to make sure that the static OpenMPI libraries exist. I think
they're built by default, but I could be wrong. I always explicitly
specify building both static and dynamic libraries for all my software
at configure time.

Also, you when you post error messages, ALWAYS include the command that
caused the error, too. An error message by itself, like the one above,
doesn't give us much information to help you diagnose the problem. If
you provided the command, it's possible that someone on the list could
immediately see a problem with the command and quickly pinpoint the
problem.

Prentice
>
>
> Thanks for your help, the Caffe listserve group doesn't have any
> answers for this issue (except use the Docker image).
>
>
> -William L.
>
> ------------------------------------------------------------------------
> *From:* users <users-***@lists.open-mpi.org> on behalf of Prentice
> Bisbal <***@pppl.gov>
> *Sent:* Friday, May 5, 2017 7:47:39 AM
> *To:* ***@lists.open-mpi.org
> *Subject:* Re: [OMPI users] Strange OpenMPI errors showing up in Caffe
> rc5 build
>
> On 05/04/2017 09:08 PM, ***@rist.or.jp wrote:
>>
>> William,
>>
>> the link error clearly shows libcaffe.so does require C++ bindings.
>>
>> did you build caffe from a fresh tree ?
>>
>> what if you
>>
>> ldd libcaffe.so
>>
>> nm libcaffe.so | grep -i ompi
>>
>> if libcaffe.so does require mpi c++ bindings, it should depend on it
>>
>> (otherwise the way it was built is questionnable)
>>
>> you might want to link with mpic++ instead of g++
>>
>
> This is a great point I missed in my previous e-mail on this topic.
> When compiling a program that uses MPI, you want to specify the MPI
> compiler wrappers for your C, C++ and Fortran Compilers, and not your
> chosen compiler directly, For example:
>
> ./configure --prefix=/usr/local/foo-1.2.3 CC=mpicc CXX=mpicxx FC=mpif90
>
> or something similar. This guarantees that the actual compiler is
> called with all the write flags for the C preprocessor, linker, etc.
> This will almost always prevent those linking errors.
>>
>> note mpi C++ bindings are no more built by default since v2.0, so you
>> likely have to
>>
>> configure --enable-mpi-cxx
>>
>> last but not least, make sure caffe and openmpi were built with the
>> same c++ compiler
>>
>> Cheers,
>>
>> Gilles
>>
>> ----- Original Message -----
>>
>> I know this could possibly be off-topic, but the errors are
>> OpenMPI errors and if anyone could shed light on the nature of
>> these errors I figure it would be this group:
>>
>> CXX/LD -o .build_release/tools/upgrade_solver_proto_text.bin
>> g++ .build_release/tools/upgrade_solver_proto_text.o -o
>> .build_release/tools/upgrade_solver_proto_text.bin -pthread
>> -fPIC -DCAFFE_VERSION=1.0.0-rc5 -DNDEBUG -O2 -DUSE_OPENCV
>> -DUSE_LEVELDB -DUSE_LMDB -DCPU_ONLY -DWITH_PYTHON_LAYER
>> -I/hpc/apps/python27/include/python2.7
>> -I/hpc/apps/python27/externals/numpy/1.9.2/lib/python2.7/site-packages/numpy/core/include
>> -I/usr/local/include -I/hpc/apps/hdf5/1.8.17/include
>> -I.build_release/src -I./src -I./include
>> -I/hpc/apps/atlas/3.10.2/include -Wall -Wno-sign-compare
>> -lcaffe -L/hpc/apps/gflags/lib -L/hpc/apps/python27/lib
>> -L/hpc/apps/python27/lib/python2.7
>> -L/hpc/apps/atlas/3.10.2/lib -L.build_release/lib-lglog
>> -lgflags -lprotobuf -lboost_system -lboost_filesystem -lm
>> -lhdf5_hl -lhdf5 -lleveldb -lsnappy -llmdb -lopencv_core
>> -lopencv_highgui -lopencv_imgproc -lboost_thread -lstdc++
>> -lboost_python -lpython2.7 -lcblas -latlas \
>> -Wl,-rpath,\$ORIGIN/../lib
>> .build_release/lib/libcaffe.so: undefined reference to
>> `ompi_mpi_cxx_op_intercept'
>> .build_release/lib/libcaffe.so: undefined reference to
>> `MPI::Datatype::Free()'
>> .build_release/lib/libcaffe.so: undefined reference to
>> `MPI::Comm::Comm()'
>> .build_release/lib/libcaffe.so: undefined reference to
>> `MPI::Win::Free()'
>> collect2: error: ld returned 1 exit status
>>
>> I've read this may be due to a dependency of Caffe that uses
>> OpenMPI (since I've been told Caffe itself doesn't use OpenMPI).
>>
>> Would adding -l directives to LIBRARIES line in the Makefile for
>> Caffe that reference all OpenMPI libraries fix this problem?
>>
>> For example, -l mpi.
>>
>> Thank you in advance. Hopefully this isn't entirely OT.
>>
>> William L.
>>
>> IMPORTANT WARNING: This message is intended for the use of the
>> person or entity to which it is addressed and may contain
>> information that is privileged and confidential, the disclosure
>> of which is governed by applicable law. If the reader of this
>> message is not the intended recipient, or the employee or agent
>> responsible for delivering it to the intended recipient, you are
>> hereby notified that any dissemination, distribution or copying
>> of this information is strictly prohibited. Thank you for your
>> cooperation. IMPORTANT WARNING: This message is intended for the
>> use of the person or entity to which it is addressed and may
>> contain information that is privileged and confidential, the
>> disclosure of which is governed by applicable law. If the reader
>> of this message is not the intended recipient, or the employee or
>> agent responsible for delivering it to the intended recipient,
>> you are hereby notified that any dissemination, distribution or
>> copying of this information is strictly prohibited. Thank you for
>> your cooperation.
>>
>>
>>
>> _______________________________________________
>> users mailing list
>> ***@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
> IMPORTANT WARNING: This message is intended for the use of the person
> or entity to which it is addressed and may contain information that is
> privileged and confidential, the disclosure of which is governed by
> applicable law. If the reader of this message is not the intended
> recipient, or the employee or agent responsible for delivering it to
> the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this information is strictly
> prohibited. Thank you for your cooperation.
>
>
> _______________________________________________
> users mailing list
> ***@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users