[OMPI users] error on dlopen

[Mahmood Naderan]

Hi
I am building scalapack with mpicc and mpifort, however this is the error I
get:

mpifort -O3 -o xCbtest blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In function
`dlopen_close':
dl_dlopen_module.c:(.text+0x29d): undefined reference to `dlclose'
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In function
`dlopen_lookup':
dl_dlopen_module.c:(.text+0x2d0): undefined reference to `dlsym'
dl_dlopen_module.c:(.text+0x2fb): undefined reference to `dlerror'
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In function
`dlopen_open':
dl_dlopen_module.c:(.text+0x3ca): undefined reference to `dlopen'
dl_dlopen_module.c:(.text+0x431): undefined reference to `dlerror'
dl_dlopen_module.c:(.text+0x456): undefined reference to `dlopen'
dl_dlopen_module.c:(.text+0x4a9): undefined reference to `dlerror'
dl_dlopen_module.c:(.text+0x501): undefined reference to `dlopen'
/opt/openmpi-2.0.1/lib/libopen-pal.a(patcher_overwrite_module.o): In
function `mca_patcher_overwrite_patch_symbol':
patcher_overwrite_module.c:(.text+0x12e): undefined reference to `dlsym'
patcher_overwrite_module.c:(.text+0x166): undefined reference to `dlsym'
patcher_overwrite_module.c:(.text+0x173): undefined reference to `dlerror'
collect2: error: ld returned 1 exit status
Makefile:18: recipe for target 'xCbtest' failed
make[2]: *** [xCbtest] Error 1



As I grep "dlopen", some OMPI binary files match. Any idea about that?


Regards,
Mahmood

[Sean Ahern]

Sounds to me like you're missing a -ldl linker flag.

-Sean

--
Sean Ahern
Computational Engineering International
919-363-0883

Post by Mahmood Naderan
Hi
I am building scalapack with mpicc and mpifort, however this is the error
mpifort -O3 -o xCbtest blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In function
dl_dlopen_module.c:(.text+0x29d): undefined reference to `dlclose'
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In function
dl_dlopen_module.c:(.text+0x2d0): undefined reference to `dlsym'
dl_dlopen_module.c:(.text+0x2fb): undefined reference to `dlerror'
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In function
dl_dlopen_module.c:(.text+0x3ca): undefined reference to `dlopen'
dl_dlopen_module.c:(.text+0x431): undefined reference to `dlerror'
dl_dlopen_module.c:(.text+0x456): undefined reference to `dlopen'
dl_dlopen_module.c:(.text+0x4a9): undefined reference to `dlerror'
dl_dlopen_module.c:(.text+0x501): undefined reference to `dlopen'
/opt/openmpi-2.0.1/lib/libopen-pal.a(patcher_overwrite_module.o): In
patcher_overwrite_module.c:(.text+0x12e): undefined reference to `dlsym'
patcher_overwrite_module.c:(.text+0x166): undefined reference to `dlsym'
patcher_overwrite_module.c:(.text+0x173): undefined reference to `dlerror'
collect2: error: ld returned 1 exit status
Makefile:18: recipe for target 'xCbtest' failed
make[2]: *** [xCbtest] Error 1
As I grep "dlopen", some OMPI binary files match. Any idea about that?
Regards,
Mahmood
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Mahmood Naderan]

I added that but still get the same error. Please see the config file for
building scalapack

# cat SLmake.inc
CDEFS = -DAdd_
FC = mpifort -ldl
CC = mpicc
NOOPT = -O0
FCFLAGS = -O3
CCFLAGS = -O3
FCLOADER = $(FC)
CCLOADER = $(CC)
FCLOADFLAGS = $(FCFLAGS)
CCLOADFLAGS = $(CCFLAGS)
ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
SCALAPACKLIB = libscalapack.a
BLASLIB = /opt/OpenBLAS-0.2.18/libopenblas.a
LAPACKLIB =
LIBS = $(LAPACKLIB) $(BLASLIB)




Regards,
Mahmood

Post by Sean Ahern
Sounds to me like you're missing a -ldl linker flag.
-Sean
--
Sean Ahern
Computational Engineering International
919-363-0883

Post by Mahmood Naderan
Hi
I am building scalapack with mpicc and mpifort, however this is the error
mpifort -O3 -o xCbtest blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In function
dl_dlopen_module.c:(.text+0x29d): undefined reference to `dlclose'
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In function
dl_dlopen_module.c:(.text+0x2d0): undefined reference to `dlsym'
dl_dlopen_module.c:(.text+0x2fb): undefined reference to `dlerror'
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In function
dl_dlopen_module.c:(.text+0x3ca): undefined reference to `dlopen'
dl_dlopen_module.c:(.text+0x431): undefined reference to `dlerror'
dl_dlopen_module.c:(.text+0x456): undefined reference to `dlopen'
dl_dlopen_module.c:(.text+0x4a9): undefined reference to `dlerror'
dl_dlopen_module.c:(.text+0x501): undefined reference to `dlopen'
/opt/openmpi-2.0.1/lib/libopen-pal.a(patcher_overwrite_module.o): In
patcher_overwrite_module.c:(.text+0x12e): undefined reference to `dlsym'
patcher_overwrite_module.c:(.text+0x166): undefined reference to `dlsym'
patcher_overwrite_module.c:(.text+0x173): undefined reference to `dlerror'
collect2: error: ld returned 1 exit status
Makefile:18: recipe for target 'xCbtest' failed
make[2]: *** [xCbtest] Error 1
As I grep "dlopen", some OMPI binary files match. Any idea about that?
Regards,
Mahmood
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Gilles Gouaillardet]

Mahmood,


did you build Open MPI as a static only library ?


i guess the -ldl position is wrong. your link command line should be

mpifort -O3 -o xCbtest blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a -ldl


you can manually

mpifort -O3 -o xCbtest --showme blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a -ldl


it should show -ldl is added *after* -lopen-pal


Cheers,

Gilles

Post by Mahmood Naderan
I added that but still get the same error. Please see the config file
for building scalapack
# cat SLmake.inc
CDEFS = -DAdd_
FC = mpifort -ldl
CC = mpicc
NOOPT = -O0
FCFLAGS = -O3
CCFLAGS = -O3
FCLOADER = $(FC)
CCLOADER = $(CC)
FCLOADFLAGS = $(FCFLAGS)
CCLOADFLAGS = $(CCFLAGS)
ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
SCALAPACKLIB = libscalapack.a
BLASLIB = /opt/OpenBLAS-0.2.18/libopenblas.a
LAPACKLIB =
LIBS = $(LAPACKLIB) $(BLASLIB)
Regards,
Mahmood
Sounds to me like you're missing a -ldl linker flag.
-Sean
--
Sean Ahern
Computational Engineering International
919-363-0883
On Thu, Nov 3, 2016 at 3:57 PM, Mahmood Naderan
Hi
I am building scalapack with mpicc and mpifort, however this
mpifort -O3 -o xCbtest blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In
dl_dlopen_module.c:(.text+0x29d): undefined reference to `dlclose'
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In
dl_dlopen_module.c:(.text+0x2d0): undefined reference to `dlsym'
dl_dlopen_module.c:(.text+0x2fb): undefined reference to `dlerror'
/opt/openmpi-2.0.1/lib/libopen-pal.a(dl_dlopen_module.o): In
dl_dlopen_module.c:(.text+0x3ca): undefined reference to `dlopen'
dl_dlopen_module.c:(.text+0x431): undefined reference to `dlerror'
dl_dlopen_module.c:(.text+0x456): undefined reference to `dlopen'
dl_dlopen_module.c:(.text+0x4a9): undefined reference to `dlerror'
dl_dlopen_module.c:(.text+0x501): undefined reference to `dlopen'
patcher_overwrite_module.c:(.text+0x12e): undefined reference to `dlsym'
patcher_overwrite_module.c:(.text+0x166): undefined reference to `dlsym'
patcher_overwrite_module.c:(.text+0x173): undefined reference to `dlerror'
collect2: error: ld returned 1 exit status
Makefile:18: recipe for target 'xCbtest' failed
make[2]: *** [xCbtest] Error 1
As I grep "dlopen", some OMPI binary files match. Any idea about that?
Regards,
Mahmood
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Mahmood Naderan]

Post by Gilles Gouaillardet
did you build Open MPI as a static only library ?

Yes, I used --enable-static --disable-shared


Please see the output

# mpifort -O3 -o xCbtest --showme blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a -ldl
gfortran -O3 -o xCbtest blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a -ldl -I/opt/openmpi-2.0.1/include -pthread
-I/opt/openmpi-2.0.1/lib -Wl,-rpath -Wl,/opt/openmpi-2.0.1/lib
-Wl,--enable-new-dtags -L/opt/openmpi-2.0.1/lib -lmpi_usempif08
-lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lm -lrt
-lutil


I don't see what you said after "-lopen-pal". Is that OK?

Regards,
Mahmood

Post by Gilles Gouaillardet
Mahmood,
did you build Open MPI as a static only library ?
i guess the -ldl position is wrong. your link command line should be
mpifort -O3 -o xCbtest blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a -ldl
you can manually
mpifort -O3 -o xCbtest --showme blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a -ldl
it should show -ldl is added *after* -lopen-pal
Cheers,
Gilles

[Gilles Gouaillardet]

Yes, that is a problem :-(


you might want to reconfigure with

--enable-static --disable-shared --disable-dlopen

and see if it helps


or you can simply manuall edit
/opt/openmpi-2.0.1/share/openmpi/mpifort-wrapper-data.txt,

and append -ldl to the libs_static definition


Cheers,


Gilles

Post by Mahmood Naderan

Post by Gilles Gouaillardet
did you build Open MPI as a static only library ?

Yes, I used --enable-static --disable-shared
Please see the output
# mpifort -O3 -o xCbtest --showme blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a -ldl
gfortran -O3 -o xCbtest blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a -ldl -I/opt/openmpi-2.0.1/include -pthread
-I/opt/openmpi-2.0.1/lib -Wl,-rpath -Wl,/opt/openmpi-2.0.1/lib
-Wl,--enable-new-dtags -L/opt/openmpi-2.0.1/lib -lmpi_usempif08
-lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lm
-lrt -lutil
I don't see what you said after "-lopen-pal". Is that OK?
Regards,
Mahmood
On Fri, Nov 4, 2016 at 10:23 AM, Gilles Gouaillardet
Mahmood,
did you build Open MPI as a static only library ?
i guess the -ldl position is wrong. your link command line should be
mpifort -O3 -o xCbtest blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a -ldl
you can manually
mpifort -O3 -o xCbtest --showme blacstest.o btprim.o tools.o Cbt.o
../../libscalapack.a -ldl
it should show -ldl is added *after* -lopen-pal
Cheers,
Gilles
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Mahmood Naderan]

I will try that. Meanwhile, I want to know what is the performance effect
of disabling/enabling dlopen?

Regards,
Mahmood

Post by Gilles Gouaillardet
Yes, that is a problem :-(
you might want to reconfigure with
--enable-static --disable-shared --disable-dlopen
and see if it helps
or you can simply manuall edit /opt/openmpi-2.0.1/share/
openmpi/mpifort-wrapper-data.txt,
and append -ldl to the libs_static definition
Cheers,
Gilles

[Gilles Gouaillardet]

not much difference from a performance point of view.

the difference is more from a space (both memory and disk) point of view


also, if you --disable-dlopen, Open MPI is rebuilt when a single
component is updated.

(without it, you can simply make install from the updated component
directory)

so if you are developing a new component, *not* using --disable-dlopen
can save you some build time


Cheers,


Gilles

Post by Mahmood Naderan
I will try that. Meanwhile, I want to know what is the performance
effect of disabling/enabling dlopen?
Regards,
Mahmood
On Fri, Nov 4, 2016 at 11:02 AM, Gilles Gouaillardet
Yes, that is a problem :-(
you might want to reconfigure with
--enable-static --disable-shared --disable-dlopen
and see if it helps
or you can simply manuall edit
/opt/openmpi-2.0.1/share/openmpi/mpifort-wrapper-data.txt,
and append -ldl to the libs_static definition
Cheers,
Gilles
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Mahmood Naderan]

Hi Gilles,
I noticed that /opt/openmpi-2.0.1/share/openmpi/mpifort-wrapper-data.txt is
created after "make install". So, I edited it and appended -ldl to
libs_static.
Then I ran "make clean && make all" for scalapack.

However, still get the same error!!

So, let me try disabling dlopen.


Regards,
Mahmood

[Gilles Gouaillardet]

You might have to remove -ldl from the scalapack makefile

If it still does not work, can you please post
mpirun --showme ...
output ?

Cheers,

Gilles

Post by Mahmood Naderan
Hi Gilles,
I noticed that /opt/openmpi-2.0.1/share/openmpi/mpifort-wrapper-data.txt
is created after "make install". So, I edited it and appended -ldl to
libs_static.
Then I ran "make clean && make all" for scalapack.
However, still get the same error!!
So, let me try disabling dlopen.
Regards,
Mahmood

[Mahmood Naderan]

Post by Gilles Gouaillardet
You might have to remove -ldl from the scalapack makefile

I removed that before... I will try one more time

Actually, using --disable-dlopen fixed the error.

Post by Gilles Gouaillardet
mpirun --showme

$ mpirun --showme
mpirun: Error: unknown option "--showme"
Type 'mpirun --help' for usage.


Regards,
Mahmood



On Fri, Nov 4, 2016 at 2:12 PM, Gilles Gouaillardet <

Post by Gilles Gouaillardet
You might have to remove -ldl from the scalapack makefile
If it still does not work, can you please post
mpirun --showme ...
output ?
Cheers,
Gilles

[Jeff Squyres (jsquyres)]

Post by Mahmood Naderan

Post by Gilles Gouaillardet
You might have to remove -ldl from the scalapack makefile

I removed that before... I will try one more time
Actually, using --disable-dlopen fixed the error.

To clarify:

1. Using --enable-static causes all the plugins in Open MPI to be "slurped up" into the MPI libraries. I.e., they won't be opened at runtime as plugins -- they're just part of the MPI library.

This does *not* disable Open MPI from trying to dlopen() additional plugins at runtime, though.

2. Using --disable-dlopen *also* causes all the plugins in Open MPI to be "slurped up" into the MPI libraries. It *also* disables Open MPI from trying to dlopen() additional plugins at runtime.

Post by Mahmood Naderan

Post by Gilles Gouaillardet
mpirun --showme

Gilles meant to say:

mpicc --showme
mpifort --showme

-----

If there's a reason you did --enable-static --disable-shared, then since you're having a problem with the dl library, you might as well also --disable-dlopen and then remove the -ldl from where you added it to the Makefile. Open MPI will no longer be using dlopen, but that does not mean that something else isn't using dlopen.

If you don't have a specific reason for using --enable-static, then you might as well not specify *any* of --enable-static, --disable-shared, or --disable-dlopen, and then I'm guessing using mpicc/mpifort in your Makefile will "just work".

[Mahmood Naderan]

​If there's a reason you did --enable-static --disable-shared​

Basically, I want to prevent dynamic library problems (ldd) on a
distributed environment.


​$ mpifort --showme
gfortran -I/opt/openmpi-2.0.1/include -pthread -I/opt/openmpi-2.0.1/lib
-Wl,-rpath -Wl,/opt/openmpi-2.0.1/lib -Wl,--enable-new-dtags
-L/opt/openmpi-2.0.1/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr
-lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lm -lrt -lutil
​
As I said, --disbale-dlopen fixed that error. But, if anybody know how to
have --enable-static --disable-shared with dlopen, please let me know.



Regards,
Mahmood

[Jeff Squyres (jsquyres)]

​If there's a reason you did --enable-static --disable-shared​

Basically, I want to prevent dynamic library problems (ldd) on a distributed environment.

What problems are you referring to?

$ mpifort --showme
gfortran -I/opt/openmpi-2.0.1/include -pthread -I/opt/openmpi-2.0.1/lib -Wl,-rpath -Wl,/opt/openmpi-2.0.1/lib -Wl,--enable-new-dtags -L/opt/openmpi-2.0.1/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte -lopen-pal -lm -lrt -lutil
​
As I said, --disbale-dlopen fixed that error. But, if anybody know how to have --enable-static --disable-shared with dlopen, please let me know.

We might need a bit more detail than that; I use "--enable-static --disable-shared" and I do not get dlopen errors.

When you enable static/disable shared, can you build simple MPI applications (e.g., hello world)? I.e., is the problem with Open MPI, or some kind of effect that happens with building your large/complex application?

--
Jeff Squyres
***@cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

[Mahmood Naderan]

​What problems are you referring to?

I mean errors that are saying failed to load X.so. Then the user has to add
some paths to LD_LIBRARY_PATH. Although such problem can be fixed by adding
an export to the .bashrc, but I prefer to avoid that.

We might need a bit more detail than that; I use "--enable-static

--disable-shared" and I do not get dlopen errors

I also have seen that on Centos. But as I test an application on
Ubuntu-15.04, I saw that error. Maybe on Centos, an external library has
been installed but it is missed on Ubuntu... This is a guess though.


Regards,
Mahmood