Discussion:
[OMPI users] mpif90 unable to find ibverbs
Mahmood Naderan
2017-09-13 08:48:59 UTC
Permalink
Hi,
I am trying to build an application with static linking that uses openmpi.
in the middle of the build, I get this

mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_kinds.o
libiotk.a
/usr/bin/ld: cannot find -libverbs
collect2: ld returned 1 exit status


However, such library exists on the system.

[***@cluster source]# find /usr/ -name *ibverb*
/usr/lib64/libibverbs.so
/usr/lib64/libibverbs.so.1.0.0
/usr/lib64/libibverbs.so.1
/usr/share/doc/libibverbs-1.1.8
[***@cluster source]# mpif90 -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla
--enable-bootstrap --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit
--disable-libunwind-exceptions --enable-gnu-unique-object
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada
--enable-java-awt=gtk --disable-dssi
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
--enable-libgcj-multifile --enable-java-maintainer-mode
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
--with-ppl --with-cloog --with-tune=generic --with-arch_32=i686
--build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)




Any idea for that?
Regards,
Mahmood
g***@rist.or.jp
2017-09-13 10:24:34 UTC
Permalink
Mahmood,

since you are building a static binary, only static library (e.g.
libibverbs.a) can be used.

on your system, only dynamic libibverbs.so is available.

simply install libibverbs.a and you should be fine.

Best regards,

Gilles

----- Original Message -----

Hi,
I am trying to build an application with static linking that uses
openmpi. in the middle of the build, I get this

mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_kinds.o
libiotk.a
/usr/bin/ld: cannot find -libverbs
collect2: ld returned 1 exit status


However, such library exists on the system.

[***@cluster source]# find /usr/ -name *ibverb*
/usr/lib64/libibverbs.so
/usr/lib64/libibverbs.so.1.0.0
/usr/lib64/libibverbs.so.1
/usr/share/doc/libibverbs-1.1.8
[***@cluster source]# mpif90 -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --with-bugurl=
http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared -
-enable-threads=posix --enable-checking=release --with-system-zlib --
enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-
object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-
java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj
-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --
with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
--with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=
x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)




Any idea for that?
Regards,
Mahmood
Mahmood Naderan
2017-09-13 10:56:07 UTC
Permalink
Thanks Gilles... That has been solved. Another issue is

mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_kinds.o
libiotk.a
/usr/bin/ld: cannot find -ldat


The name is actually hard to google! I cannot find the library name for
"dat". Have you heard of that? There is not "libdat" package as I searched.


Regards,
Mahmood
Post by g***@rist.or.jp
Mahmood,
since you are building a static binary, only static library (e.g.
libibverbs.a) can be used.
on your system, only dynamic libibverbs.so is available.
simply install libibverbs.a and you should be fine.
Best regards,
Gilles
----- Original Message -----
Hi,
I am trying to build an application with static linking that uses openmpi.
in the middle of the build, I get this
mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_kinds.o
libiotk.a
/usr/bin/ld: cannot find -libverbs
collect2: ld returned 1 exit status
However, such library exists on the system.
/usr/lib64/libibverbs.so
/usr/lib64/libibverbs.so.1.0.0
/usr/lib64/libibverbs.so.1
/usr/share/doc/libibverbs-1.1.8
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --with-bugurl=http://bugzilla.
redhat.com/bugzilla --enable-bootstrap --enable-shared
--enable-threads=posix --enable-checking=release --with-system-zlib
--enable-__cxa_atexit --disable-libunwind-exceptions
--enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
--enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
--enable-libgcj-multifile --enable-java-maintainer-mode
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
--with-ppl --with-cloog --with-tune=generic --with-arch_32=i686
--build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)
Any idea for that?
Regards,
Mahmood
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
g***@rist.or.jp
2017-09-13 11:21:08 UTC
Permalink
This is something related to DAPL.

/* just google "libdat" */

iirc, Intel MPI uses that, but i do not recall Open MPI using it (!)

are you sure you are using Open MPI ?

which interconnect do you have ?

Cheers,

Gilles

----- Original Message -----

Thanks Gilles... That has been solved. Another issue is

mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_kinds.o
libiotk.a
/usr/bin/ld: cannot find -ldat


The name is actually hard to google! I cannot find the library name
for "dat". Have you heard of that? There is not "libdat" package as I
searched.


Regards,
Mahmood



On Wed, Sep 13, 2017 at 2:54 PM, <***@rist.or.jp> wrote:

Mahmood,



since you are building a static binary, only static library (e.g.
libibverbs.a) can be used.

on your system, only dynamic libibverbs.so is available.



simply install libibverbs.a and you should be fine.



Best regards,



Gilles

----- Original Message -----

Hi,
I am trying to build an application with static linking that
uses openmpi. in the middle of the build, I get this

mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_
kinds.o libiotk.a
/usr/bin/ld: cannot find -libverbs
collect2: ld returned 1 exit status

However, such library exists on the system.

[***@cluster source]# find /usr/ -name *ibverb*
/usr/lib64/libibverbs.so
/usr/lib64/libibverbs.so.1.0.0
/usr/lib64/libibverbs.so.1
/usr/share/doc/libibverbs-1.1.8
[***@cluster source]# mpif90 -v
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/
share/man --infodir=/usr/share/info --with-bugurl=
http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared -
-enable-threads=posix --enable-checking=release --with-system-zlib --
enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-
object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-
java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj
-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --
with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
--with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=
x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)



Any idea for that?
Regards,
Mahmood


_______________________________________________
users mailing list
***@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
Jeff Squyres (jsquyres)
2017-09-13 13:42:49 UTC
Permalink
Beware: static linking is not for the meek.

Is there a reason you need to link statically?

Be sure to read this FAQ item: https://www.open-mpi.org/faq/?category=mpi-apps#static-ofa-mpi-apps (note that that FAQ item was written a long time ago; it cites the "mthca" Mellanox obverts driver; the current generation driver name is ?I think? mlx5). You'll likely also have to adapt those instructions if you're using the UCX or MXM IB libraries.
Post by g***@rist.or.jp
This is something related to DAPL.
/* just google "libdat" */
iirc, Intel MPI uses that, but i do not recall Open MPI using it (!)
are you sure you are using Open MPI ?
which interconnect do you have ?
Cheers,
Gilles
----- Original Message -----
Thanks Gilles... That has been solved. Another issue is
mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_kinds.o libiotk.a
/usr/bin/ld: cannot find -ldat
The name is actually hard to google! I cannot find the library name for "dat". Have you heard of that? There is not "libdat" package as I searched.
Regards,
Mahmood
Mahmood,
since you are building a static binary, only static library (e.g. libibverbs.a) can be used.
on your system, only dynamic libibverbs.so is available.
simply install libibverbs.a and you should be fine.
Best regards,
Gilles
----- Original Message -----
Hi,
I am trying to build an application with static linking that uses openmpi. in the middle of the build, I get this
mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_kinds.o libiotk.a
/usr/bin/ld: cannot find -libverbs
collect2: ld returned 1 exit status
However, such library exists on the system.
/usr/lib64/libibverbs.so
/usr/lib64/libibverbs.so.1.0.0
/usr/lib64/libibverbs.so.1
/usr/share/doc/libibverbs-1.1.8
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)
Any idea for that?
Regards,
Mahmood
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
--
Jeff Squyres
***@cisco.com
Mahmood Naderan
2017-09-13 15:43:54 UTC
Permalink
Post by g***@rist.or.jp
are you sure you are using Open MPI ?
I am using the openmpi shipped with Rocks 6 and trying to build Quantum
ESPRESSO 6.1
Post by g***@rist.or.jp
Beware: static linking is not for the meek.
Agree! I found that I have to install compat-dapl-static.x86_64. As can be
seen from the name, it is a compatibility library. After that, I faced an
error saying


mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_kinds.o
libiotk.a
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libpthread.a(libpthread.o):
In function `sem_open':
(.text+0x774d): warning: the use of `mktemp' is dangerous, better use
`mkstemp'
/usr/bin/ld: dynamic STT_GNU_IFUNC symbol `strcmp' with pointer equality in
`/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libc.a(strcmp.o)'
can not be used when making an executable; recompile with -fPIE and relink
with -pie
collect2: ld returned 1 exit status


With such an error, I thought it is better to forget static linking! (as it
is related to libc) and work with the shared libs and LD_LIBRARY_PATH

Anyway, thanks for your help.

Regards,
Mahmood
Post by g***@rist.or.jp
Beware: static linking is not for the meek.
Is there a reason you need to link statically?
Be sure to read this FAQ item: https://www.open-mpi.org/faq/?
category=mpi-apps#static-ofa-mpi-apps (note that that FAQ item was
written a long time ago; it cites the "mthca" Mellanox obverts driver; the
current generation driver name is ?I think? mlx5). You'll likely also have
to adapt those instructions if you're using the UCX or MXM IB libraries.
Post by g***@rist.or.jp
This is something related to DAPL.
/* just google "libdat" */
iirc, Intel MPI uses that, but i do not recall Open MPI using it (!)
are you sure you are using Open MPI ?
which interconnect do you have ?
Cheers,
Gilles
----- Original Message -----
Thanks Gilles... That has been solved. Another issue is
mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_kinds.o
libiotk.a
Post by g***@rist.or.jp
/usr/bin/ld: cannot find -ldat
The name is actually hard to google! I cannot find the library name for
"dat". Have you heard of that? There is not "libdat" package as I searched.
Post by g***@rist.or.jp
Regards,
Mahmood
Mahmood,
since you are building a static binary, only static library (e.g.
libibverbs.a) can be used.
Post by g***@rist.or.jp
on your system, only dynamic libibverbs.so is available.
simply install libibverbs.a and you should be fine.
Best regards,
Gilles
----- Original Message -----
Hi,
I am trying to build an application with static linking that uses
openmpi. in the middle of the build, I get this
Post by g***@rist.or.jp
mpif90 -g -pthread -static -o iotk_print_kinds.x iotk_print_kinds.o
libiotk.a
Post by g***@rist.or.jp
/usr/bin/ld: cannot find -libverbs
collect2: ld returned 1 exit status
However, such library exists on the system.
/usr/lib64/libibverbs.so
/usr/lib64/libibverbs.so.1.0.0
/usr/lib64/libibverbs.so.1
/usr/share/doc/libibverbs-1.1.8
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --with-bugurl=http://bugzilla.
redhat.com/bugzilla --enable-bootstrap --enable-shared
--enable-threads=posix --enable-checking=release --with-system-zlib
--enable-__cxa_atexit --disable-libunwind-exceptions
--enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
--enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
--enable-libgcj-multifile --enable-java-maintainer-mode
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
--with-ppl --with-cloog --with-tune=generic --with-arch_32=i686
--build=x86_64-redhat-linux
Post by g***@rist.or.jp
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)
Any idea for that?
Regards,
Mahmood
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
--
Jeff Squyres
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Peter Kjellström
2017-09-14 09:43:47 UTC
Permalink
On Wed, 13 Sep 2017 20:13:54 +0430
Mahmood Naderan <***@gmail.com> wrote:
...
Post by Mahmood Naderan
`/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libc.a(strcmp.o)'
can not be used when making an executable; recompile with -fPIE and
relink with -pie collect2: ld returned 1 exit status
With such an error, I thought it is better to forget static linking!
(as it is related to libc) and work with the shared libs and
LD_LIBRARY_PATH
First, I think giving up on static linking is the right choice.

If the main thing you were after was the convenience of a binary that
will run without the need to setup LD_LIBRARY_PATH correctly you should
have a look at passing -rpath to the linker.

In short, "mpicc -Wl,-rpath=/my/lib/path helloworld.c -o hello", will
compile a dynamic binary "hello" with built in search path
to "/my/lib/path".

With OpenMPI this will be added as a "runpath" due to how the wrappers
are designed. Both rpath and runpath works for finding "/my/lib/path"
wihtout LD_LIBRARY_PATH but the difference is in priority. rpath is
higher priority than LD_LIBRARY_PATH etc. and runpath is lower.

You can check your rpath or runpath in a binary using the command
chrpath (package on rhel/centos/... is chrpath):

$ chrpath hello
hello: RUNPATH=/my/lib/path

If what you really wanted is the rpath behavior (winning over any
LD_LIBRARY_PATH in the environment etc.) then you need to modify the
openmpi wrappers (rebuild openmpi) such that it does NOT pass
"--enable-new-dtags" to the linker.

/Peter
Mahmood Naderan
2017-09-14 09:58:08 UTC
Permalink
Post by Peter Kjellström
In short, "mpicc -Wl,-rpath=/my/lib/path helloworld.c -o hello", will
compile a dynamic binary "hello" with built in search path
to "/my/lib/path".
Excuse me... Is that a path or file? I get this:

mpif90 -g -pthread -Wl,rpath=/share/apps/computer/OpenBLAS-0.2.18 -o
iotk_print_kinds.x iotk_print_kinds.o libiotk.a
/usr/bin/ld: rpath=/share/apps/computer/OpenBLAS-0.2.18: No such file: No
such file or directory
collect2: ld returned 1 exit status


However, the lib files are there.

[***@cluster source]# ls -l
/share/apps/computer/OpenBLAS-0.2.18/libopenblas*
lrwxrwxrwx 1 nfsnobody nfsnobody 32 Sep 8 14:40
/share/apps/computer/OpenBLAS-0.2.18/libopenblas.a ->
libopenblas_bulldozerp-r0.2.18.a
-rw-r--r-- 1 nfsnobody nfsnobody 28075178 Sep 8 14:41
/share/apps/computer/OpenBLAS-0.2.18/libopenblas_bulldozerp-r0.2.18.a
-rwxr-xr-x 1 nfsnobody nfsnobody 14906048 Sep 8 14:41
/share/apps/computer/OpenBLAS-0.2.18/libopenblas_bulldozerp-r0.2.18.so
lrwxrwxrwx 1 nfsnobody nfsnobody 33 Sep 8 14:41
/share/apps/computer/OpenBLAS-0.2.18/libopenblas.so ->
libopenblas_bulldozerp-r0.2.18.so
lrwxrwxrwx 1 nfsnobody nfsnobody 33 Sep 8 14:41
/share/apps/computer/OpenBLAS-0.2.18/libopenblas.so.0 ->
libopenblas_bulldozerp-r0.2.18.so



Please note that, I added that option to the linker section of make.inc
from ESPRESSO

# compiler flags: C, F90, F77
# C flags must include DFLAGS and IFLAGS
# F90 flags must include MODFLAGS, IFLAGS, and FDFLAGS with appropriate
syntax
CFLAGS = -O3 $(DFLAGS) $(IFLAGS)
F90FLAGS = $(FFLAGS) -x f95-cpp-input $(FDFLAGS) $(IFLAGS) $(MODFLAGS)
FFLAGS = -O3 -g
# Linker, linker-specific flags (if any)
# Typically LD coincides with F90 or MPIF90, LD_LIBS is empty
LD = mpif90
LDFLAGS = -g -pthread -Wl,rpath=/share/apps/computer/OpenBLAS-0.2.18
LD_LIBS =


Regards,
Mahmood
Gilles Gouaillardet
2017-09-14 10:51:52 UTC
Permalink
Mahmood,

there is a typo, it should be
-Wl,-rpath,/.../

(note the minus before rpath)


Cheers,

Gilles
Post by Mahmood Naderan
Post by Peter Kjellström
In short, "mpicc -Wl,-rpath=/my/lib/path helloworld.c -o hello", will
compile a dynamic binary "hello" with built in search path
to "/my/lib/path".
mpif90 -g -pthread -Wl,rpath=/share/apps/computer/OpenBLAS-0.2.18 -o
iotk_print_kinds.x iotk_print_kinds.o libiotk.a
/usr/bin/ld: rpath=/share/apps/computer/OpenBLAS-0.2.18: No such file: No
such file or directory
collect2: ld returned 1 exit status
However, the lib files are there.
/share/apps/computer/OpenBLAS-0.2.18/libopenblas*
lrwxrwxrwx 1 nfsnobody nfsnobody 32 Sep 8 14:40
/share/apps/computer/OpenBLAS-0.2.18/libopenblas.a ->
libopenblas_bulldozerp-r0.2.18.a
-rw-r--r-- 1 nfsnobody nfsnobody 28075178 Sep 8 14:41
/share/apps/computer/OpenBLAS-0.2.18/libopenblas_bulldozerp-r0.2.18.a
-rwxr-xr-x 1 nfsnobody nfsnobody 14906048 Sep 8 14:41
/share/apps/computer/OpenBLAS-0.2.18/libopenblas_bulldozerp-r0.2.18.so
lrwxrwxrwx 1 nfsnobody nfsnobody 33 Sep 8 14:41
/share/apps/computer/OpenBLAS-0.2.18/libopenblas.so ->
libopenblas_bulldozerp-r0.2.18.so
lrwxrwxrwx 1 nfsnobody nfsnobody 33 Sep 8 14:41
/share/apps/computer/OpenBLAS-0.2.18/libopenblas.so.0 ->
libopenblas_bulldozerp-r0.2.18.so
Please note that, I added that option to the linker section of make.inc from
ESPRESSO
# compiler flags: C, F90, F77
# C flags must include DFLAGS and IFLAGS
# F90 flags must include MODFLAGS, IFLAGS, and FDFLAGS with appropriate
syntax
CFLAGS = -O3 $(DFLAGS) $(IFLAGS)
F90FLAGS = $(FFLAGS) -x f95-cpp-input $(FDFLAGS) $(IFLAGS) $(MODFLAGS)
FFLAGS = -O3 -g
# Linker, linker-specific flags (if any)
# Typically LD coincides with F90 or MPIF90, LD_LIBS is empty
LD = mpif90
LDFLAGS = -g -pthread -Wl,rpath=/share/apps/computer/OpenBLAS-0.2.18
LD_LIBS =
Regards,
Mahmood
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Peter Kjellström
2017-09-14 11:00:17 UTC
Permalink
On Thu, 14 Sep 2017 14:28:08 +0430
Post by Mahmood Naderan
Post by Peter Kjellström
In short, "mpicc -Wl,-rpath=/my/lib/path helloworld.c -o hello", will
compile a dynamic binary "hello" with built in search path
to "/my/lib/path".
It should be a path ie. directory.
Post by Mahmood Naderan
mpif90 -g -pthread -Wl,rpath=/share/apps/computer/OpenBLAS-0.2.18 -o
iotk_print_kinds.x iotk_print_kinds.o libiotk.a
/usr/bin/ld: rpath=/share/apps/computer/OpenBLAS-0.2.18: No such
file: No such file or directory
I think it didn't like passing "-rpath=/a/b/c" in one chunk. Try
this variant:

-Wl,-rpath,/path/to/directory

or even:

-Wl,-rpath -Wl,/path/to/directory

/Peter
Gilles Gouaillardet
2017-09-14 10:01:08 UTC
Permalink
Peter and all,

an easier option is to configure Open MPI with --mpirun-prefix-by-default
this will automagically add rpath to the libs.

Cheers,

Gilles
Post by Peter Kjellström
On Wed, 13 Sep 2017 20:13:54 +0430
...
Post by Mahmood Naderan
`/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/libc.a(strcmp.o)'
can not be used when making an executable; recompile with -fPIE and
relink with -pie collect2: ld returned 1 exit status
With such an error, I thought it is better to forget static linking!
(as it is related to libc) and work with the shared libs and
LD_LIBRARY_PATH
First, I think giving up on static linking is the right choice.
If the main thing you were after was the convenience of a binary that
will run without the need to setup LD_LIBRARY_PATH correctly you should
have a look at passing -rpath to the linker.
In short, "mpicc -Wl,-rpath=/my/lib/path helloworld.c -o hello", will
compile a dynamic binary "hello" with built in search path
to "/my/lib/path".
With OpenMPI this will be added as a "runpath" due to how the wrappers
are designed. Both rpath and runpath works for finding "/my/lib/path"
wihtout LD_LIBRARY_PATH but the difference is in priority. rpath is
higher priority than LD_LIBRARY_PATH etc. and runpath is lower.
You can check your rpath or runpath in a binary using the command
$ chrpath hello
hello: RUNPATH=/my/lib/path
If what you really wanted is the rpath behavior (winning over any
LD_LIBRARY_PATH in the environment etc.) then you need to modify the
openmpi wrappers (rebuild openmpi) such that it does NOT pass
"--enable-new-dtags" to the linker.
/Peter
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Mahmood Naderan
2017-09-14 10:14:19 UTC
Permalink
Well that may be good if someone intend to rebuild ompi.
Lets say, there is an ompi on the system...

Regards,
Mahmood



On Thu, Sep 14, 2017 at 2:31 PM, Gilles Gouaillardet <
Post by Gilles Gouaillardet
Peter and all,
an easier option is to configure Open MPI with --mpirun-prefix-by-default
this will automagically add rpath to the libs.
Cheers,
Gilles
Post by Peter Kjellström
On Wed, 13 Sep 2017 20:13:54 +0430
...
Post by Mahmood Naderan
`/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/
libc.a(strcmp.o)'
Post by Peter Kjellström
Post by Mahmood Naderan
can not be used when making an executable; recompile with -fPIE and
relink with -pie collect2: ld returned 1 exit status
With such an error, I thought it is better to forget static linking!
(as it is related to libc) and work with the shared libs and
LD_LIBRARY_PATH
First, I think giving up on static linking is the right choice.
If the main thing you were after was the convenience of a binary that
will run without the need to setup LD_LIBRARY_PATH correctly you should
have a look at passing -rpath to the linker.
In short, "mpicc -Wl,-rpath=/my/lib/path helloworld.c -o hello", will
compile a dynamic binary "hello" with built in search path
to "/my/lib/path".
With OpenMPI this will be added as a "runpath" due to how the wrappers
are designed. Both rpath and runpath works for finding "/my/lib/path"
wihtout LD_LIBRARY_PATH but the difference is in priority. rpath is
higher priority than LD_LIBRARY_PATH etc. and runpath is lower.
You can check your rpath or runpath in a binary using the command
$ chrpath hello
hello: RUNPATH=/my/lib/path
If what you really wanted is the rpath behavior (winning over any
LD_LIBRARY_PATH in the environment etc.) then you need to modify the
openmpi wrappers (rebuild openmpi) such that it does NOT pass
"--enable-new-dtags" to the linker.
/Peter
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Mahmood Naderan
2017-09-14 10:42:09 UTC
Permalink
So it seems that -rpath is not available with 1.4 which is ompi came with
rocks 6.

Regards,
Mahmood
Post by Mahmood Naderan
Well that may be good if someone intend to rebuild ompi.
Lets say, there is an ompi on the system...
Regards,
Mahmood
On Thu, Sep 14, 2017 at 2:31 PM, Gilles Gouaillardet <
Post by Gilles Gouaillardet
Peter and all,
an easier option is to configure Open MPI with --mpirun-prefix-by-default
this will automagically add rpath to the libs.
Cheers,
Gilles
Post by Peter Kjellström
On Wed, 13 Sep 2017 20:13:54 +0430
...
Post by Mahmood Naderan
`/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/li
bc.a(strcmp.o)'
Post by Peter Kjellström
Post by Mahmood Naderan
can not be used when making an executable; recompile with -fPIE and
relink with -pie collect2: ld returned 1 exit status
With such an error, I thought it is better to forget static linking!
(as it is related to libc) and work with the shared libs and
LD_LIBRARY_PATH
First, I think giving up on static linking is the right choice.
If the main thing you were after was the convenience of a binary that
will run without the need to setup LD_LIBRARY_PATH correctly you should
have a look at passing -rpath to the linker.
In short, "mpicc -Wl,-rpath=/my/lib/path helloworld.c -o hello", will
compile a dynamic binary "hello" with built in search path
to "/my/lib/path".
With OpenMPI this will be added as a "runpath" due to how the wrappers
are designed. Both rpath and runpath works for finding "/my/lib/path"
wihtout LD_LIBRARY_PATH but the difference is in priority. rpath is
higher priority than LD_LIBRARY_PATH etc. and runpath is lower.
You can check your rpath or runpath in a binary using the command
$ chrpath hello
hello: RUNPATH=/my/lib/path
If what you really wanted is the rpath behavior (winning over any
LD_LIBRARY_PATH in the environment etc.) then you need to modify the
openmpi wrappers (rebuild openmpi) such that it does NOT pass
"--enable-new-dtags" to the linker.
/Peter
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
John Hearns via users
2017-09-14 11:22:43 UTC
Permalink
Then let me add in my thoughts please.. Rocks is getting out of date.
Mahmood, I would imagine that you are not given the choice of installing
something more modern,
ie the place where you work has an existing Rocks cluster and is unwilling
to re-install it.

So what is wrong with using the 'module load openmpi' which you would
normally do in a job?

In a more constructive fashion, the University of California at San Diego
make available Rocks Rolls
which have a much more modern software envoronment.

https://github.com/sdsc/cluster-guide

https://github.com/sdsc/mpi-roll

Mahmood, I would ask your systems team if they are willing to install these
rolls.
Post by Mahmood Naderan
So it seems that -rpath is not available with 1.4 which is ompi came with
rocks 6.
Regards,
Mahmood
Post by Mahmood Naderan
Well that may be good if someone intend to rebuild ompi.
Lets say, there is an ompi on the system...
Regards,
Mahmood
On Thu, Sep 14, 2017 at 2:31 PM, Gilles Gouaillardet <
Post by Gilles Gouaillardet
Peter and all,
an easier option is to configure Open MPI with --mpirun-prefix-by-default
this will automagically add rpath to the libs.
Cheers,
Gilles
Post by Peter Kjellström
On Wed, 13 Sep 2017 20:13:54 +0430
...
Post by Mahmood Naderan
`/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/li
bc.a(strcmp.o)'
Post by Peter Kjellström
Post by Mahmood Naderan
can not be used when making an executable; recompile with -fPIE and
relink with -pie collect2: ld returned 1 exit status
With such an error, I thought it is better to forget static linking!
(as it is related to libc) and work with the shared libs and
LD_LIBRARY_PATH
First, I think giving up on static linking is the right choice.
If the main thing you were after was the convenience of a binary that
will run without the need to setup LD_LIBRARY_PATH correctly you should
have a look at passing -rpath to the linker.
In short, "mpicc -Wl,-rpath=/my/lib/path helloworld.c -o hello", will
compile a dynamic binary "hello" with built in search path
to "/my/lib/path".
With OpenMPI this will be added as a "runpath" due to how the wrappers
are designed. Both rpath and runpath works for finding "/my/lib/path"
wihtout LD_LIBRARY_PATH but the difference is in priority. rpath is
higher priority than LD_LIBRARY_PATH etc. and runpath is lower.
You can check your rpath or runpath in a binary using the command
$ chrpath hello
hello: RUNPATH=/my/lib/path
If what you really wanted is the rpath behavior (winning over any
LD_LIBRARY_PATH in the environment etc.) then you need to modify the
openmpi wrappers (rebuild openmpi) such that it does NOT pass
"--enable-new-dtags" to the linker.
/Peter
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Mahmood Naderan
2017-09-14 11:34:19 UTC
Permalink
Fortunately Rocks 7 beta is released. So, there is a hope that newer
version will be born one day.
https://github.com/sdsc/mpi-roll <https://github.com/sdsc/mpi-roll>
I wasn't aware of that. Thanks for sharing it.
there is a typo, it should be
-Wl,-rpath,/.../
Thanks a lot. It is now working.

$ chrpath /share/apps/chemistry/qe-6.1/bin/pw.x
/share/apps/chemistry/qe-6.1/bin/pw.x:
RPATH=/share/apps/computer/OpenBLAS-0.2.18
Peter Kjellström
2017-09-14 12:21:29 UTC
Permalink
On Thu, 14 Sep 2017 19:01:08 +0900
Post by Gilles Gouaillardet
Peter and all,
an easier option is to configure Open MPI with
--mpirun-prefix-by-default this will automagically add rpath to the
libs.
Yes that sorts out the OpenMPI libs but I was imagining a more general
situation (and the OP later tried adding openblas).

It's also only available if the OpenMPI in question is built with it
or if you can rebuild OpenMPI.

The OP seems at least partially interested in additional libraries and
not rebuilding the system provided OpenMPI.

/Peter
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Jeff Squyres (jsquyres)
2017-09-14 14:48:39 UTC
Permalink
Let me throw in one more item: I don't know what versions of Open MPI are available in those Rocks Rolls, but Open MPI v3.0.0 was released yesterday. You will be much better served with a modern version of Open MPI (vs. v1.4, the last release of which was in 2012).
Post by Peter Kjellström
On Thu, 14 Sep 2017 19:01:08 +0900
Post by Gilles Gouaillardet
Peter and all,
an easier option is to configure Open MPI with
--mpirun-prefix-by-default this will automagically add rpath to the
libs.
Yes that sorts out the OpenMPI libs but I was imagining a more general
situation (and the OP later tried adding openblas).
It's also only available if the OpenMPI in question is built with it
or if you can rebuild OpenMPI.
The OP seems at least partially interested in additional libraries and
not rebuilding the system provided OpenMPI.
/Peter
--
Sent from my Android device with K-9 Mail. Please excuse my brevity._______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
--
Jeff Squyres
***@cisco.com
John Hearns via users
2017-09-14 15:25:29 UTC
Permalink
Jeff, from what I read yesterday it is OpenMPI 2 , I am not sure of the
minor version.
I do acknowledge that Mahmood reports that the Rocks 7 beta is available -
when I last used Rocks this was not avaiable.
But still - look at something more up to date, such as OpenHPC.
There is nothing intrinsically tied up between hardware and the operating
system / software stack.
Post by Jeff Squyres (jsquyres)
Let me throw in one more item: I don't know what versions of Open MPI are
available in those Rocks Rolls, but Open MPI v3.0.0 was released
yesterday. You will be much better served with a modern version of Open
MPI (vs. v1.4, the last release of which was in 2012).
Post by Peter Kjellström
On Thu, 14 Sep 2017 19:01:08 +0900
Post by Gilles Gouaillardet
Peter and all,
an easier option is to configure Open MPI with
--mpirun-prefix-by-default this will automagically add rpath to the
libs.
Yes that sorts out the OpenMPI libs but I was imagining a more general
situation (and the OP later tried adding openblas).
It's also only available if the OpenMPI in question is built with it
or if you can rebuild OpenMPI.
The OP seems at least partially interested in additional libraries and
not rebuilding the system provided OpenMPI.
/Peter
--
Sent from my Android device with K-9 Mail. Please excuse my
brevity._______________________________________________
Post by Peter Kjellström
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
--
Jeff Squyres
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
Loading...