[OMPI users] 50% performance reduction due to OpenMPI v 1.3.2 forcing all MPI traffic over Ethernet instead of using Infiniband

For the app I am using, ORCA (a Quantum Chemistry program), when it was
compiled using openMPI 1.2.8 and run under 1.2.8 with the following in
the openmpi-mca-params.conf file:

btl=self,openib

the app ran fine with no traffic over my Ethernet network and all
traffic over my Infiniband network.

However, now that ORCA has been recompiled with openMPI v1.3.2 and run
under 1.3.2 (using the same openmpi-mca-params.conf file), the
performance has been reduced by 50% and all the MPI traffic is going
over the Ethernet network.

As a matter of fact, the openMPI v1.3.2 performance now looks exactly
like the performance I get if I use MPICH 1.2.7.

Anyone have any ideas:

1) How could this have happened?

2) How can I fix it?

a 50% reduction in performance is just not acceptable. Ideas/
suggestions would be appreciated.

Jim

Ralph Castain

2009-06-23 01:40:10 UTC

Sounds very strange, indeed. You might want to check that your app is
actually getting the MCA param that you think it is. Try adding:

-mca mpi_show_mca_params file,env

to your cmd line. This will cause rank=0 to output the MCA params it
thinks were set via the default files and/or environment (including
cmd line).

Ralph

Post by Jim Kress ORG
For the app I am using, ORCA (a Quantum Chemistry program), when it was
compiled using openMPI 1.2.8 and run under 1.2.8 with the following in
btl=self,openib
the app ran fine with no traffic over my Ethernet network and all
traffic over my Infiniband network.
However, now that ORCA has been recompiled with openMPI v1.3.2 and run
under 1.3.2 (using the same openmpi-mca-params.conf file), the
performance has been reduced by 50% and all the MPI traffic is going
over the Ethernet network.
As a matter of fact, the openMPI v1.3.2 performance now looks exactly
like the performance I get if I use MPICH 1.2.7.
1) How could this have happened?
2) How can I fix it?
a 50% reduction in performance is just not acceptable. Ideas/
suggestions would be appreciated.
Jim
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress ORG

2009-06-23 02:08:54 UTC

Thanks for the advice. Unfortunately the command line is internally
generated by the app and then invoked so I can't see it.

But, it doesn't matter anyway. It seems the Ethernet utilization
"problem" I thought I had does not exist.

So, I'm still looking for why my app using 1.2.8 is 50% faster than
using 1.3.2.

Jim

Post by Ralph Castain
Sounds very strange, indeed. You might want to check that your app is
-mca mpi_show_mca_params file,env
to your cmd line. This will cause rank=0 to output the MCA params it
thinks were set via the default files and/or environment (including
cmd line).
Ralph

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress ORG

2009-06-23 02:36:26 UTC

Is there an environment variable (or variables) I can set to do the
equivalent?

Jim

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Don Kerr

2009-06-23 02:56:38 UTC

Post by Jim Kress ORG
Is there an environment variable (or variables) I can set to do the
equivalent?

OMPI_MCA_mpi_show_mca_params

see: http://www.open-mpi.org/faq/?category=tuning#setting-mca-params

Post by Jim Kress ORG
Jim

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress ORG

2009-06-23 20:31:22 UTC

Ralph,

I did the following:

export OMPI_MCA_mpi_show_mca_params="file,env"

then I checked and found it via the set command as

OMPI_MCA_mpi_show_mca_params=file,env

I then ran my application

./orca hexatriene_TDDFT_get_asa_input_parallel_1.inp >
1.2.8_test_crafted_input_file.out

and got the expected ORCA output in the .out file but nothing at the
command line or in the .out file about mca_params

What did I do wrong?

Jim

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Ralph Castain

2009-06-23 21:20:58 UTC

Hmmm...just to be clear - did you run this against OMPI 1.3.2, or
1.2.8? I see a 1.2.8 in your app name, hence the question.

This option only works with 1.3.2, I'm afraid - it was a new feature.

Ralph

Post by Jim Kress ORG
Ralph,
export OMPI_MCA_mpi_show_mca_params="file,env"
then I checked and found it via the set command as
OMPI_MCA_mpi_show_mca_params=file,env
I then ran my application
./orca hexatriene_TDDFT_get_asa_input_parallel_1.inp >
1.2.8_test_crafted_input_file.out
and got the expected ORCA output in the .out file but nothing at the
command line or in the .out file about mca_params
What did I do wrong?
Jim

Post by Jim Kress ORG
For the app I am using, ORCA (a Quantum Chemistry program), when it was
compiled using openMPI 1.2.8 and run under 1.2.8 with the
following in
btl=self,openib
the app ran fine with no traffic over my Ethernet network and all
traffic over my Infiniband network.
However, now that ORCA has been recompiled with openMPI v1.3.2 and run
under 1.3.2 (using the same openmpi-mca-params.conf file), the
performance has been reduced by 50% and all the MPI traffic is going
over the Ethernet network.
As a matter of fact, the openMPI v1.3.2 performance now looks exactly
like the performance I get if I use MPICH 1.2.7.
1) How could this have happened?
2) How can I fix it?
a 50% reduction in performance is just not acceptable. Ideas/
suggestions would be appreciated.
Jim
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress ORG

2009-06-23 21:42:06 UTC

Sorry about the size of the last email. I wasn't aware the log file
would be so lagre ...

Jim

Post by Ralph Castain
Hmmm...just to be clear - did you run this against OMPI 1.3.2, or
1.2.8? I see a 1.2.8 in your app name, hence the question.
This option only works with 1.3.2, I'm afraid - it was a new feature.
Ralph

Post by Jim Kress ORG
For the app I am using, ORCA (a Quantum Chemistry program), when it was
compiled using openMPI 1.2.8 and run under 1.2.8 with the
following in
btl=self,openib
the app ran fine with no traffic over my Ethernet network and all
traffic over my Infiniband network.
However, now that ORCA has been recompiled with openMPI v1.3.2 and run
under 1.3.2 (using the same openmpi-mca-params.conf file), the
performance has been reduced by 50% and all the MPI traffic is going
over the Ethernet network.
As a matter of fact, the openMPI v1.3.2 performance now looks exactly
like the performance I get if I use MPICH 1.2.7.
1) How could this have happened?
2) How can I fix it?
a 50% reduction in performance is just not acceptable. Ideas/
suggestions would be appreciated.
Jim
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress ORG

2009-06-23 21:49:44 UTC

For v 1.3.2:

Here is the ompi_info -config output and I've attached a copy of the
config.log file which seems to clearly indicate it found the infiniband
libraries.

[***@master ~]# ompi_info -config
Configured by: root
Configured on: Sun Jun 21 22:02:59 EDT 2009
Configure host: master.org
Built by: root
Built on: Sun Jun 21 22:10:07 EDT 2009
Built host: master.org
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C char size: 1
C bool size: 1
C short size: 2
C int size: 4
C long size: 8
C float size: 4
C double size: 8
C pointer size: 8
C char align: 1
C bool align: 1
C int align: 4
C float align: 4
C double align: 8
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: gfortran
Fortran77 compiler abs: /usr/bin/gfortran
Fortran90 compiler: gfortran
Fortran90 compiler abs: /usr/bin/gfortran
Fort integer size: 4
Fort logical size: 4
Fort logical value true: 1
Fort have integer1: yes
Fort have integer2: yes
Fort have integer4: yes
Fort have integer8: yes
Fort have integer16: no
Fort have real4: yes
Fort have real8: yes
Fort have real16: no
Fort have complex8: yes
Fort have complex16: yes
Fort have complex32: no
Fort integer1 size: 1
Fort integer2 size: 2
Fort integer4 size: 4
Fort integer8 size: 8
Fort integer16 size: -1
Fort real size: 4
Fort real4 size: 4
Fort real8 size: 8
Fort real16 size: -1
Fort dbl prec size: 4
Fort cplx size: 4
Fort dbl cplx size: 4
Fort cplx8 size: 8
Fort cplx16 size: 16
Fort cplx32 size: -1
Fort integer align: 4
Fort integer1 align: 1
Fort integer2 align: 2
Fort integer4 align: 4
Fort integer8 align: 8
Fort integer16 align: -1
Fort real align: 4
Fort real4 align: 4
Fort real8 align: 8
Fort real16 align: -1
Fort dbl prec align: 4
Fort cplx align: 4
Fort dbl cplx align: 4
Fort cplx8 align: 4
Fort cplx16 align: 8
Fort cplx32 align: -1
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Sparse Groups: no
Build CFLAGS: -O3 -DNDEBUG -finline-functions
-fno-strict-aliasing
-pthread -fvisibility=hidden
Build CXXFLAGS: -O3 -DNDEBUG -finline-functions -pthread
Build FFLAGS:
Build FCFLAGS:
Build LDFLAGS: -export-dynamic
Build LIBS: -lnsl -lutil -lm
Wrapper extra CFLAGS: -pthread
Wrapper extra CXXFLAGS: -pthread
Wrapper extra FFLAGS: -pthread
Wrapper extra FCFLAGS: -pthread
Wrapper extra LDFLAGS:
Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -lm
-ldl
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: no
mpirun default --prefix: no
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol visibility support: yes
FT Checkpoint support: no (checkpoint thread: no)

Post by Jim Kress ORG
For the app I am using, ORCA (a Quantum Chemistry program), when it was
compiled using openMPI 1.2.8 and run under 1.2.8 with the
following in
btl=self,openib
the app ran fine with no traffic over my Ethernet network and all
traffic over my Infiniband network.
However, now that ORCA has been recompiled with openMPI v1.3.2 and run
under 1.3.2 (using the same openmpi-mca-params.conf file), the
performance has been reduced by 50% and all the MPI traffic is going
over the Ethernet network.
As a matter of fact, the openMPI v1.3.2 performance now looks exactly
like the performance I get if I use MPICH 1.2.7.
1) How could this have happened?
2) How can I fix it?
a 50% reduction in performance is just not acceptable. Ideas/
suggestions would be appreciated.
Jim
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Gus Correa

2009-06-23 22:19:47 UTC

Hi Jim, list

On my OpenMPI 1.3.2 ompi_info -config gives:

Wrapper extra LIBS: -lrdmacm -libverbs -ltorque -lnuma -ldl
-Wl,--export-dynamic -lnsl -lutil -lm -ldl

Yours doesn't seem to have the IB libraries: -lrdmacm -libverbs

So, I would guess your OpenMPI 1.3.2 build doesn't have IB support.

That is that is the point I and Scott Hamilton (in the Rocks list)
have been pointing out to you as a possible reason for your problem.

FYI, the OpenMPI configure was changed somewhere between 1.2.8
and 1.3.2 in the way it searches for libraries.
Your configure command line for 1.2.8 may not work as you expect
in 1.3.2.
If your IB libraries are in a non-standard location, you may
need to point to it explicitly.
See the README file and the FAQs.

Just to make sure, I suggest you dump the OpenMPI configure output
to a file, then search it for the IB libraries.
This helped me troubleshoot similar problems (with libnuma) here.

I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Post by Jim Kress ORG
Here is the ompi_info -config output and I've attached a copy of the
config.log file which seems to clearly indicate it found the infiniband
libraries.
Configured by: root
Configured on: Sun Jun 21 22:02:59 EDT 2009
Configure host: master.org
Built by: root
Built on: Sun Jun 21 22:10:07 EDT 2009
Built host: master.org
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C char size: 1
C bool size: 1
C short size: 2
C int size: 4
C long size: 8
C float size: 4
C double size: 8
C pointer size: 8
C char align: 1
C bool align: 1
C int align: 4
C float align: 4
C double align: 8
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: gfortran
Fortran77 compiler abs: /usr/bin/gfortran
Fortran90 compiler: gfortran
Fortran90 compiler abs: /usr/bin/gfortran
Fort integer size: 4
Fort logical size: 4
Fort logical value true: 1
Fort have integer1: yes
Fort have integer2: yes
Fort have integer4: yes
Fort have integer8: yes
Fort have integer16: no
Fort have real4: yes
Fort have real8: yes
Fort have real16: no
Fort have complex8: yes
Fort have complex16: yes
Fort have complex32: no
Fort integer1 size: 1
Fort integer2 size: 2
Fort integer4 size: 4
Fort integer8 size: 8
Fort integer16 size: -1
Fort real size: 4
Fort real4 size: 4
Fort real8 size: 8
Fort real16 size: -1
Fort dbl prec size: 4
Fort cplx size: 4
Fort dbl cplx size: 4
Fort cplx8 size: 8
Fort cplx16 size: 16
Fort cplx32 size: -1
Fort integer align: 4
Fort integer1 align: 1
Fort integer2 align: 2
Fort integer4 align: 4
Fort integer8 align: 8
Fort integer16 align: -1
Fort real align: 4
Fort real4 align: 4
Fort real8 align: 8
Fort real16 align: -1
Fort dbl prec align: 4
Fort cplx align: 4
Fort dbl cplx align: 4
Fort cplx8 align: 4
Fort cplx16 align: 8
Fort cplx32 align: -1
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Sparse Groups: no
Build CFLAGS: -O3 -DNDEBUG -finline-functions
-fno-strict-aliasing
-pthread -fvisibility=hidden
Build CXXFLAGS: -O3 -DNDEBUG -finline-functions -pthread
Build LDFLAGS: -export-dynamic
Build LIBS: -lnsl -lutil -lm
Wrapper extra CFLAGS: -pthread
Wrapper extra CXXFLAGS: -pthread
Wrapper extra FFLAGS: -pthread
Wrapper extra FCFLAGS: -pthread
Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -lm
-ldl
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: no
mpirun default --prefix: no
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol visibility support: yes
FT Checkpoint support: no (checkpoint thread: no)

Post by Jim Kress ORG
For the app I am using, ORCA (a Quantum Chemistry program), when it was
compiled using openMPI 1.2.8 and run under 1.2.8 with the
following in
btl=self,openib
the app ran fine with no traffic over my Ethernet network and all
traffic over my Infiniband network.
However, now that ORCA has been recompiled with openMPI v1.3.2 and run
under 1.3.2 (using the same openmpi-mca-params.conf file), the
performance has been reduced by 50% and all the MPI traffic is going
over the Ethernet network.
As a matter of fact, the openMPI v1.3.2 performance now looks exactly
like the performance I get if I use MPICH 1.2.7.
1) How could this have happened?
2) How can I fix it?
a 50% reduction in performance is just not acceptable. Ideas/
suggestions would be appreciated.
Jim
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users
------------------------------------------------------------------------
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jeff Squyres

2009-06-23 22:51:57 UTC

Post by Gus Correa
Wrapper extra LIBS: -lrdmacm -libverbs -ltorque -lnuma -ldl
-Wl,--export-dynamic -lnsl -lutil -lm -ldl
Yours doesn't seem to have the IB libraries: -lrdmacm -libverbs

To check definitively:

$ ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.2)

You should see the "btl: openib" line if your Open MPI has OpenFabrics
support. This shows that the openib (OpenFabrics) plugin is installed
and at least marginally operating correctly.

--
Jeff Squyres
Cisco Systems

Jim Kress ORG

2009-06-23 23:25:49 UTC

This is what I get

[***@master ~]# ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.2)
[***@master ~]#

Jim

Post by Jeff Squyres
openib (OpenFabrics) plugin is installed
and at least marginally opera

Jeff Squyres

2009-06-23 23:29:11 UTC

You mentioned that you only have a binary for your executable. Was it
compiled / linked against v1.3.2?

We did not introduce ABI compatibility until v1.3.2 -- if the
executable was compiled/linked against any version prior to that, it's
pure luck that it works with the 1.3.2 shared libraries at all.

Post by Jim Kress ORG
This is what I get
MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.2)
Jim

Post by Jeff Squyres
openib (OpenFabrics) plugin is installed
and at least marginally opera

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

--
Jeff Squyres
Cisco Systems

Jim Kress ORG

2009-06-24 01:02:28 UTC

According to the author(s) it was compiled/linked against v1.3.2

Jim

Post by Jeff Squyres
You mentioned that you only have a binary for your executable. Was it
compiled / linked against v1.3.2?
We did not introduce ABI compatibility until v1.3.2 -- if the
executable was compiled/linked against any version prior to that, it's
pure luck that it works with the 1.3.2 shared libraries at all.

Post by Jim Kress ORG
This is what I get
MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.2)
Jim

Post by Jeff Squyres
openib (OpenFabrics) plugin is installed
and at least marginally opera

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Noam Bernstein

2009-06-24 13:37:35 UTC

Post by Gus Correa
Hi Jim, list
Wrapper extra LIBS: -lrdmacm -libverbs -ltorque -lnuma -ldl -Wl,--
export-dynamic -lnsl -lutil -lm -ldl
Yours doesn't seem to have the IB libraries: -lrdmacm -libverbs
So, I would guess your OpenMPI 1.3.2 build doesn't have IB support.

The second of these statements doesn't follow from the first.

My "ompi_info -config" returns

ompi_info -config | grep LIBS
Build LIBS: -lnsl -lutil -lm
Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -
lm -ldl

But it does have openib

ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0, Component v1.3.2)

and osu_bibw returns

# OSU MPI Bi-Directional Bandwidth Test v3.0
# Size Bi-Bandwidth (MB/s)
4194304 1717.43

which it's sure not getting over ethernet. I think Jeff Squyres' test
(ompi_info | grep openib) must be more definitive.

Noam

Jim Kress

2009-06-24 15:05:13 UTC

Noam, Gus and List,

Did you statically link your openmpi when you built it? If you did (the
default is NOT to do this) then that could explain the discrepancy.

Jim

-----Original Message-----
Sent: Wednesday, June 24, 2009 9:38 AM
To: Open MPI Users
Subject: Re: [OMPI users] 50% performance reduction due to
OpenMPI v 1.3.2forcing all MPI traffic over Ethernet instead
of using Infiniband

The second of these statements doesn't follow from the first.
My "ompi_info -config" returns
ompi_info -config | grep LIBS
Build LIBS: -lnsl -lutil -lm
Wrapper extra LIBS: -ldl -Wl,--export-dynamic
-lnsl -lutil -
lm -ldl
But it does have openib
ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0,
Component v1.3.2)
and osu_bibw returns
# OSU MPI Bi-Directional Bandwidth Test v3.0
# Size Bi-Bandwidth (MB/s)
4194304 1717.43
which it's sure not getting over ethernet. I think Jeff
Squyres' test (ompi_info | grep openib) must be more definitive.
Noam
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Noam Bernstein

2009-06-24 15:18:09 UTC

Post by Jim Kress
Noam, Gus and List,
Did you statically link your openmpi when you built it? If you did (the
default is NOT to do this) then that could explain the discrepancy.

Not explicitly:

env CC=gcc CXX=g++ F77=ifort FC=ifort ./configure --prefix=/share/apps/
mpi/openmpi-$VER/intel-11.0.083 --with-sge

Noam

Gus Correa

2009-06-24 21:22:56 UTC

Hi Jim

Post by Jim Kress
Noam, Gus and List,
Did you statically link your openmpi when you built it? If you did (the
default is NOT to do this) then that could explain the discrepancy.
Jim

No, I didn't link statically.

Did you link statically?

Actually, I tried to do it, and it didn't work.
I wouldn't get OpenMPI with IB if I tried to
link statically (i.e. by passing -static or equivalent to CFLAGS,
FFLAGS, etc).
When I removed the "-static" I got OpenMPI with IB.
I always dump the configure output (and the make output, etc) to
log files to check these things out after it is done.
I really suggest you do this, it pays off, saves time, costs nothing.
I don't remember exactly what symptoms I found on the log,
whether the log definitely said that there was no IB support,
or if it didn't have the right flags (-libverbs, etc) like yours.
However, when I suppressed the "-static" from the compiler flags
then I've got all the IB goodies! :)

Here is how I run configure (CFLAGS etc only have optimization flags,
no "-static"):

./configure \
--prefix=/my/directory \
--with-libnuma=/usr \
--with-tm=/usr \
--with-openib=/usr \
--enable-static \
2>&1 configure.log

Note, "--enable-static" means OpenMPI will build static libraries
(besides the shared ones).
OpenMPI is not being linked statically to system libraries,
or to IB libraries, etc.

Some switches may not be needed,
in particularly the explicit use of /usr directory.
However, at some point the OpenMPI configure
would not work without being
told this (at least for libnuma).

BTW, I didn't claim your OpenMPI doesn't have IB support.
Not a categorical syllogism like
"you don't have the -libverbs flag, hence you don't have IB".
It is hard to make definitive statements like this
in a complex environment like this (OpenMPI build, parallel programs),
and with limited information via email.
After all, the list is peer reviewed! :)
Hence, I only guessed, as I usually do in these exchanges.
However, considering all the trouble you've been through, who knows,
maybe it was a guess in the right direction.

I wonder if there may still be a glitch in the OpenMPI configure
script, on how it searches for and uses libraries like IB, NUMA, etc,
which may be causing the problem.
Jeff: Is this possible?

In any case, we have different "Wrapper extra LIBS".
I have -lrdmacm -libverbs, you and Noam don't have them.
(Noam: I am not saying you don't have IB support! :))
My configure explicitly asks for ib support, Noam's (and maybe yours)
doesn't.
Somehow, slight differences in how one invokes
the configure script seems to produce different results.

I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

The second of these statements doesn't follow from the first.
My "ompi_info -config" returns
ompi_info -config | grep LIBS
Build LIBS: -lnsl -lutil -lm
Wrapper extra LIBS: -ldl -Wl,--export-dynamic
-lnsl -lutil -
lm -ldl
But it does have openib
ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0,
Component v1.3.2)
and osu_bibw returns
# OSU MPI Bi-Directional Bandwidth Test v3.0
# Size Bi-Bandwidth (MB/s)
4194304 1717.43
which it's sure not getting over ethernet. I think Jeff
Squyres' test (ompi_info | grep openib) must be more definitive.
Noam
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress ORG

2009-06-24 21:57:49 UTC

Well, the whole situation is really bizarre.

I just uninstalled openmpi 1.3.2 on my system. Then I installed OFED
1.4.1 to see if that resolves this situation.

Here's what I get:

[***@master ~]# ompi_info --config
Configured by: root
Configured on: Wed Jun 24 11:10:00 EDT 2009
Configure host: master.org
Built by: root
Built on: Wed Jun 24 11:13:22 EDT 2009
Built host: master.org
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C char size: 1
C bool size: 1
C short size: 2
C int size: 4
C long size: 8
C float size: 4
C double size: 8
C pointer size: 8
C char align: 1
C bool align: 1
C int align: 4
C float align: 4
C double align: 8
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: gfortran
Fortran77 compiler abs: /usr/bin/gfortran
Fortran90 compiler: gfortran
Fortran90 compiler abs: /usr/bin/gfortran
Fort integer size: 4
Fort logical size: 4
Fort logical value true: 1
Fort have integer1: yes
Fort have integer2: yes
Fort have integer4: yes
Fort have integer8: yes
Fort have integer16: no
Fort have real4: yes
Fort have real8: yes
Fort have real16: no
Fort have complex8: yes
Fort have complex16: yes
Fort have complex32: no
Fort integer1 size: 1
Fort integer2 size: 2
Fort integer4 size: 4
Fort integer8 size: 8
Fort integer16 size: -1
Fort real size: 4
Fort real4 size: 4
Fort real8 size: 8
Fort real16 size: -1
Fort dbl prec size: 4
Fort cplx size: 4
Fort dbl cplx size: 4
Fort cplx8 size: 8
Fort cplx16 size: 16
Fort cplx32 size: -1
Fort integer align: 4
Fort integer1 align: 1
Fort integer2 align: 2
Fort integer4 align: 4
Fort integer8 align: 8
Fort integer16 align: -1
Fort real align: 4
Fort real4 align: 4
Fort real8 align: 8
Fort real16 align: -1
Fort dbl prec align: 4
Fort cplx align: 4
Fort dbl cplx align: 4
Fort cplx8 align: 4
Fort cplx16 align: 8
Fort cplx32 align: -1
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Sparse Groups: no
Build CFLAGS: -DNDEBUG -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
-finline-functions -fno-strict-aliasing
-pthread
-fvisibility=hidden
Build CXXFLAGS: -DNDEBUG -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
-finline-functions -pthread
Build FFLAGS: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
Build FCFLAGS: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
Build LDFLAGS: -export-dynamic
Build LIBS: -lnsl -lutil -lm
Wrapper extra CFLAGS: -pthread
Wrapper extra CXXFLAGS: -pthread
Wrapper extra FFLAGS: -pthread
Wrapper extra FCFLAGS: -pthread
Wrapper extra LDFLAGS:
Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -lm
-ldl
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: no
mpirun default --prefix: yes
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol visibility support: yes
FT Checkpoint support: no (checkpoint thread: no)
[***@master ~]#

so you see, even the OFED 1.4.1 installation fails to put -libverbs etc
into openmpi.

Also, I think its

--enable-static

that is putting the -libverbs into your openmpi. I'll try it and see
what happens.

What are libnuma and tm? Do I need to worry about them?

Finally, I have forgotton what I do with all the RPMs OFED generates.
Do I install them all on my compute nodes or just a subset?

Thanks for the help.

Jim

Post by Gus Correa
Hi Jim

Post by Jim Kress
Noam, Gus and List,
Did you statically link your openmpi when you built it? If you did (the
default is NOT to do this) then that could explain the discrepancy.
Jim

No, I didn't link statically.
Did you link statically?
Actually, I tried to do it, and it didn't work.
I wouldn't get OpenMPI with IB if I tried to
link statically (i.e. by passing -static or equivalent to CFLAGS,
FFLAGS, etc).
When I removed the "-static" I got OpenMPI with IB.
I always dump the configure output (and the make output, etc) to
log files to check these things out after it is done.
I really suggest you do this, it pays off, saves time, costs nothing.
I don't remember exactly what symptoms I found on the log,
whether the log definitely said that there was no IB support,
or if it didn't have the right flags (-libverbs, etc) like yours.
However, when I suppressed the "-static" from the compiler flags
then I've got all the IB goodies! :)
Here is how I run configure (CFLAGS etc only have optimization flags,
./configure \
--prefix=/my/directory \
--with-libnuma=/usr \
--with-tm=/usr \
--with-openib=/usr \
--enable-static \
2>&1 configure.log
Note, "--enable-static" means OpenMPI will build static libraries
(besides the shared ones).
OpenMPI is not being linked statically to system libraries,
or to IB libraries, etc.
Some switches may not be needed,
in particularly the explicit use of /usr directory.
However, at some point the OpenMPI configure
would not work without being
told this (at least for libnuma).
BTW, I didn't claim your OpenMPI doesn't have IB support.
Not a categorical syllogism like
"you don't have the -libverbs flag, hence you don't have IB".
It is hard to make definitive statements like this
in a complex environment like this (OpenMPI build, parallel programs),
and with limited information via email.
After all, the list is peer reviewed! :)
Hence, I only guessed, as I usually do in these exchanges.
However, considering all the trouble you've been through, who knows,
maybe it was a guess in the right direction.
I wonder if there may still be a glitch in the OpenMPI configure
script, on how it searches for and uses libraries like IB, NUMA, etc,
which may be causing the problem.
Jeff: Is this possible?
In any case, we have different "Wrapper extra LIBS".
I have -lrdmacm -libverbs, you and Noam don't have them.
(Noam: I am not saying you don't have IB support! :))
My configure explicitly asks for ib support, Noam's (and maybe yours)
doesn't.
Somehow, slight differences in how one invokes
the configure script seems to produce different results.
I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

The second of these statements doesn't follow from the first.
My "ompi_info -config" returns
ompi_info -config | grep LIBS
Build LIBS: -lnsl -lutil -lm
Wrapper extra LIBS: -ldl -Wl,--export-dynamic
-lnsl -lutil -
lm -ldl
But it does have openib
ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0,
Component v1.3.2)
and osu_bibw returns
# OSU MPI Bi-Directional Bandwidth Test v3.0
# Size Bi-Bandwidth (MB/s)
4194304 1717.43
which it's sure not getting over ethernet. I think Jeff
Squyres' test (ompi_info | grep openib) must be more definitive.
Noam
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Gus Correa

2009-06-24 23:19:14 UTC

Hi Jim, list

1) libnuma (non-uniform memory access, IIRR) is important if
you have AMD Opteron (our case here),
for processor and memory affinity, etc.
I suppose you don't need it with Intel Xeon (pre-Nehalem, at least),
but I am not positive about this (we don't have a Xeon cluster here).

2) tm is the Torque resource manager.
If you don't use Torque/PBS you don't have it, you don't need it.
Noam for instance uses sge, and configured with --use-sge.
If you just launch mpiexec directly you don't need
to build OpenMPI with any resource manager library or support.
(But resource managers are great!)

3) It may be that --enable-static puts the wrapper flags you don't have.
I don't know. The OpenMPI developers may clarify this.

4) AFAIK, you need OFED on all nodes, at least on those
that have IB hardware, that are connected to your IB switch,
where you want to run MPI programs using IB.

I hope this helps.

Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Post by Jim Kress ORG
Well, the whole situation is really bizarre.
I just uninstalled openmpi 1.3.2 on my system. Then I installed OFED
1.4.1 to see if that resolves this situation.
Configured by: root
Configured on: Wed Jun 24 11:10:00 EDT 2009
Configure host: master.org
Built by: root
Built on: Wed Jun 24 11:13:22 EDT 2009
Built host: master.org
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C char size: 1
C bool size: 1
C short size: 2
C int size: 4
C long size: 8
C float size: 4
C double size: 8
C pointer size: 8
C char align: 1
C bool align: 1
C int align: 4
C float align: 4
C double align: 8
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: gfortran
Fortran77 compiler abs: /usr/bin/gfortran
Fortran90 compiler: gfortran
Fortran90 compiler abs: /usr/bin/gfortran
Fort integer size: 4
Fort logical size: 4
Fort logical value true: 1
Fort have integer1: yes
Fort have integer2: yes
Fort have integer4: yes
Fort have integer8: yes
Fort have integer16: no
Fort have real4: yes
Fort have real8: yes
Fort have real16: no
Fort have complex8: yes
Fort have complex16: yes
Fort have complex32: no
Fort integer1 size: 1
Fort integer2 size: 2
Fort integer4 size: 4
Fort integer8 size: 8
Fort integer16 size: -1
Fort real size: 4
Fort real4 size: 4
Fort real8 size: 8
Fort real16 size: -1
Fort dbl prec size: 4
Fort cplx size: 4
Fort dbl cplx size: 4
Fort cplx8 size: 8
Fort cplx16 size: 16
Fort cplx32 size: -1
Fort integer align: 4
Fort integer1 align: 1
Fort integer2 align: 2
Fort integer4 align: 4
Fort integer8 align: 8
Fort integer16 align: -1
Fort real align: 4
Fort real4 align: 4
Fort real8 align: 8
Fort real16 align: -1
Fort dbl prec align: 4
Fort cplx align: 4
Fort dbl cplx align: 4
Fort cplx8 align: 4
Fort cplx16 align: 8
Fort cplx32 align: -1
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Sparse Groups: no
Build CFLAGS: -DNDEBUG -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
-finline-functions -fno-strict-aliasing
-pthread
-fvisibility=hidden
Build CXXFLAGS: -DNDEBUG -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
-finline-functions -pthread
Build FFLAGS: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
Build FCFLAGS: -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
Build LDFLAGS: -export-dynamic
Build LIBS: -lnsl -lutil -lm
Wrapper extra CFLAGS: -pthread
Wrapper extra CXXFLAGS: -pthread
Wrapper extra FFLAGS: -pthread
Wrapper extra FCFLAGS: -pthread
Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -lm
-ldl
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: no
mpirun default --prefix: yes
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol visibility support: yes
FT Checkpoint support: no (checkpoint thread: no)
so you see, even the OFED 1.4.1 installation fails to put -libverbs etc
into openmpi.
Also, I think its
--enable-static
that is putting the -libverbs into your openmpi. I'll try it and see
what happens.
What are libnuma and tm? Do I need to worry about them?
Finally, I have forgotton what I do with all the RPMs OFED generates.
Do I install them all on my compute nodes or just a subset?
Thanks for the help.
Jim

Post by Gus Correa
Hi Jim

Post by Jim Kress
Noam, Gus and List,
Did you statically link your openmpi when you built it? If you did (the
default is NOT to do this) then that could explain the discrepancy.
Jim

No, I didn't link statically.
Did you link statically?
Actually, I tried to do it, and it didn't work.
I wouldn't get OpenMPI with IB if I tried to
link statically (i.e. by passing -static or equivalent to CFLAGS,
FFLAGS, etc).
When I removed the "-static" I got OpenMPI with IB.
I always dump the configure output (and the make output, etc) to
log files to check these things out after it is done.
I really suggest you do this, it pays off, saves time, costs nothing.
I don't remember exactly what symptoms I found on the log,
whether the log definitely said that there was no IB support,
or if it didn't have the right flags (-libverbs, etc) like yours.
However, when I suppressed the "-static" from the compiler flags
then I've got all the IB goodies! :)
Here is how I run configure (CFLAGS etc only have optimization flags,
./configure \
--prefix=/my/directory \
--with-libnuma=/usr \
--with-tm=/usr \
--with-openib=/usr \
--enable-static \
2>&1 configure.log
Note, "--enable-static" means OpenMPI will build static libraries
(besides the shared ones).
OpenMPI is not being linked statically to system libraries,
or to IB libraries, etc.
Some switches may not be needed,
in particularly the explicit use of /usr directory.
However, at some point the OpenMPI configure
would not work without being
told this (at least for libnuma).
BTW, I didn't claim your OpenMPI doesn't have IB support.
Not a categorical syllogism like
"you don't have the -libverbs flag, hence you don't have IB".
It is hard to make definitive statements like this
in a complex environment like this (OpenMPI build, parallel programs),
and with limited information via email.
After all, the list is peer reviewed! :)
Hence, I only guessed, as I usually do in these exchanges.
However, considering all the trouble you've been through, who knows,
maybe it was a guess in the right direction.
I wonder if there may still be a glitch in the OpenMPI configure
script, on how it searches for and uses libraries like IB, NUMA, etc,
which may be causing the problem.
Jeff: Is this possible?
In any case, we have different "Wrapper extra LIBS".
I have -lrdmacm -libverbs, you and Noam don't have them.
(Noam: I am not saying you don't have IB support! :))
My configure explicitly asks for ib support, Noam's (and maybe yours)
doesn't.
Somehow, slight differences in how one invokes
the configure script seems to produce different results.
I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

The second of these statements doesn't follow from the first.
My "ompi_info -config" returns
ompi_info -config | grep LIBS
Build LIBS: -lnsl -lutil -lm
Wrapper extra LIBS: -ldl -Wl,--export-dynamic
-lnsl -lutil -
lm -ldl
But it does have openib
ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0,
Component v1.3.2)
and osu_bibw returns
# OSU MPI Bi-Directional Bandwidth Test v3.0
# Size Bi-Bandwidth (MB/s)
4194304 1717.43
which it's sure not getting over ethernet. I think Jeff
Squyres' test (ompi_info | grep openib) must be more definitive.
Noam
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress ORG

2009-06-24 22:41:57 UTC

Hey Gus. I was correct.

If I did:

./configure --prefix=/my/dir --with-openib=/usr --enable-static
make all install

then reboot and use mpi-selector to choose openmpi-1.3.2, and then:

[***@master ~]# ompi_info --config
Configured by: root
Configured on: Wed Jun 24 18:02:03 EDT 2009
Configure host: master.org
Built by: root
Built on: Wed Jun 24 18:17:29 EDT 2009
Built host: master.org
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C char size: 1
C bool size: 1
C short size: 2
C int size: 4
C long size: 8
C float size: 4
C double size: 8
C pointer size: 8
C char align: 1
C bool align: 1
C int align: 4
C float align: 4
C double align: 8
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: gfortran
Fortran77 compiler abs: /usr/bin/gfortran
Fortran90 compiler: gfortran
Fortran90 compiler abs: /usr/bin/gfortran
Fort integer size: 4
Fort logical size: 4
Fort logical value true: 1
Fort have integer1: yes
Fort have integer2: yes
Fort have integer4: yes
Fort have integer8: yes
Fort have integer16: no
Fort have real4: yes
Fort have real8: yes
Fort have real16: no
Fort have complex8: yes
Fort have complex16: yes
Fort have complex32: no
Fort integer1 size: 1
Fort integer2 size: 2
Fort integer4 size: 4
Fort integer8 size: 8
Fort integer16 size: -1
Fort real size: 4
Fort real4 size: 4
Fort real8 size: 8
Fort real16 size: -1
Fort dbl prec size: 4
Fort cplx size: 4
Fort dbl cplx size: 4
Fort cplx8 size: 8
Fort cplx16 size: 16
Fort cplx32 size: -1
Fort integer align: 4
Fort integer1 align: 1
Fort integer2 align: 2
Fort integer4 align: 4
Fort integer8 align: 8
Fort integer16 align: -1
Fort real align: 4
Fort real4 align: 4
Fort real8 align: 8
Fort real16 align: -1
Fort dbl prec align: 4
Fort cplx align: 4
Fort dbl cplx align: 4
Fort cplx8 align: 4
Fort cplx16 align: 8
Fort cplx32 align: -1
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Sparse Groups: no
Build CFLAGS: -O3 -DNDEBUG -finline-functions
-fno-strict-aliasing
-pthread -fvisibility=hidden
Build CXXFLAGS: -O3 -DNDEBUG -finline-functions -pthread
Build FFLAGS:
Build FCFLAGS:
Build LDFLAGS: -export-dynamic
Build LIBS: -lnsl -lutil -lm
Wrapper extra CFLAGS: -pthread
Wrapper extra CXXFLAGS: -pthread
Wrapper extra FFLAGS: -pthread
Wrapper extra FCFLAGS: -pthread
Wrapper extra LDFLAGS:
Wrapper extra LIBS: -lrdmacm -libverbs -ldl
-Wl,--export-dynamic -lnsl
-lutil -lm -ldl
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: no
mpirun default --prefix: no
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol visibility support: yes
FT Checkpoint support: no (checkpoint thread: no)
[***@master ~]#

Magically, -lrdmacm -libverbs appear.

Well, that's one mystery solved.

Thanks for your help.

Jim

Post by Gus Correa
Hi Jim

Post by Jim Kress
Noam, Gus and List,
Did you statically link your openmpi when you built it? If you did (the
default is NOT to do this) then that could explain the discrepancy.
Jim

No, I didn't link statically.
Did you link statically?
Actually, I tried to do it, and it didn't work.
I wouldn't get OpenMPI with IB if I tried to
link statically (i.e. by passing -static or equivalent to CFLAGS,
FFLAGS, etc).
When I removed the "-static" I got OpenMPI with IB.
I always dump the configure output (and the make output, etc) to
log files to check these things out after it is done.
I really suggest you do this, it pays off, saves time, costs nothing.
I don't remember exactly what symptoms I found on the log,
whether the log definitely said that there was no IB support,
or if it didn't have the right flags (-libverbs, etc) like yours.
However, when I suppressed the "-static" from the compiler flags
then I've got all the IB goodies! :)
Here is how I run configure (CFLAGS etc only have optimization flags,
./configure \
--prefix=/my/directory \
--with-libnuma=/usr \
--with-tm=/usr \
--with-openib=/usr \
--enable-static \
2>&1 configure.log
Note, "--enable-static" means OpenMPI will build static libraries
(besides the shared ones).
OpenMPI is not being linked statically to system libraries,
or to IB libraries, etc.
Some switches may not be needed,
in particularly the explicit use of /usr directory.
However, at some point the OpenMPI configure
would not work without being
told this (at least for libnuma).
BTW, I didn't claim your OpenMPI doesn't have IB support.
Not a categorical syllogism like
"you don't have the -libverbs flag, hence you don't have IB".
It is hard to make definitive statements like this
in a complex environment like this (OpenMPI build, parallel programs),
and with limited information via email.
After all, the list is peer reviewed! :)
Hence, I only guessed, as I usually do in these exchanges.
However, considering all the trouble you've been through, who knows,
maybe it was a guess in the right direction.
I wonder if there may still be a glitch in the OpenMPI configure
script, on how it searches for and uses libraries like IB, NUMA, etc,
which may be causing the problem.
Jeff: Is this possible?
In any case, we have different "Wrapper extra LIBS".
I have -lrdmacm -libverbs, you and Noam don't have them.
(Noam: I am not saying you don't have IB support! :))
My configure explicitly asks for ib support, Noam's (and maybe yours)
doesn't.
Somehow, slight differences in how one invokes
the configure script seems to produce different results.
I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

The second of these statements doesn't follow from the first.
My "ompi_info -config" returns
ompi_info -config | grep LIBS
Build LIBS: -lnsl -lutil -lm
Wrapper extra LIBS: -ldl -Wl,--export-dynamic
-lnsl -lutil -
lm -ldl
But it does have openib
ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0,
Component v1.3.2)
and osu_bibw returns
# OSU MPI Bi-Directional Bandwidth Test v3.0
# Size Bi-Bandwidth (MB/s)
4194304 1717.43
which it's sure not getting over ethernet. I think Jeff
Squyres' test (ompi_info | grep openib) must be more definitive.
Noam
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Gus Correa

2009-06-24 23:30:57 UTC

Hi Jim

Post by Jim Kress ORG
Hey Gus. I was correct.
./configure --prefix=/my/dir --with-openib=/usr --enable-static
make all install

...

Post by Jim Kress ORG
Wrapper extra LIBS: -lrdmacm -libverbs -ldl
-Wl,--export-dynamic -lnsl
-lutil -lm -ldl

...

Post by Jim Kress ORG
Magically, -lrdmacm -libverbs appear.

Thank you for telling us!
I was too busylazy to try it once again myself.
I built OpenMPI a lot of times, different compilers,
versions, clusters ...

In any case, the ORCA mystery remains, which is rather unsettling.
Have you investigated Jeff's question on whether the code was
compiled/linked with the same OpenMPI version (1.3.2)?
I wonder if the underlying OFED libraries must be the same as well.

Gus
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Post by Jim Kress ORG
Hey Gus. I was correct.
./configure --prefix=/my/dir --with-openib=/usr --enable-static
make all install
Configured by: root
Configured on: Wed Jun 24 18:02:03 EDT 2009
Configure host: master.org
Built by: root
Built on: Wed Jun 24 18:17:29 EDT 2009
Built host: master.org
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C char size: 1
C bool size: 1
C short size: 2
C int size: 4
C long size: 8
C float size: 4
C double size: 8
C pointer size: 8
C char align: 1
C bool align: 1
C int align: 4
C float align: 4
C double align: 8
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: gfortran
Fortran77 compiler abs: /usr/bin/gfortran
Fortran90 compiler: gfortran
Fortran90 compiler abs: /usr/bin/gfortran
Fort integer size: 4
Fort logical size: 4
Fort logical value true: 1
Fort have integer1: yes
Fort have integer2: yes
Fort have integer4: yes
Fort have integer8: yes
Fort have integer16: no
Fort have real4: yes
Fort have real8: yes
Fort have real16: no
Fort have complex8: yes
Fort have complex16: yes
Fort have complex32: no
Fort integer1 size: 1
Fort integer2 size: 2
Fort integer4 size: 4
Fort integer8 size: 8
Fort integer16 size: -1
Fort real size: 4
Fort real4 size: 4
Fort real8 size: 8
Fort real16 size: -1
Fort dbl prec size: 4
Fort cplx size: 4
Fort dbl cplx size: 4
Fort cplx8 size: 8
Fort cplx16 size: 16
Fort cplx32 size: -1
Fort integer align: 4
Fort integer1 align: 1
Fort integer2 align: 2
Fort integer4 align: 4
Fort integer8 align: 8
Fort integer16 align: -1
Fort real align: 4
Fort real4 align: 4
Fort real8 align: 8
Fort real16 align: -1
Fort dbl prec align: 4
Fort cplx align: 4
Fort dbl cplx align: 4
Fort cplx8 align: 4
Fort cplx16 align: 8
Fort cplx32 align: -1
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Sparse Groups: no
Build CFLAGS: -O3 -DNDEBUG -finline-functions
-fno-strict-aliasing
-pthread -fvisibility=hidden
Build CXXFLAGS: -O3 -DNDEBUG -finline-functions -pthread
Build LDFLAGS: -export-dynamic
Build LIBS: -lnsl -lutil -lm
Wrapper extra CFLAGS: -pthread
Wrapper extra CXXFLAGS: -pthread
Wrapper extra FFLAGS: -pthread
Wrapper extra FCFLAGS: -pthread
Wrapper extra LIBS: -lrdmacm -libverbs -ldl
-Wl,--export-dynamic -lnsl
-lutil -lm -ldl
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: no
mpirun default --prefix: no
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol visibility support: yes
FT Checkpoint support: no (checkpoint thread: no)
Magically, -lrdmacm -libverbs appear.
Well, that's one mystery solved.
Thanks for your help.
Jim

Post by Gus Correa
Hi Jim

Post by Jim Kress
Noam, Gus and List,
Did you statically link your openmpi when you built it? If you did (the
default is NOT to do this) then that could explain the discrepancy.
Jim

No, I didn't link statically.
Did you link statically?
Actually, I tried to do it, and it didn't work.
I wouldn't get OpenMPI with IB if I tried to
link statically (i.e. by passing -static or equivalent to CFLAGS,
FFLAGS, etc).
When I removed the "-static" I got OpenMPI with IB.
I always dump the configure output (and the make output, etc) to
log files to check these things out after it is done.
I really suggest you do this, it pays off, saves time, costs nothing.
I don't remember exactly what symptoms I found on the log,
whether the log definitely said that there was no IB support,
or if it didn't have the right flags (-libverbs, etc) like yours.
However, when I suppressed the "-static" from the compiler flags
then I've got all the IB goodies! :)
Here is how I run configure (CFLAGS etc only have optimization flags,
./configure \
--prefix=/my/directory \
--with-libnuma=/usr \
--with-tm=/usr \
--with-openib=/usr \
--enable-static \
2>&1 configure.log
Note, "--enable-static" means OpenMPI will build static libraries
(besides the shared ones).
OpenMPI is not being linked statically to system libraries,
or to IB libraries, etc.
Some switches may not be needed,
in particularly the explicit use of /usr directory.
However, at some point the OpenMPI configure
would not work without being
told this (at least for libnuma).
BTW, I didn't claim your OpenMPI doesn't have IB support.
Not a categorical syllogism like
"you don't have the -libverbs flag, hence you don't have IB".
It is hard to make definitive statements like this
in a complex environment like this (OpenMPI build, parallel programs),
and with limited information via email.
After all, the list is peer reviewed! :)
Hence, I only guessed, as I usually do in these exchanges.
However, considering all the trouble you've been through, who knows,
maybe it was a guess in the right direction.
I wonder if there may still be a glitch in the OpenMPI configure
script, on how it searches for and uses libraries like IB, NUMA, etc,
which may be causing the problem.
Jeff: Is this possible?
In any case, we have different "Wrapper extra LIBS".
I have -lrdmacm -libverbs, you and Noam don't have them.
(Noam: I am not saying you don't have IB support! :))
My configure explicitly asks for ib support, Noam's (and maybe yours)
doesn't.
Somehow, slight differences in how one invokes
the configure script seems to produce different results.
I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

The second of these statements doesn't follow from the first.
My "ompi_info -config" returns
ompi_info -config | grep LIBS
Build LIBS: -lnsl -lutil -lm
Wrapper extra LIBS: -ldl -Wl,--export-dynamic
-lnsl -lutil -
lm -ldl
But it does have openib
ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0,
Component v1.3.2)
and osu_bibw returns
# OSU MPI Bi-Directional Bandwidth Test v3.0
# Size Bi-Bandwidth (MB/s)
4194304 1717.43
which it's sure not getting over ethernet. I think Jeff
Squyres' test (ompi_info | grep openib) must be more definitive.
Noam
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress ORG

2009-06-25 00:18:47 UTC

Post by Gus Correa
Have you investigated Jeff's question on whether the code was
compiled/linked with the same OpenMPI version (1.3.2)?
I wonder if the underlying OFED libraries must be the same as well.

I was told that 1.3.2 was used. However, I have not asked about which
OFED libraries were used nor have I asked about the use of
--enable-static for their 1.3.2 configurations.

I will have to follow-up on that.

Jim

Post by Gus Correa
Hi Jim

Post by Jim Kress ORG
Hey Gus. I was correct.
./configure --prefix=/my/dir --with-openib=/usr --enable-static
make all install

...

Post by Jim Kress ORG
Wrapper extra LIBS: -lrdmacm -libverbs -ldl
-Wl,--export-dynamic -lnsl
-lutil -lm -ldl

...

Post by Jim Kress ORG
Magically, -lrdmacm -libverbs appear.

Thank you for telling us!
I was too busylazy to try it once again myself.
I built OpenMPI a lot of times, different compilers,
versions, clusters ...
In any case, the ORCA mystery remains, which is rather unsettling.
Have you investigated Jeff's question on whether the code was
compiled/linked with the same OpenMPI version (1.3.2)?
I wonder if the underlying OFED libraries must be the same as well.
Gus
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Post by Gus Correa
Hi Jim

Post by Jim Kress
Noam, Gus and List,
Did you statically link your openmpi when you built it? If you did (the
default is NOT to do this) then that could explain the discrepancy.
Jim

No, I didn't link statically.
Did you link statically?
Actually, I tried to do it, and it didn't work.
I wouldn't get OpenMPI with IB if I tried to
link statically (i.e. by passing -static or equivalent to CFLAGS,
FFLAGS, etc).
When I removed the "-static" I got OpenMPI with IB.
I always dump the configure output (and the make output, etc) to
log files to check these things out after it is done.
I really suggest you do this, it pays off, saves time, costs nothing.
I don't remember exactly what symptoms I found on the log,
whether the log definitely said that there was no IB support,
or if it didn't have the right flags (-libverbs, etc) like yours.
However, when I suppressed the "-static" from the compiler flags
then I've got all the IB goodies! :)
Here is how I run configure (CFLAGS etc only have optimization flags,
./configure \
--prefix=/my/directory \
--with-libnuma=/usr \
--with-tm=/usr \
--with-openib=/usr \
--enable-static \
2>&1 configure.log
Note, "--enable-static" means OpenMPI will build static libraries
(besides the shared ones).
OpenMPI is not being linked statically to system libraries,
or to IB libraries, etc.
Some switches may not be needed,
in particularly the explicit use of /usr directory.
However, at some point the OpenMPI configure
would not work without being
told this (at least for libnuma).
BTW, I didn't claim your OpenMPI doesn't have IB support.
Not a categorical syllogism like
"you don't have the -libverbs flag, hence you don't have IB".
It is hard to make definitive statements like this
in a complex environment like this (OpenMPI build, parallel programs),
and with limited information via email.
After all, the list is peer reviewed! :)
Hence, I only guessed, as I usually do in these exchanges.
However, considering all the trouble you've been through, who knows,
maybe it was a guess in the right direction.
I wonder if there may still be a glitch in the OpenMPI configure
script, on how it searches for and uses libraries like IB, NUMA, etc,
which may be causing the problem.
Jeff: Is this possible?
In any case, we have different "Wrapper extra LIBS".
I have -lrdmacm -libverbs, you and Noam don't have them.
(Noam: I am not saying you don't have IB support! :))
My configure explicitly asks for ib support, Noam's (and maybe yours)
doesn't.
Somehow, slight differences in how one invokes
the configure script seems to produce different results.
I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

The second of these statements doesn't follow from the first.
My "ompi_info -config" returns
ompi_info -config | grep LIBS
Build LIBS: -lnsl -lutil -lm
Wrapper extra LIBS: -ldl -Wl,--export-dynamic
-lnsl -lutil -
lm -ldl
But it does have openib
ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0,
Component v1.3.2)
and osu_bibw returns
# OSU MPI Bi-Directional Bandwidth Test v3.0
# Size Bi-Bandwidth (MB/s)
4194304 1717.43
which it's sure not getting over ethernet. I think Jeff
Squyres' test (ompi_info | grep openib) must be more definitive.
Noam
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jeff Squyres

2009-06-26 00:37:21 UTC

This thread diverged quite a bit into Open MPI configuration and build
issues -- did the original question get answered?

Post by Gus Correa
Hi Jim

Post by Jim Kress ORG
Hey Gus. I was correct.
./configure --prefix=/my/dir --with-openib=/usr --enable-static
make all install

...

Post by Jim Kress ORG
Wrapper extra LIBS: -lrdmacm -libverbs -ldl
-Wl,--export-dynamic -lnsl
-lutil -lm -ldl

...

Post by Jim Kress ORG
Magically, -lrdmacm -libverbs appear.

---------------------------------------------------------------------

Post by Gus Correa
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA

---------------------------------------------------------------------

Post by Jim Kress ORG
Hey Gus. I was correct.
./configure --prefix=/my/dir --with-openib=/usr --enable-static
make all install
then reboot and use mpi-selector to choose openmpi-1.3.2, and
Configured by: root
Configured on: Wed Jun 24 18:02:03 EDT 2009
Configure host: master.org
Built by: root
Built on: Wed Jun 24 18:17:29 EDT 2009
Built host: master.org
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C char size: 1
C bool size: 1
C short size: 2
C int size: 4
C long size: 8
C float size: 4
C double size: 8
C pointer size: 8
C char align: 1
C bool align: 1
C int align: 4
C float align: 4
C double align: 8
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: gfortran
Fortran77 compiler abs: /usr/bin/gfortran
Fortran90 compiler: gfortran
Fortran90 compiler abs: /usr/bin/gfortran
Fort integer size: 4
Fort logical size: 4
Fort logical value true: 1
Fort have integer1: yes
Fort have integer2: yes
Fort have integer4: yes
Fort have integer8: yes
Fort have integer16: no
Fort have real4: yes
Fort have real8: yes
Fort have real16: no
Fort have complex8: yes
Fort have complex16: yes
Fort have complex32: no
Fort integer1 size: 1
Fort integer2 size: 2
Fort integer4 size: 4
Fort integer8 size: 8
Fort integer16 size: -1
Fort real size: 4
Fort real4 size: 4
Fort real8 size: 8
Fort real16 size: -1
Fort dbl prec size: 4
Fort cplx size: 4
Fort dbl cplx size: 4
Fort cplx8 size: 8
Fort cplx16 size: 16
Fort cplx32 size: -1
Fort integer align: 4
Fort integer1 align: 1
Fort integer2 align: 2
Fort integer4 align: 4
Fort integer8 align: 8
Fort integer16 align: -1
Fort real align: 4
Fort real4 align: 4
Fort real8 align: 8
Fort real16 align: -1
Fort dbl prec align: 4
Fort cplx align: 4
Fort dbl cplx align: 4
Fort cplx8 align: 4
Fort cplx16 align: 8
Fort cplx32 align: -1
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Sparse Groups: no
Build CFLAGS: -O3 -DNDEBUG -finline-functions
-fno-strict-aliasing
-pthread -fvisibility=hidden
Build CXXFLAGS: -O3 -DNDEBUG -finline-functions -pthread
Build LDFLAGS: -export-dynamic
Build LIBS: -lnsl -lutil -lm
Wrapper extra CFLAGS: -pthread
Wrapper extra CXXFLAGS: -pthread
Wrapper extra FFLAGS: -pthread
Wrapper extra FCFLAGS: -pthread
Wrapper extra LIBS: -lrdmacm -libverbs -ldl
-Wl,--export-dynamic -lnsl
-lutil -lm -ldl
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: no
mpirun default --prefix: no
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol visibility support: yes
FT Checkpoint support: no (checkpoint thread: no)
Magically, -lrdmacm -libverbs appear.
Well, that's one mystery solved.
Thanks for your help.
Jim

Post by Gus Correa
Hi Jim

Post by Jim Kress
Noam, Gus and List,
Did you statically link your openmpi when you built it? If

you did (the

Post by Jim Kress
default is NOT to do this) then that could explain the

discrepancy.

Post by Jim Kress
Jim

nothing.

Post by Gus Correa
I don't remember exactly what symptoms I found on the log,
whether the log definitely said that there was no IB support,
or if it didn't have the right flags (-libverbs, etc) like yours.
However, when I suppressed the "-static" from the compiler flags
then I've got all the IB goodies! :)
Here is how I run configure (CFLAGS etc only have optimization

flags,

Post by Gus Correa
./configure \
--prefix=/my/directory \
--with-libnuma=/usr \
--with-tm=/usr \
--with-openib=/usr \
--enable-static \
2>&1 configure.log
Note, "--enable-static" means OpenMPI will build static libraries
(besides the shared ones).
OpenMPI is not being linked statically to system libraries,
or to IB libraries, etc.
Some switches may not be needed,
in particularly the explicit use of /usr directory.
However, at some point the OpenMPI configure
would not work without being
told this (at least for libnuma).
BTW, I didn't claim your OpenMPI doesn't have IB support.
Not a categorical syllogism like
"you don't have the -libverbs flag, hence you don't have IB".
It is hard to make definitive statements like this
in a complex environment like this (OpenMPI build, parallel

programs),

Post by Gus Correa
and with limited information via email.
After all, the list is peer reviewed! :)
Hence, I only guessed, as I usually do in these exchanges.
However, considering all the trouble you've been through, who

knows,

Post by Gus Correa
maybe it was a guess in the right direction.
I wonder if there may still be a glitch in the OpenMPI configure
script, on how it searches for and uses libraries like IB,

NUMA, etc,

Post by Gus Correa
which may be causing the problem.
Jeff: Is this possible?
In any case, we have different "Wrapper extra LIBS".
I have -lrdmacm -libverbs, you and Noam don't have them.
(Noam: I am not saying you don't have IB support! :))
My configure explicitly asks for ib support, Noam's (and maybe

yours)

Post by Gus Correa
doesn't.
Somehow, slight differences in how one invokes
the configure script seems to produce different results.
I hope this helps,
Gus Correa

---------------------------------------------------------------------

Post by Gus Correa
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA

---------------------------------------------------------------------

Post by Gus Correa
Hi Jim, list
Wrapper extra LIBS: -lrdmacm -libverbs -ltorque -lnuma -ldl -

Wl,--

Post by Gus Correa
export-dynamic -lnsl -lutil -lm -ldl
Yours doesn't seem to have the IB libraries: -lrdmacm -

libverbs

Post by Gus Correa
So, I would guess your OpenMPI 1.3.2 build doesn't have IB

support.

The second of these statements doesn't follow from the first.
My "ompi_info -config" returns
ompi_info -config | grep LIBS
Build LIBS: -lnsl -lutil -lm
Wrapper extra LIBS: -ldl -Wl,--export-dynamic
-lnsl -lutil -
lm -ldl
But it does have openib
ompi_info | grep openib
MCA btl: openib (MCA v2.0, API v2.0,
Component v1.3.2)
and osu_bibw returns
# OSU MPI Bi-Directional Bandwidth Test v3.0
# Size Bi-Bandwidth (MB/s)
4194304 1717.43
which it's sure not getting over ethernet. I think Jeff
Squyres' test (ompi_info | grep openib) must be more

definitive.

Noam
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

--
Jeff Squyres
Cisco Systems

Jim Kress

2009-06-25 16:53:54 UTC

Is it correct to assume that, when one is configuring openmpi v1.3.2 and if
one leaves out the

--with-openib=/dir

from the ./configure command line, that InfiniBand support will NOT be built
into openmpi v1.3.2? Then, if an Ethernet network is present that connects
all the nodes, openmpi will use that network?

Also, is it required to add

--enable-static

to the ./configure command line to make sure Infiniband support is
available?
If I do not then the ompi_info --config command yields

Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl

Note the absence of -lrdmacm and -libverbs which, I am told, are essential
for Infiniband support.

Whereas if --enable-static IS included, the ompi_info --config command
yields

Wrapper extra LIBS: -lrdmacm -libverbs -ldl -Wl,--export-dynamic -lnsl
-lutil -lm -ldl

And -lrdmacm and -libverbs are now present.

Thanks for your help.

Jim

Gus Correa

2009-06-25 22:32:32 UTC

Hi Jim, list

1) Your first question:

I opened a thread on this list two months or so ago about a similar
situation: when OpenMPI would use/not use libnuma.
I asked a question very similar to your question about IB support,
and how the configure script would provide it or not.
Jeff answerer it, and I asked him to post the answer in the FAQ,
which he kindly did (or an edited version of it):

http://www.open-mpi.org/faq/?category=building#default-build

The wisdom is that OpenMPI will search for IB on standard places,
and will use it if it finds it.
If you don't have IB on a standard place, then you can use the
switch --with-openib=/dir to force IB to be part of your OpenMPI.
If I understood it right, the bottom line is that you
only don't get IB if it is hidden, or doesn't exist.

2) Your second question:

As for the --enable-static question, I don't know the answer.
I defer this one to the OpenMPI developers.
I confess I have no explanation for why "-lrdmacm -libverbs"
are listed amongst the "Wrapper extra LIBS" when OpenMPI is configured
with --enable-static, but those IB library flags are absent
when --enable-static is not used.
(Maybe because the dynamic linker does a smarter search.)

I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Post by Jim Kress
Is it correct to assume that, when one is configuring openmpi v1.3.2 and if
one leaves out the
--with-openib=/dir
from the ./configure command line, that InfiniBand support will NOT be built
into openmpi v1.3.2? Then, if an Ethernet network is present that connects
all the nodes, openmpi will use that network?
Also, is it required to add
--enable-static
to the ./configure command line to make sure Infiniband support is
available?
If I do not then the ompi_info --config command yields
Wrapper extra LIBS: -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl
Note the absence of -lrdmacm and -libverbs which, I am told, are essential
for Infiniband support.
Whereas if --enable-static IS included, the ompi_info --config command
yields
Wrapper extra LIBS: -lrdmacm -libverbs -ldl -Wl,--export-dynamic -lnsl
-lutil -lm -ldl
And -lrdmacm and -libverbs are now present.
Thanks for your help.
Jim
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Prentice Bisbal

2009-06-30 15:42:38 UTC

Post by Gus Correa
Hi Jim, list
I opened a thread on this list two months or so ago about a similar
situation: when OpenMPI would use/not use libnuma.
I asked a question very similar to your question about IB support,
and how the configure script would provide it or not.
Jeff answerer it, and I asked him to post the answer in the FAQ,
http://www.open-mpi.org/faq/?category=building#default-build
The wisdom is that OpenMPI will search for IB on standard places,
and will use it if it finds it.
If you don't have IB on a standard place, then you can use the
switch --with-openib=/dir to force IB to be part of your OpenMPI.
If I understood it right, the bottom line is that you
only don't get IB if it is hidden, or doesn't exist.

I've found that on 64-bit RHEL systems, many configure scripts don't
consider /lib64 and /usr/lib64 "standard" locations to look for
libraries, so I often have to do something like

./configure --with-pkg=/usr/lib64

or ./configure --with-pkg-lib=/usr/lib64

depending on the package's configure script. I just checked my notes
form compiling OMPI 1.2.8 and 1.3, and all I needed was "--with-openib"
for my distro (a rebuild of RHEL 5.3), but you never know - you may need
just such a trick.

--
Prentice

Jeff Squyres

2009-06-26 00:36:19 UTC

No, that is not a safe assumption. Sorry for being a bit tardy in
this discussion -- it's been a crazy week. :-)

The best thing to do is to actually look at the output of configure.
We actually put a lot of effort to actually make the output be fairly
descriptive and informative about what the build process will do when
configure is complete. I find it easiest to dump the output of
configure to a file and looking at it later in a text editor. For
example, searching for btl:openib in my configure output, I see:

--- MCA component btl:openib (m4 configuration macro)
...bunch of stuff...
checking if MCA component btl:openib can compile... yes

The last line of the section indicates whether that plugin
("component") is going to build or not. In my case, the openib plugin
was happy and was going to build.

If it's not happy / won't build, then the reason why will be in one of
the lines above it (e.g., some header files not found, etc.).

We have some "rules of thumb" about --with-<foo> options for configure
(including --with-openib); thanks to Gus' questions a month or three
ago, I amended our FAQ to list all of them:

http://www.open-mpi.org/faq/?category=building#default-build

Post by Jim Kress
Also, is it required to add
--enable-static
to the ./configure command line to make sure Infiniband support is
available?

No. I rarely (never?) build statically. Indeed, configuring with
OpenFabrics support statically is a rather dicey proposition. See:

http://www.open-mpi.org/faq/?category=mpi-apps#static-mpi-apps
http://www.open-mpi.org/faq/?category=mpi-apps#static-ofa-mpi-apps

Post by Jim Kress
If I do not then the ompi_info --config command yields

ompi_info --config is not the important one that you want to look at
here.

You want to do what I suggested before: "ompi_info | grep btl" and
look for the "openib" component. If you run "ompi_info" with no
options, it shows you all the plugins that are installed. You want to
see that the openib plugin is successfully installed.

It's [somewhat] irrelevant as to which libraries are shown in the
ompi_info --config output; that's under-the-covers mumbo-jumbo that
you shouldn't usually need to worry about.

Hope that helps!

--
Jeff Squyres
Cisco Systems

Jim Kress ORG

2009-06-23 21:27:10 UTC

OK. For version 1.2.8, I do this:

export OMPI_MCA_mpi_show_mca_params=1

Post by Jim Kress ORG
1.2.8_test_crafted_input_file.out

[master.org:07943]
mca_param_files=/root/.openmpi/mca-params.conf:/usr/mpi/gcc/openmpi-1.2.8/etc/openmpi-mca-params.conf
[master.org:07943] opal_signal=6,7,8,11
[master.org:07943]
mca_component_path=/usr/mpi/gcc/openmpi-1.2.8/lib/openmpi:/root/.openmpi/components
[master.org:07943] mca_verbose=
[master.org:07943] mca_component_show_load_errors=1
[master.org:07943] mca_component_disable_dlopen=0
[master.org:07943] paffinity_base_verbose=0
[master.org:07943] paffinity=
[master.org:07943] paffinity_linux_priority=10
[master.org:07943] paffinity_linux_have_cpu_set_t=1
[master.org:07943] paffinity_linux_CPU_ZERO_ok=1
[master.org:07943] paffinity_linux_sched_setaffinity_num_params=3
[master.org:07943] memcpy=
[master.org:07943] memcpy_base_verbose=0
[master.org:07943] memory=
[master.org:07943] memory_base_verbose=0
[master.org:07943] memory_ptmalloc2_priority=0
[master.org:07943] backtrace=
[master.org:07943] backtrace_base_verbose=0
[master.org:07943] backtrace_execinfo_priority=0
[master.org:07943] timer=
[master.org:07943] timer_base_verbose=0
[master.org:07943] timer_linux_priority=0
[master.org:07943] ompi_timing=0
[master.org:07943] orte_debug=0
[master.org:07943] orte_debug_daemons_file=0
[master.org:07943] orte_no_daemonize=0
[master.org:07943] orte_debug_daemons=0
[master.org:07943] orte_base_user_debugger=totalview @mpirun@ -a
@mpirun_args@ : ddt -n @np@ -start @executable@ @executable_argv@
@single_app@ : fxp @mpirun@ -a @mpirun_args@
[master.org:07943] orte_abort_timeout=10
[master.org:07943] orte_base_nodename=master
[master.org:07943] seed=0
[master.org:07943] orte_app_num=0
[master.org:07943] tmpdir_base=
[master.org:07943] universe=***@master.org:default-universe-7933
[master.org:07943] universe_persistence=0
[master.org:07943] universe_scope=
[master.org:07943] universe_console=0
[master.org:07943] universe_uri=
[master.org:07943] universe_script=
[master.org:07943] dss_buffer_type=0
[master.org:07943] dss_page_size=1
[master.org:07943] ns_base_verbose=0
[master.org:07943] ns=
[master.org:07943] ns_proxy_debug=0
[master.org:07943] ns_proxy_maxsize=2147483647
[master.org:07943] ns_proxy_blocksize=512
[master.org:07943] ns_proxy_priority=0
[master.org:07943] ns_replica_debug=0
[master.org:07943] ns_replica_isolate=0
[master.org:07943] ns_replica_maxsize=2147483647
[master.org:07943] ns_replica_blocksize=512
[master.org:07943] ns_replica_priority=0
[master.org:07943] errmgr_base_verbose=0
[master.org:07943] errmgr=
[master.org:07943] errmgr_hnp_debug=0
[master.org:07943] errmgr_hnp_priority=0
[master.org:07943] errmgr_orted_debug=0
[master.org:07943] errmgr_orted_priority=0
[master.org:07943] errmgr_proxy_debug=0
[master.org:07943] errmgr_proxy_priority=0
[master.org:07943] rml_base_debug=0
[master.org:07943] rml=
[master.org:07943] rml_base_verbose=0
[master.org:07943] oob=
[master.org:07943] oob_base_verbose=0
[master.org:07943] oob_tcp_peer_limit=-1
[master.org:07943] oob_tcp_peer_retries=60
[master.org:07943] oob_tcp_debug=0
[master.org:07943] oob_tcp_sndbuf=131072
[master.org:07943] oob_tcp_rcvbuf=131072
[master.org:07943] oob_tcp_if_include=
[master.org:07943] oob_tcp_if_exclude=
[master.org:07943] oob_tcp_connect_sleep=1
[master.org:07943] oob_tcp_listen_mode=event
[master.org:07943] oob_tcp_listen_thread_max_queue=10
[master.org:07943] oob_tcp_listen_thread_max_time=10
[master.org:07943] oob_tcp_accept_spin_count=10
[master.org:07943] oob_tcp_priority=0
[master.org:07943] oob_base_include=
[master.org:07943] oob_base_exclude=
[master.org:07943] orte_timing=0
[master.org:07943] oob_xcast_timing=0
[master.org:07943] oob_xcast_mode=linear
[master.org:07943] rml_oob_priority=0
[master.org:07943] gpr_base_verbose=0
[master.org:07943] gpr_base_maxsize=2147483647
[master.org:07943] gpr_base_blocksize=512
[master.org:07943] gpr=
[master.org:07943] gpr_null_priority=0
[master.org:07943] gpr_proxy_debug=0
[master.org:07943] gpr_proxy_priority=0
[master.org:07943] gpr_replica_debug=0
[master.org:07943] gpr_replica_isolate=0
[master.org:07943] gpr_replica_priority=0
[master.org:07943] schema_base_verbose=0
[master.org:07943] schema=
[master.org:07943] sds=
[master.org:07943] sds_base_verbose=0
[master.org:07943] sds_env_priority=0
[master.org:07943] sds_pipe_priority=0
[master.org:07943] sds_seed_priority=0
[master.org:07943] sds_singleton_priority=0
[master.org:07943] sds_slurm_priority=0
[master.org:07943] ns_nds=env
[master.org:07943] ns_nds_name=
[master.org:07943] ns_nds_cellid=0
[master.org:07943] ns_nds_jobid=1
[master.org:07943] ns_nds_vpid=0
[master.org:07943] ns_nds_vpid_start=0
[master.org:07943] ns_nds_num_procs=32
[master.org:07943] rds_base_verbose=0
[master.org:07943] rds=proxy
[master.org:07943] rds_proxy_priority=0
[master.org:07943] ras_base_verbose=0
[master.org:07943] ras=proxy
[master.org:07943] rmaps_base_verbose=0
[master.org:07943] rmaps_base_schedule_policy=unspec
[master.org:07943] rmaps_base_pernode=0
[master.org:07943] rmaps_base_n_pernode=-1
[master.org:07943] rmaps_base_no_schedule_local=0
[master.org:07943] rmaps_base_no_oversubscribe=0
[master.org:07943] rmaps_base_display_map=0
[master.org:07943] rmaps=proxy
[master.org:07943] pls_base_reuse_daemons=0
[master.org:07943] pls=proxy
[master.org:07943] pls_base_verbose=0
[master.org:07943] pls_proxy_priority=0
[master.org:07943] odls_base_verbose=0
[master.org:07943] odls_base_sigkill_timeout=1
[master.org:07943] odls=
[master.org:07943] odls_default_priority=0
[master.org:07943] rmgr_base_verbose=0
[master.org:07943] rmgr=proxy
[master.org:07943] rmgr_proxy_priority=0
[master.org:07943] smr_base_verbose=0
[master.org:07943] smr=
[master.org:07943] iof_base_window_size=4096
[master.org:07943] iof_base_service=0.0.0
[master.org:07943] iof_base_verbose=0
[master.org:07943] iof=
[master.org:07943] iof_proxy_priority=0
[master.org:07943] iof_svc_priority=0
[master.org:07943] mpi_param_check=1
[master.org:07943] mpi_yield_when_idle=0
[master.org:07943] mpi_event_tick_rate=-1
[master.org:07943] mpi_show_handle_leaks=0
[master.org:07943] mpi_no_free_handles=0
[master.org:07943] mpi_show_mca_params=1
[master.org:07943] mpi_show_mca_params_file=
[master.org:07943] mpi_paffinity_alone=0
[master.org:07943] mpi_keep_peer_hostnames=1
[master.org:07943] mpi_abort_delay=0
[master.org:07943] mpi_abort_print_stack=0
[master.org:07943] mpi_preconnect_all=0
[master.org:07943] mpi_preconnect_oob=0
[master.org:07943] mpi_leave_pinned=0
[master.org:07943] mpi_leave_pinned_pipeline=0
[master.org:07943] mpi_warn_if_thread_multiple=1
[master.org:07943] mpi_warn_if_progress_threads=1
[master.org:07943] allocator=
[master.org:07943] allocator_base_verbose=0
[master.org:07943] allocator_basic_priority=0
[master.org:07943] allocator_bucket_num_buckets=30
[master.org:07943] allocator_bucket_priority=0
[master.org:07943] rcache=
[master.org:07943] rcache_base_verbose=0
[master.org:07943] rcache_vma_priority=0
[master.org:07943] mpool=
[master.org:07943] mpool_base_verbose=0
[master.org:07943] mpool_rdma_rcache_name=vma
[master.org:07943] mpool_rdma_rcache_size_limit=0
[master.org:07943] mpool_rdma_print_stats=0
[master.org:07943] mpool_rdma_priority=0
[master.org:07943] mpool_sm_allocator=bucket
[master.org:07943] mpool_sm_max_size=536870912
[master.org:07943] mpool_sm_min_size=134217728
[master.org:07943] mpool_sm_per_peer_size=33554432
[master.org:07943] mpool_sm_verbose=0
[master.org:07943] mpool_sm_priority=0
[master.org:07943] mpool_base_use_mem_hooks=0
[master.org:07943] mpool_use_mem_hooks=0
[master.org:07943] mpool_base_disable_sbrk=0
[master.org:07943] mpool_disable_sbrk=0
[master.org:07943] pml=
[master.org:07943] pml_base_verbose=0
[master.org:07943] mtl=
[master.org:07943] mtl_base_verbose=0
[master.org:07943] pml_cm_free_list_num=4
[master.org:07943] pml_cm_free_list_max=-1
[master.org:07943] pml_cm_free_list_inc=64
[master.org:07943] pml_cm_priority=30
[master.org:07943] pml_ob1_free_list_num=4
[master.org:07943] pml_ob1_free_list_max=-1
[master.org:07943] pml_ob1_free_list_inc=64
[master.org:07943] pml_ob1_priority=20
[master.org:07943] pml_ob1_eager_limit=131072
[master.org:07943] pml_ob1_send_pipeline_depth=3
[master.org:07943] pml_ob1_recv_pipeline_depth=4
[master.org:07943] pml_ob1_use_early_completion=1
[master.org:07943] bml=
[master.org:07943] bml_base_verbose=0
[master.org:07943] bml_r2_show_unreach_errors=1
[master.org:07943] bml_r2_priority=0
[master.org:07943] btl_base_debug=0
[master.org:07943] btl=self,openib
[master.org:07943] btl_base_verbose=0
[master.org:07943] btl_self_free_list_num=0
[master.org:07943] btl_self_free_list_max=-1
[master.org:07943] btl_self_free_list_inc=32
[master.org:07943] btl_self_eager_limit=131072
[master.org:07943] btl_self_min_send_size=262144
[master.org:07943] btl_self_max_send_size=262144
[master.org:07943] btl_self_min_rdma_size=2147483647
[master.org:07943] btl_self_max_rdma_size=2147483647
[master.org:07943] btl_self_exclusivity=65536
[master.org:07943] btl_self_flags=10
[master.org:07943] btl_self_priority=0
[master.org:07943] btl_openib_verbose=0
[master.org:07943] btl_openib_warn_no_hca_params_found=1
[master.org:07943] btl_openib_warn_default_gid_prefix=1
[master.org:07943] btl_openib_want_fork_support=-1
[master.org:07943]
btl_openib_hca_param_files=/usr/mpi/gcc/openmpi-1.2.8/share/openmpi/mca-btl-openib-hca-params.ini
[master.org:07943] btl_openib_max_btls=-1
[master.org:07943] btl_openib_free_list_num=8
[master.org:07943] btl_openib_free_list_max=-1
[master.org:07943] btl_openib_free_list_inc=32
[master.org:07943] btl_openib_mpool=rdma
[master.org:07943] btl_openib_reg_mru_len=16
[master.org:07943] btl_openib_ib_cq_size=1000
[master.org:07943] btl_openib_ib_sg_list_size=4
[master.org:07943] btl_openib_ib_pkey_ix=0
[master.org:07943] btl_openib_ib_pkey_val=0
[master.org:07943] btl_openib_ib_psn=0
[master.org:07943] btl_openib_ib_qp_ous_rd_atom=4
[master.org:07943] btl_openib_ib_mtu=3
[master.org:07943] btl_openib_ib_min_rnr_timer=5
[master.org:07943] btl_openib_ib_timeout=10
[master.org:07943] btl_openib_ib_retry_count=7
[master.org:07943] btl_openib_ib_rnr_retry=7
[master.org:07943] btl_openib_ib_max_rdma_dst_ops=4
[master.org:07943] btl_openib_ib_service_level=0
[master.org:07943] btl_openib_ib_static_rate=0
[master.org:07943] btl_openib_exclusivity=1024
[master.org:07943] btl_openib_rd_num=8
[master.org:07943] btl_openib_rd_low=6
[master.org:07943] btl_openib_rd_win=4
[master.org:07943] btl_openib_use_srq=0
[master.org:07943] btl_openib_srq_rd_max=1000
[master.org:07943] btl_openib_srq_rd_per_peer=16
[master.org:07943] btl_openib_srq_sd_max=8
[master.org:07943] btl_openib_use_eager_rdma=1
[master.org:07943] btl_openib_eager_rdma_threshold=16
[master.org:07943] btl_openib_max_eager_rdma=16
[master.org:07943] btl_openib_eager_rdma_num=16
[master.org:07943] btl_openib_btls_per_lid=1
[master.org:07943] btl_openib_max_lmc=0
[master.org:07943] btl_openib_buffer_alignment=64
[master.org:07943] btl_openib_eager_limit=12288
[master.org:07943] btl_openib_min_send_size=32768
[master.org:07943] btl_openib_max_send_size=65536
[master.org:07943] btl_openib_min_rdma_size=1048576
[master.org:07943] btl_openib_max_rdma_size=1048576
[master.org:07943] btl_openib_flags=54
[master.org:07943] btl_openib_bandwidth=0
[master.org:07943] btl_openib_have_fork_support=1
[master.org:07943] btl_openib_priority=0
[master.org:07943] btl_base_include=
[master.org:07943] btl_base_exclude=
[master.org:07943] btl_base_warn_component_unused=1
[master.org:07943] coll=
[master.org:07943] coll_base_verbose=0
[master.org:07943] coll_basic_priority=10
[master.org:07943] coll_basic_crossover=4
[master.org:07943] coll_self_priority=75
[master.org:07943] coll_sm_priority=0
[master.org:07943] coll_sm_control_size=4096
[master.org:07943] coll_sm_bootstrap_filename=shared_mem_sm_bootstrap
[master.org:07943] coll_sm_bootstrap_num_segments=8
[master.org:07943] coll_sm_fragment_size=8192
[master.org:07943] coll_sm_mpool=sm
[master.org:07943] coll_sm_comm_in_use_flags=2
[master.org:07943] coll_sm_comm_num_segments=8
[master.org:07943] coll_sm_tree_degree=4
[master.org:07943] coll_sm_shared_mem_used_bootstrap=216
[master.org:07943] coll_sm_info_num_procs=4
[master.org:07943] coll_sm_shared_mem_used_data=548864
[master.org:07943] coll_tuned_priority=30
[master.org:07943] coll_tuned_pre_allocate_memory_comm_size_limit=32768
[master.org:07943] coll_tuned_init_tree_fanout=4
[master.org:07943] coll_tuned_init_chain_fanout=4
[master.org:07943] coll_tuned_use_dynamic_rules=0
[master.org:07943] osc=
[master.org:07943] osc_base_verbose=0
[master.org:07943] osc_pt2pt_no_locks=0
[master.org:07943] osc_pt2pt_eager_limit=16384
[master.org:07943] osc_pt2pt_priority=0
[master.org:07957]
mca_param_files=/root/.openmpi/mca-params.conf:/usr/mpi/gcc/openmpi-1.2.8/etc/openmpi-mca-params.conf
[master.org:07957] opal_signal=6,7,8,11
[master.org:07957]
mca_component_path=/usr/mpi/gcc/openmpi-1.2.8/lib/openmpi:/root/.openmpi/components
[master.org:07957] mca_verbose=
[master.org:07957] mca_component_show_load_errors=1
[master.org:07957] mca_component_disable_dlopen=0
[master.org:07957] paffinity_base_verbose=0
[master.org:07957] paffinity=
[master.org:07957] paffinity_linux_priority=10
[master.org:07957] paffinity_linux_have_cpu_set_t=1
[master.org:07957] paffinity_linux_CPU_ZERO_ok=1
[master.org:07957] paffinity_linux_sched_setaffinity_num_params=3
[master.org:07957] memcpy=
[master.org:07957] memcpy_base_verbose=0
[master.org:07957] memory=
[master.org:07957] memory_base_verbose=0
[master.org:07957] memory_ptmalloc2_priority=0
[master.org:07957] backtrace=
[master.org:07957] backtrace_base_verbose=0
[master.org:07957] backtrace_execinfo_priority=0
[master.org:07957] timer=
[master.org:07957] timer_base_verbose=0
[master.org:07957] timer_linux_priority=0
[master.org:07957] ompi_timing=0
[master.org:07957] orte_debug=0
[master.org:07957] orte_debug_daemons_file=0
[master.org:07957] orte_no_daemonize=0
[master.org:07957] orte_debug_daemons=0
[master.org:07957] orte_base_user_debugger=totalview @mpirun@ -a
@mpirun_args@ : ddt -n @np@ -start @executable@ @executable_argv@
@single_app@ : fxp @mpirun@ -a @mpirun_args@
[master.org:07957] orte_abort_timeout=10
[master.org:07957] orte_base_nodename=master
[master.org:07957] seed=0
[master.org:07957] orte_app_num=0
[master.org:07957] tmpdir_base=
[master.org:07957] universe=***@master.org:default-universe-7947
[master.org:07957] universe_persistence=0
[master.org:07957] universe_scope=
[master.org:07957] universe_console=0
[master.org:07957] universe_uri=
[master.org:07957] universe_script=
[master.org:07957] dss_buffer_type=0
[master.org:07957] dss_page_size=1
[master.org:07957] ns_base_verbose=0
[master.org:07957] ns=
[master.org:07957] ns_proxy_debug=0
[master.org:07957] ns_proxy_maxsize=2147483647
[master.org:07957] ns_proxy_blocksize=512
[master.org:07957] ns_proxy_priority=0
[master.org:07957] ns_replica_debug=0
[master.org:07957] ns_replica_isolate=0
[master.org:07957] ns_replica_maxsize=2147483647
[master.org:07957] ns_replica_blocksize=512
[master.org:07957] ns_replica_priority=0
[master.org:07957] errmgr_base_verbose=0
[master.org:07957] errmgr=
[master.org:07957] errmgr_hnp_debug=0
[master.org:07957] errmgr_hnp_priority=0
[master.org:07957] errmgr_orted_debug=0
[master.org:07957] errmgr_orted_priority=0
[master.org:07957] errmgr_proxy_debug=0
[master.org:07957] errmgr_proxy_priority=0
[master.org:07957] rml_base_debug=0
[master.org:07957] rml=
[master.org:07957] rml_base_verbose=0
[master.org:07957] oob=
[master.org:07957] oob_base_verbose=0
[master.org:07957] oob_tcp_peer_limit=-1
[master.org:07957] oob_tcp_peer_retries=60
[master.org:07957] oob_tcp_debug=0
[master.org:07957] oob_tcp_sndbuf=131072
[master.org:07957] oob_tcp_rcvbuf=131072
[master.org:07957] oob_tcp_if_include=
[master.org:07957] oob_tcp_if_exclude=
[master.org:07957] oob_tcp_connect_sleep=1
[master.org:07957] oob_tcp_listen_mode=event
[master.org:07957] oob_tcp_listen_thread_max_queue=10
[master.org:07957] oob_tcp_listen_thread_max_time=10
[master.org:07957] oob_tcp_accept_spin_count=10
[master.org:07957] oob_tcp_priority=0
[master.org:07957] oob_base_include=
[master.org:07957] oob_base_exclude=
[master.org:07957] orte_timing=0
[master.org:07957] oob_xcast_timing=0
[master.org:07957] oob_xcast_mode=linear
[master.org:07957] rml_oob_priority=0
[master.org:07957] gpr_base_verbose=0
[master.org:07957] gpr_base_maxsize=2147483647
[master.org:07957] gpr_base_blocksize=512
[master.org:07957] gpr=
[master.org:07957] gpr_null_priority=0
[master.org:07957] gpr_proxy_debug=0
[master.org:07957] gpr_proxy_priority=0
[master.org:07957] gpr_replica_debug=0
[master.org:07957] gpr_replica_isolate=0
[master.org:07957] gpr_replica_priority=0
[master.org:07957] schema_base_verbose=0
[master.org:07957] schema=
[master.org:07957] sds=
[master.org:07957] sds_base_verbose=0
[master.org:07957] sds_env_priority=0
[master.org:07957] sds_pipe_priority=0
[master.org:07957] sds_seed_priority=0
[master.org:07957] sds_singleton_priority=0
[master.org:07957] sds_slurm_priority=0
[master.org:07957] ns_nds=env
[master.org:07957] ns_nds_name=
[master.org:07957] ns_nds_cellid=0
[master.org:07957] ns_nds_jobid=1
[master.org:07957] ns_nds_vpid=0
[master.org:07957] ns_nds_vpid_start=0
[master.org:07957] ns_nds_num_procs=32
[master.org:07957] rds_base_verbose=0
[master.org:07957] rds=proxy
[master.org:07957] rds_proxy_priority=0
[master.org:07957] ras_base_verbose=0
[master.org:07957] ras=proxy
[master.org:07957] rmaps_base_verbose=0
[master.org:07957] rmaps_base_schedule_policy=unspec
[master.org:07957] rmaps_base_pernode=0
[master.org:07957] rmaps_base_n_pernode=-1
[master.org:07957] rmaps_base_no_schedule_local=0
[master.org:07957] rmaps_base_no_oversubscribe=0
[master.org:07957] rmaps_base_display_map=0
[master.org:07957] rmaps=proxy
[master.org:07957] pls_base_reuse_daemons=0
[master.org:07957] pls=proxy
[master.org:07957] pls_base_verbose=0
[master.org:07957] pls_proxy_priority=0
[master.org:07957] odls_base_verbose=0
[master.org:07957] odls_base_sigkill_timeout=1
[master.org:07957] odls=
[master.org:07957] odls_default_priority=0
[master.org:07957] rmgr_base_verbose=0
[master.org:07957] rmgr=proxy
[master.org:07957] rmgr_proxy_priority=0
[master.org:07957] smr_base_verbose=0
[master.org:07957] smr=
[master.org:07957] iof_base_window_size=4096
[master.org:07957] iof_base_service=0.0.0
[master.org:07957] iof_base_verbose=0
[master.org:07957] iof=
[master.org:07957] iof_proxy_priority=0
[master.org:07957] iof_svc_priority=0
[master.org:07957] mpi_param_check=1
[master.org:07957] mpi_yield_when_idle=0
[master.org:07957] mpi_event_tick_rate=-1
[master.org:07957] mpi_show_handle_leaks=0
[master.org:07957] mpi_no_free_handles=0
[master.org:07957] mpi_show_mca_params=1
[master.org:07957] mpi_show_mca_params_file=
[master.org:07957] mpi_paffinity_alone=0
[master.org:07957] mpi_keep_peer_hostnames=1
[master.org:07957] mpi_abort_delay=0
[master.org:07957] mpi_abort_print_stack=0
[master.org:07957] mpi_preconnect_all=0
[master.org:07957] mpi_preconnect_oob=0
[master.org:07957] mpi_leave_pinned=0
[master.org:07957] mpi_leave_pinned_pipeline=0
[master.org:07957] mpi_warn_if_thread_multiple=1
[master.org:07957] mpi_warn_if_progress_threads=1
[master.org:07957] allocator=
[master.org:07957] allocator_base_verbose=0
[master.org:07957] allocator_basic_priority=0
[master.org:07957] allocator_bucket_num_buckets=30
[master.org:07957] allocator_bucket_priority=0
[master.org:07957] rcache=
[master.org:07957] rcache_base_verbose=0
[master.org:07957] rcache_vma_priority=0
[master.org:07957] mpool=
[master.org:07957] mpool_base_verbose=0
[master.org:07957] mpool_rdma_rcache_name=vma
[master.org:07957] mpool_rdma_rcache_size_limit=0
[master.org:07957] mpool_rdma_print_stats=0
[master.org:07957] mpool_rdma_priority=0
[master.org:07957] mpool_sm_allocator=bucket
[master.org:07957] mpool_sm_max_size=536870912
[master.org:07957] mpool_sm_min_size=134217728
[master.org:07957] mpool_sm_per_peer_size=33554432
[master.org:07957] mpool_sm_verbose=0
[master.org:07957] mpool_sm_priority=0
[master.org:07957] mpool_base_use_mem_hooks=0
[master.org:07957] mpool_use_mem_hooks=0
[master.org:07957] mpool_base_disable_sbrk=0
[master.org:07957] mpool_disable_sbrk=0
[master.org:07957] pml=
[master.org:07957] pml_base_verbose=0
[master.org:07957] mtl=
[master.org:07957] mtl_base_verbose=0
[master.org:07957] pml_cm_free_list_num=4
[master.org:07957] pml_cm_free_list_max=-1
[master.org:07957] pml_cm_free_list_inc=64
[master.org:07957] pml_cm_priority=30
[master.org:07957] pml_ob1_free_list_num=4
[master.org:07957] pml_ob1_free_list_max=-1
[master.org:07957] pml_ob1_free_list_inc=64
[master.org:07957] pml_ob1_priority=20
[master.org:07957] pml_ob1_eager_limit=131072
[master.org:07957] pml_ob1_send_pipeline_depth=3
[master.org:07957] pml_ob1_recv_pipeline_depth=4
[master.org:07957] pml_ob1_use_early_completion=1
[master.org:07957] bml=
[master.org:07957] bml_base_verbose=0
[master.org:07957] bml_r2_show_unreach_errors=1
[master.org:07957] bml_r2_priority=0
[master.org:07957] btl_base_debug=0
[master.org:07957] btl=self,openib
[master.org:07957] btl_base_verbose=0
[master.org:07957] btl_self_free_list_num=0
[master.org:07957] btl_self_free_list_max=-1
[master.org:07957] btl_self_free_list_inc=32
[master.org:07957] btl_self_eager_limit=131072
[master.org:07957] btl_self_min_send_size=262144
[master.org:07957] btl_self_max_send_size=262144
[master.org:07957] btl_self_min_rdma_size=2147483647
[master.org:07957] btl_self_max_rdma_size=2147483647
[master.org:07957] btl_self_exclusivity=65536
[master.org:07957] btl_self_flags=10
[master.org:07957] btl_self_priority=0
[master.org:07957] btl_openib_verbose=0
[master.org:07957] btl_openib_warn_no_hca_params_found=1
[master.org:07957] btl_openib_warn_default_gid_prefix=1
[master.org:07957] btl_openib_want_fork_support=-1
...

Post by Jim Kress ORG
1.3.2_test_crafted_input_file_mca_params.out

[master.org:11857]
mca_param_files=/root/.openmpi/mca-params.conf:/usr/mpi/gcc/openmpi-1.3.2/etc/openmpi-mca-params.conf (default value)
[master.org:11857] mca_base_param_file_prefix= (default value)
[master.org:11857]
mca_base_param_file_path=/usr/mpi/gcc/openmpi-1.3.2/share/openmpi/amca-param-sets:/state/partition1/orca (default value)
[master.org:11857] mca_base_param_file_path_force= (default value)
[master.org:11857] opal_signal=6,7,8,11 (default value)
[master.org:11857] opal_set_max_sys_limits=0 (default value)
[master.org:11857] dss_buffer_type=0 (default value)
[master.org:11857] dss_buffer_initial_size=128 (default value)
[master.org:11857] dss_buffer_threshold_size=1024 (default value)
[master.org:11857] opal_event_include=all (API override)
[master.org:11857] ompi_timing=0 (default value)
[master.org:11857]
mca_component_path=/usr/mpi/gcc/openmpi-1.3.2/lib/openmpi:/root/.openmpi/components (default value)
[master.org:11857] mca_verbose= (default value)
[master.org:11857] mca_component_show_load_errors=1 (default value)
[master.org:11857] mca_component_disable_dlopen=0 (default value)
[master.org:11857] paffinity_base_verbose=0 (default value)
[master.org:11857] paffinity= (default value)
[master.org:11857] paffinity_linux_priority=10 (default value)
[master.org:11857] paffinity_linux_plpa_version=1.3rc4 (default value)
[master.org:11857] memcpy= (default value)
[master.org:11857] memcpy_base_verbose=0 (default value)
[master.org:11857] memory= (default value)
[master.org:11857] memory_base_verbose=0 (default value)
[master.org:11857] memory_ptmalloc2_priority=0 (default value)
[master.org:11857] memchecker_base_verbose=0 (default value)
[master.org:11857] memchecker= (default value)
[master.org:11857] backtrace= (default value)
[master.org:11857] backtrace_base_verbose=0 (default value)
[master.org:11857] backtrace_execinfo_priority=0 (default value)
[master.org:11857] timer= (default value)
[master.org:11857] timer_base_verbose=0 (default value)
[master.org:11857] timer_linux_priority=0 (default value)
[master.org:11857] carto_base_verbose=0 (default value)
[master.org:11857] carto= (default value)
[master.org:11857] carto_auto_detect_priority=11 (default value)
[master.org:11857] carto_file_path= (default value)
[master.org:11857] carto_file_priority=10 (default value)
[master.org:11857] opal_cr_verbose=0 (default value)
[master.org:11857] ft_cr_enabled=0 (default value)
[master.org:11857] opal_cr_enable_timer=0 (default value)
[master.org:11857] opal_cr_enable_timer_barrier=0 (default value)
[master.org:11857] opal_cr_timer_target_rank=0 (default value)
[master.org:11857] opal_cr_is_tool=0 (default value)
[master.org:11857] opal_cr_signal=10 (default value)
[master.org:11857] opal_cr_debug_sigpipe=0 (default value)
[master.org:11857] opal_cr_tmp_dir=/tmp (default value)
[master.org:11857] orte_base_help_aggregate=1 (default value)
[master.org:11857] orte_tmpdir_base= (default value)
[master.org:11857] orte_no_session_dirs= (default value)
[master.org:11857] orte_debug=0 (default value)
[master.org:11857] orte_debug_verbose=-1 (default value)
[master.org:11857] orte_debug_daemons=0 (default value)
[master.org:11857] orte_debug_daemons_file=0 (default value)
[master.org:11857] orte_leave_session_attached=0 (default value)
[master.org:11857] orte_do_not_launch=0 (default value)
[master.org:11857] orte_daemon_spin=0 (default value)
[master.org:11857] orte_daemon_fail=-1 (default value)
[master.org:11857] orte_daemon_fail_delay=0 (default value)
[master.org:11857] orte_heartbeat_rate=0 (default value)
[master.org:11857] orte_startup_timeout=0 (default value)
[master.org:11857] orte_timing=0 (default value)
[master.org:11857] orte_base_user_debugger=totalview @mpirun@ -a
@mpirun_args@ : ddt -n @np@ -start @executable@ @executable_argv@
@single_app@ : fxp @mpirun@ -a @mpirun_args@ (default value)
[master.org:11857] orte_abort_timeout=1 (default value)
[master.org:11857] orte_timeout_step=1000 (default value)
[master.org:11857] orte_default_hostfile= (default value)
[master.org:11857] orte_keep_fqdn_hostnames=0 (default value)
[master.org:11857] orte_contiguous_nodes=2147483647 (default value)
[master.org:11857] orte_tag_output=0 (default value)
[master.org:11857] orte_xml_output=0 (default value)
[master.org:11857] orte_timestamp_output=0 (default value)
[master.org:11857] orte_output_filename= (default value)
[master.org:11857] orte_show_resolved_nodenames=0 (default value)
[master.org:11857] orte_hetero_apps=0 (default value)
[master.org:11857] orte_launch_agent=orted (default value)
[master.org:11857] orte_allocation_required=0 (default value)
[master.org:11857] orte_xterm= (default value)
[master.org:11857] orte_forward_job_control=0 (default value)
[master.org:11857] ess=env (environment)
[master.org:11857] ess_base_verbose=0 (default value)
[master.org:11857] ess_env_priority=0 (default value)
[master.org:11857] orte_ess_jobid=3016687617 (environment)
[master.org:11857] orte_ess_vpid=0 (environment)
[master.org:11857] rml_wrapper= (default value)
[master.org:11857] rml= (default value)
[master.org:11857] rml_base_verbose=0 (default value)
[master.org:11857] oob= (default value)
[master.org:11857] oob_base_verbose=0 (default value)
[master.org:11857] oob_tcp_verbose=0 (default value)
[master.org:11857] oob_tcp_peer_limit=-1 (default value)
[master.org:11857] oob_tcp_peer_retries=60 (default value)
[master.org:11857] oob_tcp_debug=0 (default value)
[master.org:11857] oob_tcp_sndbuf=131072 (default value)
[master.org:11857] oob_tcp_rcvbuf=131072 (default value)
[master.org:11857] oob_tcp_if_include= (default value)
[master.org:11857] oob_tcp_if_exclude= (default value)
[master.org:11857] oob_tcp_connect_sleep=1 (default value)
[master.org:11857] oob_tcp_listen_mode=event (default value)
[master.org:11857] oob_tcp_listen_thread_max_queue=10 (default value)
[master.org:11857] oob_tcp_listen_thread_wait_time=10 (default value)
[master.org:11857] oob_tcp_port_min_v4=0 (default value)
[master.org:11857] oob_tcp_port_range_v4=65535 (default value)
[master.org:11857] oob_tcp_disable_family=0 (default value)
[master.org:11857] oob_tcp_port_min_v6=0 (default value)
[master.org:11857] oob_tcp_port_range_v6=65535 (default value)
[master.org:11857] oob_tcp_priority=0 (default value)
[master.org:11857] rml_oob_priority=0 (default value)
[master.org:11857] opal_if_do_not_resolve=0 (default value)
[master.org:11857] routed= (default value)
[master.org:11857] routed_base_verbose=0 (default value)
[master.org:11857] routed_binomial_priority=0 (default value)
[master.org:11857] routed_direct_priority=0 (default value)
[master.org:11857] routed_linear_priority=0 (default value)
[master.org:11857] grpcomm= (default value)
[master.org:11857] grpcomm_base_verbose=0 (default value)
[master.org:11857] grpcomm_bad_priority=0 (default value)
[master.org:11857] grpcomm_basic_priority=0 (default value)
[master.org:11857] orte_cr_verbose=0 (default value)
[master.org:11857] notifier= (default value)
[master.org:11857] notifier_base_verbose=0 (default value)
[master.org:11857] notifier_syslog_priority=0 (default value)
[master.org:11857] mpi_param_check=1 (default value)
[master.org:11857] mpi_yield_when_idle=0 (environment)
[master.org:11857] mpi_event_tick_rate=-1 (default value)
[master.org:11857] mpi_show_handle_leaks=0 (default value)
[master.org:11857] mpi_no_free_handles=0 (default value)
[master.org:11857] mpi_show_mpi_alloc_mem_leaks=0 (default value)
[master.org:11857] mpi_show_mca_params=1 (environment)
[master.org:11857] mpi_show_mca_params_file= (default value)
[master.org:11857] mpi_keep_peer_hostnames=1 (default value)
[master.org:11857] mpi_abort_delay=0 (default value)
[master.org:11857] mpi_abort_print_stack=0 (default value)
[master.org:11857] mpi_preconnect_mpi=0 (default value)
[master.org:11857] mpi_preconnect_all=0 (default value)
[master.org:11857] mpi_leave_pinned=-1 (default value)
[master.org:11857] mpi_leave_pinned_pipeline=0 (default value)
[master.org:11857] mpi_paffinity_alone=0 (default value)
[master.org:11857] mpi_warn_on_fork=1 (default value)
[master.org:11857] mpi_have_sparse_group_storage=0 (default value)
[master.org:11857] mpi_use_sparse_group_storage=0 (default value)
[master.org:11857] allocator= (default value)
[master.org:11857] allocator_base_verbose=0 (default value)
[master.org:11857] allocator_basic_priority=0 (default value)
[master.org:11857] allocator_bucket_num_buckets=30 (default value)
[master.org:11857] allocator_bucket_priority=0 (default value)
[master.org:11857] rcache= (default value)
[master.org:11857] rcache_base_verbose=0 (default value)
[master.org:11857] rcache_vma_priority=0 (default value)
[master.org:11857] mpool= (default value)
[master.org:11857] mpool_base_verbose=0 (default value)
[master.org:11857] mpool_fake_priority=0 (default value)
[master.org:11857] mpool_rdma_rcache_name=vma (default value)
[master.org:11857] mpool_rdma_rcache_size_limit=0 (default value)
[master.org:11857] mpool_rdma_print_stats=0 (default value)
[master.org:11857] mpool_rdma_priority=0 (default value)
[master.org:11857] mpool_sm_allocator=bucket (default value)
[master.org:11857] mpool_sm_min_size=67108864 (default value)
[master.org:11857] mpool_sm_verbose=0 (default value)
[master.org:11857] mpool_sm_priority=0 (default value)
[master.org:11857] pml_base_verbose=0 (default value)
[master.org:11857] pml= (default value)
[master.org:11857] mtl= (default value)
[master.org:11857] mtl_base_verbose=0 (default value)
[master.org:11857] pml_cm_free_list_num=4 (default value)
[master.org:11857] pml_cm_free_list_max=-1 (default value)
[master.org:11857] pml_cm_free_list_inc=64 (default value)
[master.org:11857] pml_cm_priority=30 (default value)
[master.org:11857] pml_csum_free_list_num=4 (default value)
[master.org:11857] pml_csum_free_list_max=-1 (default value)
[master.org:11857] pml_csum_free_list_inc=64 (default value)
[master.org:11857] pml_csum_send_pipeline_depth=3 (default value)
[master.org:11857] pml_csum_recv_pipeline_depth=4 (default value)
[master.org:11857] pml_csum_rdma_put_retries_limit=5 (default value)
[master.org:11857] pml_csum_max_rdma_per_request=4 (default value)
[master.org:11857] pml_csum_max_send_per_range=4 (default value)
[master.org:11857] pml_csum_unexpected_limit=128 (default value)
[master.org:11857] pml_csum_allocator=bucket (default value)
[master.org:11857] bml= (default value)
[master.org:11857] bml_base_verbose=0 (default value)
[master.org:11857] bml_r2_show_unreach_errors=1 (default value)
[master.org:11857] bml_r2_priority=0 (default value)
[master.org:11857] btl_base_verbose=0 (default value)
[master.org:11857] btl=self,openib
(file:/usr/mpi/gcc/openmpi-1.3.2/etc/openmpi-mca-params.conf)
[master.org:11857] btl_openib_verbose=0 (default value)
[master.org:11857] btl_openib_warn_no_device_params_found=1 (default
value)
[master.org:11857] btl_openib_warn_no_hca_params_found=1 (default value)
[master.org:11857] btl_openib_warn_default_gid_prefix=1 (default value)
[master.org:11857] btl_openib_warn_nonexistent_if=1 (default value)
[master.org:11857] btl_openib_want_fork_support=-1 (default value)
[master.org:11857]
btl_openib_device_param_files=/usr/mpi/gcc/openmpi-1.3.2/share/openmpi/mca-btl-openib-device-params.ini (default value)
[master.org:11857]
btl_openib_hca_param_files=/usr/mpi/gcc/openmpi-1.3.2/share/openmpi/mca-btl-openib-device-params.ini (default value)
[master.org:11857] btl_openib_device_type=all (default value)
[master.org:11857] btl_openib_max_btls=-1 (default value)
[master.org:11857] btl_openib_free_list_num=8 (default value)
[master.org:11857] btl_openib_free_list_max=-1 (default value)
[master.org:11857] btl_openib_free_list_inc=32 (default value)
[master.org:11857] btl_openib_mpool=rdma (default value)
[master.org:11857] btl_openib_reg_mru_len=16 (default value)
[master.org:11857] btl_openib_cq_size=1000 (default value)
[master.org:11857] btl_openib_ib_cq_size=1000 (default value)
[master.org:11857] btl_openib_max_inline_data=-1 (default value)
[master.org:11857] btl_openib_ib_max_inline_data=-1 (default value)
[master.org:11857] btl_openib_pkey=0 (default value)
[master.org:11857] btl_openib_ib_pkey_val=0 (default value)
[master.org:11857] btl_openib_psn=0 (default value)
[master.org:11857] btl_openib_ib_psn=0 (default value)
[master.org:11857] btl_openib_ib_qp_ous_rd_atom=4 (default value)
[master.org:11857] btl_openib_mtu=3 (default value)
[master.org:11857] btl_openib_ib_mtu=3 (default value)
[master.org:11857] btl_openib_ib_min_rnr_timer=25 (default value)
[master.org:11857] btl_openib_ib_timeout=20 (default value)
[master.org:11857] btl_openib_ib_retry_count=7 (default value)
[master.org:11857] btl_openib_ib_rnr_retry=7 (default value)
[master.org:11857] btl_openib_ib_max_rdma_dst_ops=4 (default value)
[master.org:11857] btl_openib_ib_service_level=0 (default value)
[master.org:11857] btl_openib_use_eager_rdma=-1 (default value)
[master.org:11857] btl_openib_eager_rdma_threshold=16 (default value)
[master.org:11857] btl_openib_max_eager_rdma=16 (default value)
[master.org:11857] btl_openib_eager_rdma_num=16 (default value)
[master.org:11857] btl_openib_btls_per_lid=1 (default value)
[master.org:11857] btl_openib_max_lmc=0 (default value)
[master.org:11857] btl_openib_enable_apm_over_lmc=0 (default value)
[master.org:11857] btl_openib_enable_apm_over_ports=0 (default value)
[master.org:11857] btl_openib_use_async_event_thread=1 (default value)
[master.org:11857] btl_openib_buffer_alignment=64 (default value)
[master.org:11857] btl_openib_use_message_coalescing=1 (default value)
[master.org:11857] btl_openib_cq_poll_ratio=100 (default value)
[master.org:11857] btl_openib_eager_rdma_poll_ratio=100 (default value)
[master.org:11857] btl_openib_hp_cq_poll_per_progress=10 (default value)
[master.org:11857] btl_openib_have_fork_support=1 (default value)
[master.org:11857] btl_openib_exclusivity=1024 (default value)
[master.org:11857] btl_openib_flags=310 (default value)
[master.org:11857] btl_openib_rndv_eager_limit=12288 (default value)
[master.org:11857] btl_openib_eager_limit=12288 (default value)
[master.org:11857] btl_openib_max_send_size=65536 (default value)
[master.org:11857] btl_openib_rdma_pipeline_send_length=1048576 (default
value)
[master.org:11857] btl_openib_rdma_pipeline_frag_size=1048576 (default
value)
[master.org:11857] btl_openib_min_rdma_pipeline_size=262144 (default
value)
[master.org:11857] btl_openib_bandwidth=800 (default value)
[master.org:11857] btl_openib_latency=10 (default value)
[master.org:11857]
btl_openib_receive_queues=P,128,256,192,128:S,2048,256,128,32:S,12288,256,128,32:S,65536,256,128,32 (default value)
[master.org:11857] btl_openib_if_include= (default value)
[master.org:11857] btl_openib_if_exclude= (default value)
[master.org:11857] btl_openib_ipaddr_include= (default value)
[master.org:11857] btl_openib_ipaddr_exclude= (default value)
[master.org:11857] btl_openib_cpc_include= (default value)
[master.org:11857] btl_openib_cpc_exclude= (default value)
[master.org:11857] btl_openib_connect_oob_priority=50 (default value)
[master.org:11857] btl_openib_connect_xoob_priority=60 (default value)
[master.org:11857] btl_openib_connect_rdmacm_priority=30 (default value)
[master.org:11857] btl_openib_connect_rdmacm_port=0 (default value)
[master.org:11857] btl_openib_connect_rdmacm_resolve_timeout=2000
(default value)
[master.org:11857]
btl_openib_connect_rdmacm_reject_causes_connect_error=0 (default value)
[master.org:11857] btl_openib_priority=0 (default value)
[master.org:11857] btl_self_free_list_num=0 (default value)
[master.org:11857] btl_self_free_list_max=-1 (default value)
[master.org:11857] btl_self_free_list_inc=32 (default value)
[master.org:11857] btl_self_exclusivity=65536 (default value)
[master.org:11857] btl_self_flags=10 (default value)
[master.org:11857] btl_self_rndv_eager_limit=131072 (default value)
[master.org:11857] btl_self_eager_limit=131072 (default value)
[master.org:11857] btl_self_max_send_size=262144 (default value)
[master.org:11857] btl_self_rdma_pipeline_send_length=2147483647
(default value)
[master.org:11857] btl_self_rdma_pipeline_frag_size=2147483647 (default
value)
[master.org:11857] btl_self_min_rdma_pipeline_size=0 (default value)
[master.org:11857] btl_self_bandwidth=100 (default value)
[master.org:11857] btl_self_latency=0 (default value)
[master.org:11857] btl_self_priority=0 (default value)
[master.org:11857] btl_base_include= (default value)
[master.org:11857] btl_base_exclude= (default value)
[master.org:11857] btl_base_warn_component_unused=1 (default value)
[master.org:11857] pml_csum_priority=0 (default value)
[master.org:11857] pml_ob1_free_list_num=4 (default value)
[master.org:11857] pml_ob1_free_list_max=-1 (default value)
[master.org:11857] pml_ob1_free_list_inc=64 (default value)
[master.org:11857] pml_ob1_priority=20 (default value)
[master.org:11857] pml_ob1_send_pipeline_depth=3 (default value)
[master.org:11857] pml_ob1_recv_pipeline_depth=4 (default value)
[master.org:11857] pml_ob1_rdma_put_retries_limit=5 (default value)
[master.org:11857] pml_ob1_max_rdma_per_request=4 (default value)
[master.org:11857] pml_ob1_max_send_per_range=4 (default value)
[master.org:11857] pml_ob1_unexpected_limit=128 (default value)
[master.org:11857] pml_ob1_allocator=bucket (default value)
[master.org:11857] pml_v_priority=-1 (default value)
[master.org:11857] pml_v_output=stderr (default value)
[master.org:11857] pml_v_verbose=0 (default value)
[master.org:11857] vprotocol= (default value)
[master.org:11857] coll= (default value)
[master.org:11857] coll_base_verbose=0 (default value)
[master.org:11857] coll_basic_priority=10 (default value)
[master.org:11857] coll_basic_crossover=4 (default value)
[master.org:11857] coll_hierarch_priority=0 (default value)
[master.org:11857] coll_hierarch_verbose=0 (default value)
[master.org:11857] coll_hierarch_use_rdma=0 (default value)
[master.org:11857] coll_hierarch_ignore_sm=0 (default value)
[master.org:11857] coll_hierarch_detection_alg=2 (default value)
[master.org:11857] coll_inter_priority=40 (default value)
[master.org:11857] coll_inter_verbose=0 (default value)
[master.org:11857] coll_self_priority=75 (default value)
[master.org:11857] coll_sm_priority=0 (default value)
[master.org:11857] coll_sm_control_size=4096 (default value)
[master.org:11857] coll_sm_bootstrap_filename=shared_mem_sm_bootstrap
(default value)
[master.org:11857] coll_sm_bootstrap_num_segments=8 (default value)
[master.org:11857] coll_sm_fragment_size=8192 (default value)
[master.org:11857] coll_sm_mpool=sm (default value)
[master.org:11857] coll_sm_comm_in_use_flags=2 (default value)
[master.org:11857] coll_sm_comm_num_segments=8 (default value)
[master.org:11857] coll_sm_tree_degree=4 (default value)
[master.org:11857] coll_sm_shared_mem_used_bootstrap=200 (default value)
[master.org:11857] coll_sm_info_num_procs=4 (default value)
[master.org:11857] coll_sm_shared_mem_used_data=548864 (default value)
[master.org:11857] coll_sync_priority=50 (default value)
[master.org:11857] coll_sync_barrier_before=0 (default value)
[master.org:11857] coll_sync_barrier_after=0 (default value)
[master.org:11857] coll_tuned_priority=30 (default value)
[master.org:11857] coll_tuned_pre_allocate_memory_comm_size_limit=32768
(default value)
[master.org:11857] coll_tuned_init_tree_fanout=4 (default value)
[master.org:11857] coll_tuned_init_chain_fanout=4 (default value)
[master.org:11857] coll_tuned_use_dynamic_rules=0 (default value)
[master.org:11857] osc= (default value)
[master.org:11857] osc_base_verbose=0 (default value)
[master.org:11857] osc_pt2pt_no_locks=0 (default value)
[master.org:11857] osc_pt2pt_eager_limit=16384 (default value)
[master.org:11857] osc_pt2pt_priority=0 (default value)
[master.org:11857] osc_rdma_eager_send=1 (default value)
[master.org:11857] osc_rdma_use_buffers=0 (default value)
[master.org:11857] osc_rdma_use_rdma=0 (default value)
[master.org:11857] osc_rdma_rdma_completion_wait=1 (default value)
[master.org:11857] osc_rdma_no_locks=0 (default value)
[master.org:11857] osc_rdma_priority=0 (default value)
[master.org:11857] btl_openib_bandwidth_mthca0=800 (default value)
[master.org:11857] btl_openib_bandwidth_mthca0:1=800 (default value)
[master.org:11857] btl_openib_bandwidth_mthca0:1:3=800 (default value)
[master.org:11857] btl_openib_latency_mthca0=10 (default value)
[master.org:11857] btl_openib_latency_mthca0:1=10 (default value)
[master.org:11857] btl_openib_latency_mthca0:1:3=10 (default value)
[master.org:12011]
mca_param_files=/root/.openmpi/mca-params.conf:/usr/mpi/gcc/openmpi-1.3.2/etc/openmpi-mca-params.conf (default value)
[master.org:12011] mca_base_param_file_prefix= (default value)
[master.org:12011]
mca_base_param_file_path=/usr/mpi/gcc/openmpi-1.3.2/share/openmpi/amca-param-sets:/state/partition1/orca (default value)
[master.org:12011] mca_base_param_file_path_force= (default value)
[master.org:12011] opal_signal=6,7,8,11 (default value)
[master.org:12011] opal_set_max_sys_limits=0 (default value)
[master.org:12011] dss_buffer_type=0 (default value)
[master.org:12011] dss_buffer_initial_size=128 (default value)
[master.org:12011] dss_buffer_threshold_size=1024 (default value)
[master.org:12011] opal_event_include=all (API override)
[master.org:12011] ompi_timing=0 (default value)
[master.org:12011]
mca_component_path=/usr/mpi/gcc/openmpi-1.3.2/lib/openmpi:/root/.openmpi/components (default value)
[master.org:12011] mca_verbose= (default value)
[master.org:12011] mca_component_show_load_errors=1 (default value)
[master.org:12011] mca_component_disable_dlopen=0 (default value)
[master.org:12011] paffinity_base_verbose=0 (default value)
[master.org:12011] paffinity= (default value)
[master.org:12011] paffinity_linux_priority=10 (default value)
[master.org:12011] paffinity_linux_plpa_version=1.3rc4 (default value)
[master.org:12011] memcpy= (default value)
[master.org:12011] memcpy_base_verbose=0 (default value)
[master.org:12011] memory= (default value)
[master.org:12011] memory_base_verbose=0 (default value)
[master.org:12011] memory_ptmalloc2_priority=0 (default value)
[master.org:12011] memchecker_base_verbose=0 (default value)
[master.org:12011] memchecker= (default value)
[master.org:12011] backtrace= (default value)
[master.org:12011] backtrace_base_verbose=0 (default value)
[master.org:12011] backtrace_execinfo_priority=0 (default value)
[master.org:12011] timer= (default value)
[master.org:12011] timer_base_verbose=0 (default value)
[master.org:12011] timer_linux_priority=0 (default value)
[master.org:12011] carto_base_verbose=0 (default value)
[master.org:12011] carto= (default value)
[master.org:12011] carto_auto_detect_priority=11 (default value)
[master.org:12011] carto_file_path= (default value)
[master.org:12011] carto_file_priority=10 (default value)
[master.org:12011] opal_cr_verbose=0 (default value)
[master.org:12011] ft_cr_enabled=0 (default value)
[master.org:12011] opal_cr_enable_timer=0 (default value)
[master.org:12011] opal_cr_enable_timer_barrier=0 (default value)
[master.org:12011] opal_cr_timer_target_rank=0 (default value)
[master.org:12011] opal_cr_is_tool=0 (default value)
[master.org:12011] opal_cr_signal=10 (default value)
[master.org:12011] opal_cr_debug_sigpipe=0 (default value)
[master.org:12011] opal_cr_tmp_dir=/tmp (default value)
[master.org:12011] orte_base_help_aggregate=1 (default value)
[master.org:12011] orte_tmpdir_base= (default value)
[master.org:12011] orte_no_session_dirs= (default value)
[master.org:12011] orte_debug=0 (default value)
[master.org:12011] orte_debug_verbose=-1 (default value)
[master.org:12011] orte_debug_daemons=0 (default value)
[master.org:12011] orte_debug_daemons_file=0 (default value)
[master.org:12011] orte_leave_session_attached=0 (default value)
[master.org:12011] orte_do_not_launch=0 (default value)
[master.org:12011] orte_daemon_spin=0 (default value)
[master.org:12011] orte_daemon_fail=-1 (default value)
[master.org:12011] orte_daemon_fail_delay=0 (default value)
[master.org:12011] orte_heartbeat_rate=0 (default value)
[master.org:12011] orte_startup_timeout=0 (default value)
[master.org:12011] orte_timing=0 (default value)
[master.org:12011] orte_base_user_debugger=totalview @mpirun@ -a
@mpirun_args@ : ddt -n @np@ -start @executable@ @executable_argv@
@single_app@ : fxp @mpirun@ -a @mpirun_args@ (default value)
[master.org:12011] orte_abort_timeout=1 (default value)
[master.org:12011] orte_timeout_step=1000 (default value)
[master.org:12011] orte_default_hostfile= (default value)
[master.org:12011] orte_keep_fqdn_hostnames=0 (default value)
[master.org:12011] orte_contiguous_nodes=2147483647 (default value)
[master.org:12011] orte_tag_output=0 (default value)
[master.org:12011] orte_xml_output=0 (default value)
[master.org:12011] orte_timestamp_output=0 (default value)
[master.org:12011] orte_output_filename= (default value)
[master.org:12011] orte_show_resolved_nodenames=0 (default value)
[master.org:12011] orte_hetero_apps=0 (default value)
[master.org:12011] orte_launch_agent=orted (default value)
[master.org:12011] orte_allocation_required=0 (default value)
[master.org:12011] orte_xterm= (default value)
[master.org:12011] orte_forward_job_control=0 (default value)
[master.org:12011] ess=env (environment)
[master.org:12011] ess_base_verbose=0 (default value)
[master.org:12011] ess_env_priority=0 (default value)
[master.org:12011] orte_ess_jobid=3004366849 (environment)
[master.org:12011] orte_ess_vpid=0 (environment)
[master.org:12011] rml_wrapper= (default value)
[master.org:12011] rml= (default value)
[master.org:12011] rml_base_verbose=0 (default value)
[master.org:12011] oob= (default value)
[master.org:12011] oob_base_verbose=0 (default value)
[master.org:12011] oob_tcp_verbose=0 (default value)
[master.org:12011] oob_tcp_peer_limit=-1 (default value)
[master.org:12011] oob_tcp_peer_retries=60 (default value)
[master.org:12011] oob_tcp_debug=0 (default value)
[master.org:12011] oob_tcp_sndbuf=131072 (default value)
[master.org:12011] oob_tcp_rcvbuf=131072 (default value)
[master.org:12011] oob_tcp_if_include= (default value)
[master.org:12011] oob_tcp_if_exclude= (default value)
[master.org:12011] oob_tcp_connect_sleep=1 (default value)
[master.org:12011] oob_tcp_listen_mode=event (default value)
[master.org:12011] oob_tcp_listen_thread_max_queue=10 (default value)
[master.org:12011] oob_tcp_listen_thread_wait_time=10 (default value)
[master.org:12011] oob_tcp_port_min_v4=0 (default value)
[master.org:12011] oob_tcp_port_range_v4=65535 (default value)
[master.org:12011] oob_tcp_disable_family=0 (default value)
[master.org:12011] oob_tcp_port_min_v6=0 (default value)
[master.org:12011] oob_tcp_port_range_v6=65535 (default value)
[master.org:12011] oob_tcp_priority=0 (default value)
...

Are there any useful clues here? Please note, the app launches a number
of parallel programs in a sequence determined by the input file. The
same input file was used for both runs.

Jim

Post by Jim Kress ORG
Sounds very strange, indeed. You might want to check that your app is
-mca mpi_show_mca_params file,env
to your cmd line. This will cause rank=0 to output the MCA params it
thinks were set via the default files and/or environment (including
cmd line).
Ralph

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Pavel Shamis (Pasha)

2009-06-23 11:24:09 UTC

Jim,
Can you please share with us you mca conf file.

Pasha.

Gus Correa

2009-06-23 18:00:40 UTC

Hi Jim, list

Have you checked if configure caught your IB libraries properly?
IIRR there has been some changes since 1.2.8 on how configure searches
for libraries (e.g. finding libnuma was a problem, now fixed).
Chances are that if you used some old script
or command line to run configure, it may not have worked as you expected.

Check the output of ompi_info -config.
It should show -lrdmacm -libverbs, otherwise it skipped IB.
In this case you can reconfigure, pointing to the IB library location.

If you have a log of your configure step you can also search it for
openib, libverbs, etc, to see if it did what you expected.

I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Post by Pavel Shamis (Pasha)
Jim,
Can you please share with us you mca conf file.
Pasha.

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress

2009-06-23 19:29:50 UTC

Are you speaking of the configure for the application or for OpenMPI?

I have no control over the application since it is provided as an executable
only.

Jim

-----Original Message-----
Sent: Tuesday, June 23, 2009 2:01 PM
To: Open MPI Users
Subject: Re: [OMPI users] 50% performance reduction due to
OpenMPI v 1.3.2 forcing all MPI traffic over Ethernet instead
of using Infiniband
Hi Jim, list
Have you checked if configure caught your IB libraries properly?
IIRR there has been some changes since 1.2.8 on how configure
searches for libraries (e.g. finding libnuma was a problem,
now fixed).
Chances are that if you used some old script or command line
to run configure, it may not have worked as you expected.
Check the output of ompi_info -config.
It should show -lrdmacm -libverbs, otherwise it skipped IB.
In this case you can reconfigure, pointing to the IB library location.
If you have a log of your configure step you can also search
it for openib, libverbs, etc, to see if it did what you expected.
I hope this helps,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Post by Pavel Shamis (Pasha)
Jim,
Can you please share with us you mca conf file.
Pasha.

Post by Jim Kress ORG
For the app I am using, ORCA (a Quantum Chemistry

program), when it

Post by Jim Kress ORG
was compiled using openMPI 1.2.8 and run under 1.2.8 with the
btl=self,openib
the app ran fine with no traffic over my Ethernet network and all
traffic over my Infiniband network.
However, now that ORCA has been recompiled with openMPI v1.3.2 and
run under 1.3.2 (using the same openmpi-mca-params.conf file), the
performance has been reduced by 50% and all the MPI

traffic is going

Post by Jim Kress ORG
over the Ethernet network.
As a matter of fact, the openMPI v1.3.2 performance now

looks exactly

Post by Jim Kress ORG
like the performance I get if I use MPICH 1.2.7.
1) How could this have happened?
2) How can I fix it?
a 50% reduction in performance is just not acceptable. Ideas/
suggestions would be appreciated.
Jim
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Gus Correa

2009-06-23 20:18:15 UTC

Hi Jim

Post by Jim Kress
Are you speaking of the configure for the application or for OpenMPI?

I am speaking of OpenMPI configure.
Scott Hamilton also mentioned this,
when he answered you in the Rocks mailing list.

Post by Jim Kress
I have no control over the application since it is provided as an executable
only.

I understand that ORCA is a black box or a black killer whale,
but if your OpenMPI was not built with IB,
there is no hope that ORCA will use IB.
Did you do ompi_info -config?

Some of my builds missed libnuma, others missed libtorque,
eventually I got it right.
Then the OpenMPI team changed configure
(somewhere along the 1.3 series), so I had to change again.

If the libraries aren't in standard places (/usr/lib /usr/lib64),
and the includes also (/usr/include) you need to tell configure where
they are. See the OpenMPI README file and FAQ.

My $0.02.
Gus Correa

PS - BTW, what is your advice for a fellow trying to run the
computational chemistry software from Schroedinger.com?
I know nothing of comput-chem, an area where you are the pro.
This question came on the Beowulf list, and apparently the darn piece of
software requires MPICH-1, and only executables are provided.
I know (You told me!) that MPICH-1 fails miserably with those
p4 errors on later Linux kernels, which is what the poor guy
is getting.
If he at least had the object files he could try to link to MPICH2,
but apparently he only has executables (statically linked to MPICH-1,
I suppose).

---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Post by Jim Kress
Jim

Post by Pavel Shamis (Pasha)
Jim,
Can you please share with us you mca conf file.
Pasha.

Post by Jim Kress ORG
For the app I am using, ORCA (a Quantum Chemistry

program), when it

traffic is going

Post by Jim Kress ORG
over the Ethernet network.
As a matter of fact, the openMPI v1.3.2 performance now

looks exactly

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress

2009-06-23 18:35:35 UTC

I assume you a referring to the openmpi-mca-params.conf file

As I indicated previously, my first run was with the line

btl=self,openib

As the only entry in the openmpi-mca-params.conf file. This my default
setting and was what I used, and it worked well, for v 1.2.8

Then I tried

btl=self,openib
mpi_yield_when_idle=0

As the only entries in the openmpi-mca-params.conf file. No difference in
the results.

Then I tried

btl=self,openib
mpi_yield_when_idle=0

As the only entries in the openmpi-mca-params.conf file and also set the
environment variable OMPI_MCA_mpi_leave_pinned=0
No difference in the results.

What else can I provide?

By the way, did you read the message where I retracted my assumption about
MPI traffic being forced over Ethernet?

Jim

-----Original Message-----
Sent: Tuesday, June 23, 2009 7:24 AM
To: Open MPI Users
Subject: Re: [OMPI users] 50% performance reduction due to
OpenMPI v 1.3.2 forcing all MPI traffic over Ethernet instead
of using Infiniband
Jim,
Can you please share with us you mca conf file.
Pasha.

Post by Jim Kress ORG
For the app I am using, ORCA (a Quantum Chemistry program), when it
was compiled using openMPI 1.2.8 and run under 1.2.8 with the
btl=self,openib
the app ran fine with no traffic over my Ethernet network and all
traffic over my Infiniband network.
However, now that ORCA has been recompiled with openMPI

v1.3.2 and run

Post by Jim Kress ORG
under 1.3.2 (using the same openmpi-mca-params.conf file), the
performance has been reduced by 50% and all the MPI traffic

is going

Post by Jim Kress ORG
over the Ethernet network.
As a matter of fact, the openMPI v1.3.2 performance now

looks exactly

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Ralph Castain

2009-06-23 18:42:57 UTC

Assuming you aren't oversubscribing your nodes, set mpi_paffinity_alone=1.

BTW: did you set that mpi_show_mca_params option to ensure the app is
actually seeing these params?

Post by Jim Kress
I assume you a referring to the openmpi-mca-params.conf file
As I indicated previously, my first run was with the line
btl=self,openib
As the only entry in the openmpi-mca-params.conf file. This my default
setting and was what I used, and it worked well, for v 1.2.8
Then I tried
btl=self,openib
mpi_yield_when_idle=0
As the only entries in the openmpi-mca-params.conf file. No difference in
the results.
Then I tried
btl=self,openib
mpi_yield_when_idle=0
As the only entries in the openmpi-mca-params.conf file and also set the
environment variable OMPI_MCA_mpi_leave_pinned=0
No difference in the results.
What else can I provide?
By the way, did you read the message where I retracted my assumption about
MPI traffic being forced over Ethernet?
Jim

v1.3.2 and run

Post by Jim Kress ORG
under 1.3.2 (using the same openmpi-mca-params.conf file), the
performance has been reduced by 50% and all the MPI traffic

is going

Post by Jim Kress ORG
over the Ethernet network.
As a matter of fact, the openMPI v1.3.2 performance now

looks exactly

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jim Kress

2009-06-23 19:31:56 UTC