[OMPI users] Mellanox EDR performance

[Yong Qin]

Hi,

I'm wondering if anybody who has done perf testing on Mellanox EDR with
OMPI can shed some light here?

We have a pair of EDR HCAs connected back to back. We are testing with two
dual-socked Intel Xeon E5-2670v3 (Haswell) nodes @2.30GHz, 64GB memory. OS
is Scientific Linux 6.7 with kernel
2.6.32-642.6.2.el6.x86_64, vanilla OFED 3.18-2. HCAs are running the latest
FW. OMPI 2.0.2.

OSU bandwidth test only delivers ~5.5 GB/s at 4MB message size, latency is
~2.7 us at 0B message size. Both are far behind the claimed values. RDMA
perf on the same set up was not too shabby - bandwidth ~10.6 GB/s, latency
~ 1.0 us.

So I'm wondering if I'm missing anything in the OMPI setup that causes such
a huge delta? OMPI command was simplely: mpirun -np 2 -H host1,host2 -mca
btl openib,sm,self osu_bw

Thanks,

Yong Qin

[Yong Qin]

OK, did some testing with MVAPICH and everything is normal so this is
clearly with OMPI. Is there anything that I should try?

Thanks,

Yong Qin

Post by Yong Qin
Hi,
I'm wondering if anybody who has done perf testing on Mellanox EDR with
OMPI can shed some light here?
We have a pair of EDR HCAs connected back to back. We are testing with two
is Scientific Linux 6.7 with kernel
2.6.32-642.6.2.el6.x86_64, vanilla OFED 3.18-2. HCAs are running the
latest FW. OMPI 2.0.2.
OSU bandwidth test only delivers ~5.5 GB/s at 4MB message size, latency is
~2.7 us at 0B message size. Both are far behind the claimed values. RDMA
perf on the same set up was not too shabby - bandwidth ~10.6 GB/s, latency
~ 1.0 us.
So I'm wondering if I'm missing anything in the OMPI setup that causes
such a huge delta? OMPI command was simplely: mpirun -np 2 -H host1,host2
-mca btl openib,sm,self osu_bw
Thanks,
Yong Qin

[Yong Qin]

I'm not sure why nobody has encountered this issue on the mailing list.
After some fiddling I was finally able to isolated to a performance
regression introduced between 2.0.1 and 2.0.2. While trying to binary
search the exact commit causing the performance regression my colleague
brought this to my attention,

https://github.com/open-mpi/ompi/issues/3003

Yes this is exactly the issue!

So just in case anybody run into the same issue again ...

Post by Yong Qin
OK, did some testing with MVAPICH and everything is normal so this is
clearly with OMPI. Is there anything that I should try?
Thanks,
Yong Qin

Post by Yong Qin
Hi,
I'm wondering if anybody who has done perf testing on Mellanox EDR with
OMPI can shed some light here?
We have a pair of EDR HCAs connected back to back. We are testing with
OS is Scientific Linux 6.7 with kernel
2.6.32-642.6.2.el6.x86_64, vanilla OFED 3.18-2. HCAs are running the
latest FW. OMPI 2.0.2.
OSU bandwidth test only delivers ~5.5 GB/s at 4MB message size, latency
is ~2.7 us at 0B message size. Both are far behind the claimed values. RDMA
perf on the same set up was not too shabby - bandwidth ~10.6 GB/s, latency
~ 1.0 us.
So I'm wondering if I'm missing anything in the OMPI setup that causes
such a huge delta? OMPI command was simplely: mpirun -np 2 -H host1,host2
-mca btl openib,sm,self osu_bw
Thanks,
Yong Qin

[g***@rist.or.jp]

Thanks for sharing your findings.

just to be clear, your application is running at full speed.

only MPI_Wtime() is busted, so timers used internally in your app might

mislead you and suggests performance are worst than what they really are

Cheers,

Gilles

----- Original Message -----

I'm not sure why nobody has encountered this issue on the mailing
list. After some fiddling I was finally able to isolated to a
performance regression introduced between 2.0.1 and 2.0.2. While trying
to binary search the exact commit causing the performance regression my
colleague brought this to my attention,

https://github.com/open-mpi/ompi/issues/3003

Yes this is exactly the issue!

So just in case anybody run into the same issue again ...

On Tue, Mar 7, 2017 at 12:59 PM, Yong Qin <***@gmail.com> wrote:

OK, did some testing with MVAPICH and everything is normal so
this is clearly with OMPI. Is there anything that I should try?

Thanks,

Yong Qin

On Mon, Mar 6, 2017 at 11:46 AM, Yong Qin <***@gmail.com>
wrote:

Hi,

I'm wondering if anybody who has done perf testing on
Mellanox EDR with OMPI can shed some light here?

We have a pair of EDR HCAs connected back to back. We are
testing with two dual-socked Intel Xeon E5-2670v3 (Haswell) nodes @2.
30GHz, 64GB memory. OS is Scientific Linux 6.7 with kernel
2.6.32-642.6.2.el6.x86_64, vanilla OFED 3.18-2. HCAs are
running the latest FW. OMPI 2.0.2.

OSU bandwidth test only delivers ~5.5 GB/s at 4MB message
size, latency is ~2.7 us at 0B message size. Both are far behind the
claimed values. RDMA perf on the same set up was not too shabby -
bandwidth ~10.6 GB/s, latency ~ 1.0 us.

So I'm wondering if I'm missing anything in the OMPI setup
that causes such a huge delta? OMPI command was simplely: mpirun -np 2 -
H host1,host2 -mca btl openib,sm,self osu_bw

Thanks,

Yong Qin