[OMPI users] Eager RDMA causing slow osu

Discussion:

[OMPI users] Eager RDMA causing slow osu_bibw with 3.0.0

Ben Menadue

2018-04-05 07:03:01 UTC

Hi,

Weâve just been running some OSU benchmarks with OpenMPI 3.0.0 and noticed that osu_bibw gives nowhere near the bandwidth Iâd expect (this is on FDR IB). However, osu_bw is fine.

If I disable eager RDMA, then osu_bibw gives the expected numbers. Similarly, if I increase the number of eager RDMA buffers, it gives the expected results.

OpenMPI 1.10.7 gives consistent, reasonable numbers with default settings, but theyâre not as good as 3.0.0 (when tuned) for large buffers. The same option changes produce no different in the performance for 1.10.7.

I was wondering if anyone else has noticed anything similar, and if this is unexpected, if anyone has a suggestion on how to investigate further?

Thanks,
Ben

Hereâs are the numbers:

3.0.0, osu_bw, default settings

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bw

# OSU MPI Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
1 1.13
2 2.29
4 4.63
8 9.21
16 18.18
32 36.46
64 69.95
128 128.55
256 250.74
512 451.54
1024 829.44
2048 1475.87
4096 2119.99
8192 3452.37
16384 2866.51
32768 4048.17
65536 5030.54
131072 5573.81
262144 5861.61
524288 6015.15
1048576 6099.46
2097152 989.82
4194304 989.81

3.0.0, osu_bibw, default settings

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
1 0.00
2 0.01
4 0.01
8 0.02
16 0.04
32 0.09
64 0.16
128 135.30
256 265.35
512 499.92
1024 949.22
2048 1440.27
4096 1960.09
8192 3166.97
16384 127.62
32768 165.12
65536 312.80
131072 1120.03
262144 4724.01
524288 4545.93
1048576 5186.51
2097152 989.84
4194304 989.88

3.0.0, osu_bibw, eager RDMA disabled

mpirun -mca btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
1 1.49
2 2.97
4 5.96
8 11.98
16 23.95
32 47.39
64 93.57
128 153.82
256 304.69
512 572.30
1024 1003.52
2048 1083.89
4096 1879.32
8192 2785.18
16384 3535.77
32768 5614.72
65536 8113.69
131072 9666.74
262144 10738.97
524288 11247.02
1048576 11416.50
2097152 989.88
4194304 989.88

3.0.0, osu_bibw, increased eager RDMA buffer count

mpirun -mca btl_openib_eager_rdma_num 32768 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
1 1.42
2 2.84
4 5.67
8 11.18
16 22.46
32 44.65
64 83.10
128 154.00
256 291.63
512 537.66
1024 942.35
2048 1433.09
4096 2356.40
8192 1998.54
16384 3584.82
32768 5523.08
65536 7717.63
131072 9419.50
262144 10564.77
524288 11104.71
1048576 11130.75
2097152 7943.89
4194304 5270.00

1.10.7, osu_bibw, default settings

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
1 1.70
2 3.45
4 6.95
8 13.68
16 27.41
32 53.80
64 105.34
128 164.40
256 324.63
512 623.95
1024 1127.35
2048 1784.58
4096 3305.45
8192 3697.55
16384 4935.75
32768 7186.28
65536 8996.94
131072 9301.78
262144 4691.36
524288 7039.18
1048576 7213.33
2097152 9601.41
4194304 9281.31

Ben Menadue

2018-04-05 07:12:46 UTC

Permalink

Hi,

Another interesting point. I noticed that the last two message sizes tested (2MB and 4MB) are lower than expected for both osu_bw and osu_bibw. Increasing the minimum size to use the RDMA pipeline to above these sizes brings those two data-points up to scratch for both benchmarks:

3.0.0, osu_bw, no rdma for large messages

mpirun -mca btl_openib_min_rdma_pipeline_size 4194304 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bw -m 2097152:4194304

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
2097152 6133.22
4194304 6054.06

3.0.0, osu_bibw, eager rdma disabled, no rdma for large messages

mpirun -mca btl_openib_min_rdma_pipeline_size 4194304 -mca btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw -m 2097152:4194304

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
2097152 11397.85
4194304 11389.64

This makes me think something odd is going on in the RDMA pipeline.

Cheers,
Ben

Hi,
Weâve just been running some OSU benchmarks with OpenMPI 3.0.0 and noticed that osu_bibw gives nowhere near the bandwidth Iâd expect (this is on FDR IB). However, osu_bw is fine.
If I disable eager RDMA, then osu_bibw gives the expected numbers. Similarly, if I increase the number of eager RDMA buffers, it gives the expected results.
OpenMPI 1.10.7 gives consistent, reasonable numbers with default settings, but theyâre not as good as 3.0.0 (when tuned) for large buffers. The same option changes produce no different in the performance for 1.10.7.
I was wondering if anyone else has noticed anything similar, and if this is unexpected, if anyone has a suggestion on how to investigate further?
Thanks,
Ben
3.0.0, osu_bw, default settings

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bw

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

mpirun -mca btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

mpirun -mca btl_openib_eager_rdma_num 32768 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

Howard Pritchard

2018-04-05 15:47:26 UTC

Permalink

Hello Ben,

Thanks for the info. You would probably be better off installing UCX on
your cluster and rebuilding your Open MPI with the
--with-ucx
configure option.

Here's what I'm seeing with Open MPI 3.0.1 on a ConnectX5 based cluster
using ob1/openib BTL:

mpirun -map-by ppr:1:node -np 2 ./osu_bibw

# OSU MPI Bi-Directional Bandwidth Test v5.1

# Size Bandwidth (MB/s)

1 0.00

2 0.00

4 0.01

8 0.02

16 0.04

32 0.07

64 0.13

128 273.64

256 485.04

512 869.51

1024 1434.99

2048 2208.12

4096 3055.67

8192 3896.93

16384 89.29

32768 252.59

65536 614.42

131072 22878.74

262144 23846.93

524288 24256.23

1048576 24498.27

2097152 24615.64

4194304 24632.58

export OMPI_MCA_pml=ucx

# OSU MPI Bi-Directional Bandwidth Test v5.1

# Size Bandwidth (MB/s)

1 4.57

2 8.95

4 17.67

8 35.99

16 71.99

32 141.56

64 208.86

128 410.32

256 495.56

512 1455.98

1024 2414.78

2048 3008.19

4096 5351.62

8192 5563.66

16384 5945.16

32768 6061.33

65536 21376.89

131072 23462.99

262144 24064.56

524288 24366.84

1048576 24550.75

2097152 24649.03

4194304 24693.77

You can get ucx off of GitHub

https://github.com/openucx/ucx/releases

There is also a pre-release version of UCX (1.3.0RCX?) packaged as an RPM

available in MOFED 4.3. See

http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

I was using UCX 1.2.2 for the results above.

Good luck,

Howard

Post by Ben Menadue
Hi,
Another interesting point. I noticed that the last two message sizes
tested (2MB and 4MB) are lower than expected for both osu_bw and osu_bibw.
Increasing the minimum size to use the RDMA pipeline to above these sizes
*3.0.0, osu_bw, no rdma for large messages*

mpirun -mca btl_openib_min_rdma_pipeline_size 4194304 -map-by

ppr:1:node -np 2 -H r6,r7 ./osu_bw -m 2097152:4194304
# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
2097152 6133.22
4194304 6054.06
*3.0.0, osu_bibw, eager rdma disabled, no rdma for large messages*

mpirun -mca btl_openib_min_rdma_pipeline_size 4194304 -mca

btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw -m
2097152:4194304
# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
2097152 11397.85
4194304 11389.64
This makes me think something odd is going on in the RDMA pipeline.
Cheers,
Ben
Hi,
Weâve just been running some OSU benchmarks with OpenMPI 3.0.0 and noticed
that *osu_bibw* gives nowhere near the bandwidth Iâd expect (this is on
FDR IB). However, *osu_bw* is fine.
If I disable eager RDMA, then *osu_bibw* gives the expected
numbers. Similarly, if I increase the number of eager RDMA buffers, it
gives the expected results.
OpenMPI 1.10.7 gives consistent, reasonable numbers with default settings,
but theyâre not as good as 3.0.0 (when tuned) for large buffers. The same
option changes produce no different in the performance for 1.10.7.
I was wondering if anyone else has noticed anything similar, and if this
is unexpected, if anyone has a suggestion on how to investigate further?
Thanks,
Ben
*3.0.0, osu_bw, default settings*

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bw

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

mpirun -mca btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H

r6,r7 ./osu_bibw
# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
1 1.49
2 2.97
4 5.96
8 11.98
16 23.95
32 47.39
64 93.57
128 153.82
256 304.69
512 572.30
1024 1003.52
2048 1083.89
4096 1879.32
8192 2785.18
16384 3535.77
32768 5614.72
65536 8113.69
131072 9666.74
262144 10738.97
524288 11247.02
1048576 11416.50
2097152 989.88
4194304 989.88
*3.0.0, **osu_bibw, increased eager RDMA buffer count*

mpirun -mca btl_openib_eager_rdma_num 32768 -map-by ppr:1:node -np 2 -H

r6,r7 ./osu_bibw
# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
1 1.42
2 2.84
4 5.67
8 11.18
16 22.46
32 44.65
64 83.10
128 154.00
256 291.63
512 537.66
1024 942.35
2048 1433.09
4096 2356.40
8192 1998.54
16384 3584.82
32768 5523.08
65536 7717.63
131072 9419.50
262144 10564.77
524288 11104.71
1048576 11130.75
2097152 7943.89
4194304 5270.00
*1.10.7, osu_bibw, default settings*

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

Nathan Hjelm

2018-04-05 16:48:08 UTC

Permalink

Honestly, this is a configuration issue with the openib btl. There is no reason to keep either eager RDMA nor is there a reason to pipeline RDMA. I haven't found an app where either of these "features" helps you with infiniband. You have the right idea with the parameter changes but Howard is correct, for Mellanox the future is UCX not verbs. I would try it and see if it works for you but if it doesn't I would set those two parameters in your /etc/openmpi-mca-params.conf and run like that.

-Nathan

On Apr 05, 2018, at 01:18 AM, Ben Menadue <***@nci.org.au> wrote:

Hi,

Another interesting point. I noticed that the last two message sizes tested (2MB and 4MB) are lower than expected for both osu_bw and osu_bibw. Increasing the minimum size to use the RDMA pipeline to above these sizes brings those two data-points up to scratch for both benchmarks:

3.0.0, osu_bw, no rdma for large messages

mpirun -mca btl_openib_min_rdma_pipeline_size 4194304 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bw -m 2097152:4194304

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# SizeÂ Â Â Â Bandwidth (MB/s)
2097152Â Â Â Â Â Â Â Â 6133.22
4194304Â Â Â Â Â Â Â Â 6054.06

3.0.0, osu_bibw, eager rdma disabled, no rdma for large messages

mpirun -mca btl_openib_min_rdma_pipeline_size 4194304 -mca btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw -m 2097152:4194304

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# SizeÂ Â Â Bandwidth (MB/s)
2097152 Â Â Â Â Â Â 11397.85
4194304 Â Â Â Â Â Â 11389.64

This makes me think something odd is going on in the RDMA pipeline.

Cheers,
Ben

On 5 Apr 2018, at 5:03 pm, Ben Menadue <***@nci.org.au> wrote:
Hi,

Weâve just been running some OSU benchmarks with OpenMPI 3.0.0 and noticed that osu_bibw gives nowhere near the bandwidth Iâd expect (this is on FDR IB). However, osu_bw is fine.

If I disable eager RDMA, then osu_bibw gives the expected numbers.Â Similarly, if I increase the number of eager RDMA buffers, it gives the expected results.

OpenMPI 1.10.7 gives consistent, reasonable numbers with default settings, but theyâre not as good as 3.0.0 (when tuned) for large buffers. The same option changes produce no different in the performance for 1.10.7.

I was wondering if anyone else has noticed anything similar, and if this is unexpected, if anyone has a suggestion on how to investigate further?

Thanks,
Ben

Hereâs are the numbers:

3.0.0, osu_bw, default settings

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bw

# OSU MPI Bandwidth Test v5.4.0
# SizeÂ Â Â Bandwidth (MB/s)
1 Â Â Â Â Â Â Â Â Â Â Â 1.13
2 Â Â Â Â Â Â Â Â Â Â Â 2.29
4 Â Â Â Â Â Â Â Â Â Â Â 4.63
8 Â Â Â Â Â Â Â Â Â Â Â 9.21
16 Â Â Â Â Â Â Â Â Â Â 18.18
32 Â Â Â Â Â Â Â Â Â Â 36.46
64 Â Â Â Â Â Â Â Â Â Â 69.95
128 Â Â Â Â Â Â Â Â Â 128.55
256 Â Â Â Â Â Â Â Â Â 250.74
512 Â Â Â Â Â Â Â Â Â 451.54
1024Â Â Â Â Â Â Â Â Â 829.44
2048 Â Â Â Â Â Â Â Â 1475.87
4096 Â Â Â Â Â Â Â Â 2119.99
8192 Â Â Â Â Â Â Â Â 3452.37
16384Â Â Â Â Â Â Â Â 2866.51
32768Â Â Â Â Â Â Â Â 4048.17
65536Â Â Â Â Â Â Â Â 5030.54
131072 Â Â Â Â Â Â Â 5573.81
262144 Â Â Â Â Â Â Â 5861.61
524288 Â Â Â Â Â Â Â 6015.15
1048576Â Â Â Â Â Â Â 6099.46
2097152 Â Â Â Â Â Â Â 989.82
4194304 Â Â Â Â Â Â Â 989.81

3.0.0, osu_bibw, default settings

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# SizeÂ Â Â Bandwidth (MB/s)
1 Â Â Â Â Â Â Â Â Â Â Â 0.00
2 Â Â Â Â Â Â Â Â Â Â Â 0.01
4 Â Â Â Â Â Â Â Â Â Â Â 0.01
8 Â Â Â Â Â Â Â Â Â Â Â 0.02
16Â Â Â Â Â Â Â Â Â Â Â 0.04
32Â Â Â Â Â Â Â Â Â Â Â 0.09
64Â Â Â Â Â Â Â Â Â Â Â 0.16
128 Â Â Â Â Â Â Â Â Â 135.30
256 Â Â Â Â Â Â Â Â Â 265.35
512 Â Â Â Â Â Â Â Â Â 499.92
1024Â Â Â Â Â Â Â Â Â 949.22
2048 Â Â Â Â Â Â Â Â 1440.27
4096 Â Â Â Â Â Â Â Â 1960.09
8192 Â Â Â Â Â Â Â Â 3166.97
16384 Â Â Â Â Â Â Â Â 127.62
32768 Â Â Â Â Â Â Â Â 165.12
65536 Â Â Â Â Â Â Â Â 312.80
131072 Â Â Â Â Â Â Â 1120.03
262144 Â Â Â Â Â Â Â 4724.01
524288 Â Â Â Â Â Â Â 4545.93
1048576Â Â Â Â Â Â Â 5186.51
2097152 Â Â Â Â Â Â Â 989.84
4194304 Â Â Â Â Â Â Â 989.88

3.0.0,Â osu_bibw, eager RDMA disabled

mpirun -mca btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# SizeÂ Â Â Bandwidth (MB/s)
1 Â Â Â Â Â Â Â Â Â Â Â 1.49
2 Â Â Â Â Â Â Â Â Â Â Â 2.97
4 Â Â Â Â Â Â Â Â Â Â Â 5.96
8Â Â Â Â Â Â Â Â Â Â Â 11.98
16 Â Â Â Â Â Â Â Â Â Â 23.95
32 Â Â Â Â Â Â Â Â Â Â 47.39
64 Â Â Â Â Â Â Â Â Â Â 93.57
128 Â Â Â Â Â Â Â Â Â 153.82
256 Â Â Â Â Â Â Â Â Â 304.69
512 Â Â Â Â Â Â Â Â Â 572.30
1024 Â Â Â Â Â Â Â Â 1003.52
2048 Â Â Â Â Â Â Â Â 1083.89
4096 Â Â Â Â Â Â Â Â 1879.32
8192 Â Â Â Â Â Â Â Â 2785.18
16384Â Â Â Â Â Â Â Â 3535.77
32768Â Â Â Â Â Â Â Â 5614.72
65536Â Â Â Â Â Â Â Â 8113.69
131072 Â Â Â Â Â Â Â 9666.74
262144Â Â Â Â Â Â Â 10738.97
524288Â Â Â Â Â Â Â 11247.02
1048576 Â Â Â Â Â Â 11416.50
2097152 Â Â Â Â Â Â Â 989.88
4194304 Â Â Â Â Â Â Â 989.88

3.0.0,Â osu_bibw, increased eager RDMA buffer count

mpirun -mca btl_openib_eager_rdma_num 32768 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# SizeÂ Â Â Bandwidth (MB/s)
1 Â Â Â Â Â Â Â Â Â Â Â 1.42
2 Â Â Â Â Â Â Â Â Â Â Â 2.84
4 Â Â Â Â Â Â Â Â Â Â Â 5.67
8Â Â Â Â Â Â Â Â Â Â Â 11.18
16 Â Â Â Â Â Â Â Â Â Â 22.46
32 Â Â Â Â Â Â Â Â Â Â 44.65
64 Â Â Â Â Â Â Â Â Â Â 83.10
128 Â Â Â Â Â Â Â Â Â 154.00
256 Â Â Â Â Â Â Â Â Â 291.63
512 Â Â Â Â Â Â Â Â Â 537.66
1024Â Â Â Â Â Â Â Â Â 942.35
2048 Â Â Â Â Â Â Â Â 1433.09
4096 Â Â Â Â Â Â Â Â 2356.40
8192 Â Â Â Â Â Â Â Â 1998.54
16384Â Â Â Â Â Â Â Â 3584.82
32768Â Â Â Â Â Â Â Â 5523.08
65536Â Â Â Â Â Â Â Â 7717.63
131072 Â Â Â Â Â Â Â 9419.50
262144Â Â Â Â Â Â Â 10564.77
524288Â Â Â Â Â Â Â 11104.71
1048576 Â Â Â Â Â Â 11130.75
2097152Â Â Â Â Â Â Â 7943.89
4194304Â Â Â Â Â Â Â 5270.00

1.10.7, osu_bibw, default settings

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# SizeÂ Â Â Bandwidth (MB/s)
1 Â Â Â Â Â Â Â Â Â Â Â 1.70
2 Â Â Â Â Â Â Â Â Â Â Â 3.45
4 Â Â Â Â Â Â Â Â Â Â Â 6.95
8Â Â Â Â Â Â Â Â Â Â Â 13.68
16 Â Â Â Â Â Â Â Â Â Â 27.41
32 Â Â Â Â Â Â Â Â Â Â 53.80
64Â Â Â Â Â Â Â Â Â Â 105.34
128 Â Â Â Â Â Â Â Â Â 164.40
256 Â Â Â Â Â Â Â Â Â 324.63
512 Â Â Â Â Â Â Â Â Â 623.95
1024 Â Â Â Â Â Â Â Â 1127.35
2048 Â Â Â Â Â Â Â Â 1784.58
4096 Â Â Â Â Â Â Â Â 3305.45
8192 Â Â Â Â Â Â Â Â 3697.55
16384Â Â Â Â Â Â Â Â 4935.75
32768Â Â Â Â Â Â Â Â 7186.28
65536Â Â Â Â Â Â Â Â 8996.94
131072 Â Â Â Â Â Â Â 9301.78
262144 Â Â Â Â Â Â Â 4691.36
524288 Â Â Â Â Â Â Â 7039.18
1048576Â Â Â Â Â Â Â 7213.33
2097152Â Â Â Â Â Â Â 9601.41
4194304Â Â Â Â Â Â Â 9281.31

Ben Menadue

2018-04-05 22:49:37 UTC

Permalink

Hi Nathan, Howard,

Thanks for the feedback. Yes, we do already have UCX compiled in to our OpenMPI installations, but itâs disabled by default on our system because some users were reporting problems with it previously. But Iâm not sure what the status of these are with OpenMPI 3.0, something for me to follow up with them.

Cheers,
Ben

Post by Nathan Hjelm
Honestly, this is a configuration issue with the openib btl. There is no reason to keep either eager RDMA nor is there a reason to pipeline RDMA. I haven't found an app where either of these "features" helps you with infiniband. You have the right idea with the parameter changes but Howard is correct, for Mellanox the future is UCX not verbs. I would try it and see if it works for you but if it doesn't I would set those two parameters in your /etc/openmpi-mca-params.conf and run like that.
-Nathan

Post by Ben Menadue
Hi,
3.0.0, osu_bw, no rdma for large messages

mpirun -mca btl_openib_min_rdma_pipeline_size 4194304 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bw -m 2097152:4194304

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
2097152 6133.22
4194304 6054.06
3.0.0, osu_bibw, eager rdma disabled, no rdma for large messages

mpirun -mca btl_openib_min_rdma_pipeline_size 4194304 -mca btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw -m 2097152:4194304

# OSU MPI Bi-Directional Bandwidth Test v5.4.0
# Size Bandwidth (MB/s)
2097152 11397.85
4194304 11389.64
This makes me think something odd is going on in the RDMA pipeline.
Cheers,
Ben

Hi,
Weâve just been running some OSU benchmarks with OpenMPI 3.0.0 and noticed that osu_bibw gives nowhere near the bandwidth Iâd expect (this is on FDR IB). However, osu_bw is fine.
If I disable eager RDMA, then osu_bibw gives the expected numbers. Similarly, if I increase the number of eager RDMA buffers, it gives the expected results.
OpenMPI 1.10.7 gives consistent, reasonable numbers with default settings, but theyâre not as good as 3.0.0 (when tuned) for large buffers. The same option changes produce no different in the performance for 1.10.7.
I was wondering if anyone else has noticed anything similar, and if this is unexpected, if anyone has a suggestion on how to investigate further?
Thanks,
Ben
3.0.0, osu_bw, default settings

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bw

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

mpirun -mca btl_openib_use_eager_rdma 0 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

mpirun -mca btl_openib_eager_rdma_num 32768 -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

mpirun -map-by ppr:1:node -np 2 -H r6,r7 ./osu_bibw

_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users <https://lists.open-mpi.org/mailman/listinfo/users>_______________________________________________

users mailing list
https://lists.open-mpi.org/mailman/listinfo/users