Joseph Schuchart
2018-05-23 10:45:12 UTC
All,
We are observing some strange/interesting performance issues in
accessing memory that has been allocated through MPI_Win_allocate. I am
attaching our test case, which allocates memory for 100M integer values
on each process both through malloc and MPI_Win_allocate and writes to
the local ranges sequentially.
On different systems (incl. SuperMUC and a Bull Cluster), we see that
accessing the memory allocated through MPI is significantly slower than
accessing the malloc'ed memory if multiple processes run on a single
node, increasing the effect with increasing number of processes per
node. As an example, running 24 processes per node with the example
attached we see the operations on the malloc'ed memory to take ~0.4s
while the MPI allocated memory takes up to 10s.
After some experiments, I think there are two factors involved:
1) Initialization: it appears that the first iteration is significantly
slower than any subsequent accesses (1.1s vs 0.4s with 12 processes on a
single socket). Excluding the first iteration from the timing or
memsetting the range leads to comparable performance. I assume that this
is due to page faults that stem from first accessing the mmap'ed memory
that backs the shared memory used in the window. The effect of
presetting the malloc'ed memory seems smaller (0.4s vs 0.6s).
2) NUMA effects: Given proper initialization, running on two sockets
still leads to fluctuating performance degradation under the MPI window
memory, which ranges up to 20x (in extreme cases). The performance of
accessing the malloc'ed memory is rather stable. The difference seems to
get smaller (but does not disappear) with increasing number of
repetitions. I am not sure what causes these effects as each process
should first-touch their local memory.
Are these known issues? Does anyone have any thoughts on my analysis?
It is problematic for us that replacing local memory allocation with MPI
memory allocation leads to performance degradation as we rely on this
mechanism in our distributed data structures. While we can ensure proper
initialization of the memory to mitigate 1) for performance
measurements, I don't see a way to control the NUMA effects. If there is
one I'd be happy about any hints :)
I should note that we also tested MPICH-based implementations, which
showed similar effects (as they also mmap their window memory). Not
surprisingly, using MPI_Alloc_mem and attaching that memory to a dynamic
window does not cause these effects while using shared memory windows
does. I ran my experiments using Open MPI 3.1.0 with the following
command lines:
- 12 cores / 1 socket:
mpirun -n 12 --bind-to socket --map-by ppr:12:socket
- 24 cores / 2 sockets:
mpirun -n 24 --bind-to socket
and verified the binding using --report-bindings.
Any help or comment would be much appreciated.
Cheers
Joseph
We are observing some strange/interesting performance issues in
accessing memory that has been allocated through MPI_Win_allocate. I am
attaching our test case, which allocates memory for 100M integer values
on each process both through malloc and MPI_Win_allocate and writes to
the local ranges sequentially.
On different systems (incl. SuperMUC and a Bull Cluster), we see that
accessing the memory allocated through MPI is significantly slower than
accessing the malloc'ed memory if multiple processes run on a single
node, increasing the effect with increasing number of processes per
node. As an example, running 24 processes per node with the example
attached we see the operations on the malloc'ed memory to take ~0.4s
while the MPI allocated memory takes up to 10s.
After some experiments, I think there are two factors involved:
1) Initialization: it appears that the first iteration is significantly
slower than any subsequent accesses (1.1s vs 0.4s with 12 processes on a
single socket). Excluding the first iteration from the timing or
memsetting the range leads to comparable performance. I assume that this
is due to page faults that stem from first accessing the mmap'ed memory
that backs the shared memory used in the window. The effect of
presetting the malloc'ed memory seems smaller (0.4s vs 0.6s).
2) NUMA effects: Given proper initialization, running on two sockets
still leads to fluctuating performance degradation under the MPI window
memory, which ranges up to 20x (in extreme cases). The performance of
accessing the malloc'ed memory is rather stable. The difference seems to
get smaller (but does not disappear) with increasing number of
repetitions. I am not sure what causes these effects as each process
should first-touch their local memory.
Are these known issues? Does anyone have any thoughts on my analysis?
It is problematic for us that replacing local memory allocation with MPI
memory allocation leads to performance degradation as we rely on this
mechanism in our distributed data structures. While we can ensure proper
initialization of the memory to mitigate 1) for performance
measurements, I don't see a way to control the NUMA effects. If there is
one I'd be happy about any hints :)
I should note that we also tested MPICH-based implementations, which
showed similar effects (as they also mmap their window memory). Not
surprisingly, using MPI_Alloc_mem and attaching that memory to a dynamic
window does not cause these effects while using shared memory windows
does. I ran my experiments using Open MPI 3.1.0 with the following
command lines:
- 12 cores / 1 socket:
mpirun -n 12 --bind-to socket --map-by ppr:12:socket
- 24 cores / 2 sockets:
mpirun -n 24 --bind-to socket
and verified the binding using --report-bindings.
Any help or comment would be much appreciated.
Cheers
Joseph
--
Dipl.-Inf. Joseph Schuchart
High Performance Computing Center Stuttgart (HLRS)
Nobelstr. 19
D-70569 Stuttgart
Tel.: +49(0)711-68565890
Fax: +49(0)711-6856832
E-Mail: ***@hlrs.de
Dipl.-Inf. Joseph Schuchart
High Performance Computing Center Stuttgart (HLRS)
Nobelstr. 19
D-70569 Stuttgart
Tel.: +49(0)711-68565890
Fax: +49(0)711-6856832
E-Mail: ***@hlrs.de