Wirawan Purwanto
2016-10-03 21:22:33 UTC
Hi,
I have been trying to understand how to correctly launch hybrid
MPI/OpenMP (i.e. multi-threaded MPI jobs) with mpirun. I am quite
puzzled as to what is the correct command-line options to use. The
description on mpirun man page is very confusing and I could not get
what I wanted.
A background: The cluster is using SGE, and I am using OpenMPI 1.10.2
compiled with & for gcc 4.9.3. The MPI library was configured with SGE
support. The compute nodes have 32 cores, which are basically 2
sockets of Xeon E5-2698 v3 (16-core Haswell).
A colleague told me the following:
$ export OMP_NUM_THREADS=2
$ mpirun -np 16 -map-by node:PE=2 ./EXECUTABLE
I could see the executable using 200% of CPU per process--that's good.
There is one catch in the general case. "-map-by node" will assign the
MPI processes in a round-robin fashion (so MPI rank 0 gets node 0, mpi
rank 1 gets node 1, and so on until all nodes are given 1 process,
then it will go back to node 0,1, ...).
Instead of the scenario above, I was trying to get the MPI processes
side-by-side (more like "fill_up" policy in SGE scheduler), i.e. fill
node 0 first, then fill node 1, and so on. How do I do this properly?
I tried a few attempts that fail:
$ export OMP_NUM_THREADS=2
$ mpirun -np 16 -map-by core:PE=2 ./EXECUTABLE
or
$ export OMP_NUM_THREADS=2
$ mpirun -np 16 -map-by socket:PE=2 ./EXECUTABLE
Both failed with an error mesage:
--------------------------------------------------------------------------
A request for multiple cpus-per-proc was given, but a directive
was also give to map to an object level that cannot support that
directive.
Please specify a mapping level that has more than one cpu, or
else let us define a default mapping that will allow multiple
cpus-per-proc.
--------------------------------------------------------------------------
Another attempt was:
$ export OMP_NUM_THREADS=2
$ mpirun -np 16 -map-by socket:PE=2 -bind-to socket ./EXECUTABLE
Here's the error message:
--------------------------------------------------------------------------
A request for multiple cpus-per-proc was given, but a conflicting binding
policy was specified:
#cpus-per-proc: 2
type of cpus: cores as cpus
binding policy given: SOCKET
The correct binding policy for the given type of cpu is:
correct binding policy: bind-to core
This is the binding policy we would apply by default for this
situation, so no binding need be specified. Please correct the
situation and try again.
--------------------------------------------------------------------------
Clearly I am not understanding how this map-by works. Could somebody
help me? There was a wiki article partially written:
https://github.com/open-mpi/ompi/wiki/ProcessPlacement
but unfortunately it is also not clear to me.
I have been trying to understand how to correctly launch hybrid
MPI/OpenMP (i.e. multi-threaded MPI jobs) with mpirun. I am quite
puzzled as to what is the correct command-line options to use. The
description on mpirun man page is very confusing and I could not get
what I wanted.
A background: The cluster is using SGE, and I am using OpenMPI 1.10.2
compiled with & for gcc 4.9.3. The MPI library was configured with SGE
support. The compute nodes have 32 cores, which are basically 2
sockets of Xeon E5-2698 v3 (16-core Haswell).
A colleague told me the following:
$ export OMP_NUM_THREADS=2
$ mpirun -np 16 -map-by node:PE=2 ./EXECUTABLE
I could see the executable using 200% of CPU per process--that's good.
There is one catch in the general case. "-map-by node" will assign the
MPI processes in a round-robin fashion (so MPI rank 0 gets node 0, mpi
rank 1 gets node 1, and so on until all nodes are given 1 process,
then it will go back to node 0,1, ...).
Instead of the scenario above, I was trying to get the MPI processes
side-by-side (more like "fill_up" policy in SGE scheduler), i.e. fill
node 0 first, then fill node 1, and so on. How do I do this properly?
I tried a few attempts that fail:
$ export OMP_NUM_THREADS=2
$ mpirun -np 16 -map-by core:PE=2 ./EXECUTABLE
or
$ export OMP_NUM_THREADS=2
$ mpirun -np 16 -map-by socket:PE=2 ./EXECUTABLE
Both failed with an error mesage:
--------------------------------------------------------------------------
A request for multiple cpus-per-proc was given, but a directive
was also give to map to an object level that cannot support that
directive.
Please specify a mapping level that has more than one cpu, or
else let us define a default mapping that will allow multiple
cpus-per-proc.
--------------------------------------------------------------------------
Another attempt was:
$ export OMP_NUM_THREADS=2
$ mpirun -np 16 -map-by socket:PE=2 -bind-to socket ./EXECUTABLE
Here's the error message:
--------------------------------------------------------------------------
A request for multiple cpus-per-proc was given, but a conflicting binding
policy was specified:
#cpus-per-proc: 2
type of cpus: cores as cpus
binding policy given: SOCKET
The correct binding policy for the given type of cpu is:
correct binding policy: bind-to core
This is the binding policy we would apply by default for this
situation, so no binding need be specified. Please correct the
situation and try again.
--------------------------------------------------------------------------
Clearly I am not understanding how this map-by works. Could somebody
help me? There was a wiki article partially written:
https://github.com/open-mpi/ompi/wiki/ProcessPlacement
but unfortunately it is also not clear to me.
--
Wirawan Purwanto
Computational Scientist, HPC Group
Information Technology Services
Old Dominion University
Norfolk, VA 23529
Wirawan Purwanto
Computational Scientist, HPC Group
Information Technology Services
Old Dominion University
Norfolk, VA 23529