Discussion:
[OMPI users] More confusion about --map-by!
Mark Dixon
2017-02-23 12:06:24 UTC
Permalink
Hi,

I'm still trying to figure out how to express the core binding I want to
openmpi 2.x via the --map-by option. Can anyone help, please?

I bet I'm being dumb, but it's proving tricky to achieve the following
aims (most important first):

1) Maximise memory bandwidth usage (e.g. load balance ranks across
processor sockets)
2) Optimise for nearest-neighbour comms (in MPI_COMM_WORLD) (e.g. put
neighbouring ranks on the same socket)
3) Have an incantation that's simple to change based on number of ranks
and processes per rank I want.

Example:

Considering a 2 socket, 12 cores/socket box and a program with 2 threads
per rank...

... this is great if I fully-populate the node:

$ mpirun -np 12 -map-by slot:PE=2 --bind-to core --report-bindings ./prog
[somehost:101235] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket 0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
[somehost:101235] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
[somehost:101235] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
[somehost:101235] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket 0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
[somehost:101235] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket 0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
[somehost:101235] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket 0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
[somehost:101235] MCW rank 6 bound to socket 1[core 12[hwt 0]], socket 1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
[somehost:101235] MCW rank 7 bound to socket 1[core 14[hwt 0]], socket 1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
[somehost:101235] MCW rank 8 bound to socket 1[core 16[hwt 0]], socket 1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
[somehost:101235] MCW rank 9 bound to socket 1[core 18[hwt 0]], socket 1[core 19[hwt 0]]: [./././././././././././.][././././././B/B/./././.]
[somehost:101235] MCW rank 10 bound to socket 1[core 20[hwt 0]], socket 1[core 21[hwt 0]]: [./././././././././././.][././././././././B/B/./.]
[somehost:101235] MCW rank 11 bound to socket 1[core 22[hwt 0]], socket 1[core 23[hwt 0]]: [./././././././././././.][././././././././././B/B]


... but not if I don't [fails aim (1)]:

$ mpirun -np 6 -map-by slot:PE=2 --bind-to core --report-bindings ./prog
[somehost:102035] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket 0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
[somehost:102035] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
[somehost:102035] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
[somehost:102035] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket 0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
[somehost:102035] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket 0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
[somehost:102035] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket 0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]


... whereas if I map by socket instead of slot, I achieve aim (1) but
fail on aim (2):

$ mpirun -np 6 -map-by socket:PE=2 --bind-to core --report-bindings ./prog
[somehost:105601] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket 0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
[somehost:105601] MCW rank 1 bound to socket 1[core 12[hwt 0]], socket 1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
[somehost:105601] MCW rank 2 bound to socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
[somehost:105601] MCW rank 3 bound to socket 1[core 14[hwt 0]], socket 1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
[somehost:105601] MCW rank 4 bound to socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
[somehost:105601] MCW rank 5 bound to socket 1[core 16[hwt 0]], socket 1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]


Any ideas, please?

Thanks,

Mark
g***@rist.or.jp
2017-02-23 14:18:12 UTC
Permalink
Mark,

what about
mpirun -np 6 -map-by slot:PE=4 --bind-to core --report-bindings ./prog

it is a fit for 1) and 2) but not 3)

if you use OpenMP and want 2 threads per task, then you can
export OMP_NUM_THREADS=2
not to use 4 threads by default (with most OpenMP runtimes)

Cheers,

Gilles
----- Original Message -----
Post by Mark Dixon
Hi,
I'm still trying to figure out how to express the core binding I want to
openmpi 2.x via the --map-by option. Can anyone help, please?
I bet I'm being dumb, but it's proving tricky to achieve the following
1) Maximise memory bandwidth usage (e.g. load balance ranks across
processor sockets)
2) Optimise for nearest-neighbour comms (in MPI_COMM_WORLD) (e.g. put
neighbouring ranks on the same socket)
3) Have an incantation that's simple to change based on number of ranks
and processes per rank I want.
Considering a 2 socket, 12 cores/socket box and a program with 2 threads
per rank...
$ mpirun -np 12 -map-by slot:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:101235] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket
0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket
0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket
0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 6 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 7 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 8 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 9 bound to socket 1[core 18[hwt 0]], socket
1[core 19[hwt 0]]: [./././././././././././.][././././././B/B/./././.]
Post by Mark Dixon
[somehost:101235] MCW rank 10 bound to socket 1[core 20[hwt 0]],
socket 1[core 21[hwt 0]]: [./././././././././././.][././././././././B/B/.
/.]
Post by Mark Dixon
[somehost:101235] MCW rank 11 bound to socket 1[core 22[hwt 0]],
socket 1[core 23[hwt 0]]: [./././././././././././.][././././././././././
B/B]
Post by Mark Dixon
$ mpirun -np 6 -map-by slot:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:102035] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket
0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket
0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket
0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
Post by Mark Dixon
... whereas if I map by socket instead of slot, I achieve aim (1) but
$ mpirun -np 6 -map-by socket:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:105601] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 1 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 2 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 3 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 4 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 5 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
Post by Mark Dixon
Any ideas, please?
Thanks,
Mark
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
r***@open-mpi.org
2017-02-23 14:38:26 UTC
Permalink
From the mpirun man page:

******************
Open MPI employs a three-phase procedure for assigning process locations and ranks:
mapping
Assigns a default location to each process
ranking
Assigns an MPI_COMM_WORLD rank value to each process
binding
Constrains each process to run on specific processors
The mapping step is used to assign a default location to each process based on the mapper being employed. Mapping by slot, node, and sequentially results in the assignment of the processes to the node level. In contrast, mapping by object, allows the mapper to assign the process to an actual object on each node.

Note: the location assigned to the process is independent of where it will be bound - the assignment is used solely as input to the binding algorithm.

The mapping of process processes to nodes can be defined not just with general policies but also, if necessary, using arbitrary mappings that cannot be described by a simple policy. One can use the "sequential mapper," which reads the hostfile line by line, assigning processes to nodes in whatever order the hostfile specifies. Use the -mca rmaps seq option. For example, using the same hostfile as before:

mpirun -hostfile myhostfile -mca rmaps seq ./a.out

will launch three processes, one on each of nodes aa, bb, and cc, respectively. The slot counts don’t matter; one process is launched per line on whatever node is listed on the line.

Another way to specify arbitrary mappings is with a rankfile, which gives you detailed control over process binding as well. Rankfiles are discussed below.

The second phase focuses on the ranking of the process within the job’s MPI_COMM_WORLD. Open MPI separates this from the mapping procedure to allow more flexibility in the relative placement of MPI processes.

The binding phase actually binds each process to a given set of processors. This can improve performance if the operating system is placing processes suboptimally. For example, it might oversubscribe some multi-core processor sockets, leaving other sockets idle; this can lead processes to contend unnecessarily for common resources. Or, it might spread processes out too widely; this can be suboptimal if application performance is sensitive to interprocess communication costs. Binding can also keep the operating system from migrating processes excessively, regardless of how optimally those processes were placed to begin with.
********************

So what you probably want is: --map-by socket:pe=N --rank-by core

Remember, the pe=N modifier automatically forces binding at the cpu level. The rank-by directive defaults to rank-by socket when you map-by socket, hence you need to specify that you want it to map by core instead. Here is the result of doing that on my box:

$ mpirun --map-by socket:pe=2 --rank-by core --report-bindings -n 8 hostname
[rhc001:154283] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]]: [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]]: [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 2 bound to socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]]: [../../../../BB/BB/../../../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 3 bound to socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: [../../../../../../BB/BB/../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 4 bound to socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]]: [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
[rhc001:154283] MCW rank 5 bound to socket 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: [../../../../../../../../../../../..][../../BB/BB/../../../../../../../..]
[rhc001:154283] MCW rank 6 bound to socket 1[core 16[hwt 0-1]], socket 1[core 17[hwt 0-1]]: [../../../../../../../../../../../..][../../../../BB/BB/../../../../../..]
[rhc001:154283] MCW rank 7 bound to socket 1[core 18[hwt 0-1]], socket 1[core 19[hwt 0-1]]: [../../../../../../../../../../../..][../../../../../../BB/BB/../../../..]


HTH
Ralph
Post by g***@rist.or.jp
Mark,
what about
mpirun -np 6 -map-by slot:PE=4 --bind-to core --report-bindings ./prog
it is a fit for 1) and 2) but not 3)
if you use OpenMP and want 2 threads per task, then you can
export OMP_NUM_THREADS=2
not to use 4 threads by default (with most OpenMP runtimes)
Cheers,
Gilles
----- Original Message -----
Post by Mark Dixon
Hi,
I'm still trying to figure out how to express the core binding I want
to
Post by Mark Dixon
openmpi 2.x via the --map-by option. Can anyone help, please?
I bet I'm being dumb, but it's proving tricky to achieve the following
1) Maximise memory bandwidth usage (e.g. load balance ranks across
processor sockets)
2) Optimise for nearest-neighbour comms (in MPI_COMM_WORLD) (e.g. put
neighbouring ranks on the same socket)
3) Have an incantation that's simple to change based on number of
ranks
Post by Mark Dixon
and processes per rank I want.
Considering a 2 socket, 12 cores/socket box and a program with 2
threads
Post by Mark Dixon
per rank...
$ mpirun -np 12 -map-by slot:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:101235] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket
0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket
0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket
0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 6 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 7 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 8 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 9 bound to socket 1[core 18[hwt 0]], socket
1[core 19[hwt 0]]: [./././././././././././.][././././././B/B/./././.]
Post by Mark Dixon
[somehost:101235] MCW rank 10 bound to socket 1[core 20[hwt 0]],
socket 1[core 21[hwt 0]]: [./././././././././././.][././././././././B/B/.
/.]
Post by Mark Dixon
[somehost:101235] MCW rank 11 bound to socket 1[core 22[hwt 0]],
socket 1[core 23[hwt 0]]: [./././././././././././.][././././././././././
B/B]
Post by Mark Dixon
$ mpirun -np 6 -map-by slot:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:102035] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket
0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket
0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket
0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
Post by Mark Dixon
... whereas if I map by socket instead of slot, I achieve aim (1) but
$ mpirun -np 6 -map-by socket:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:105601] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 1 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 2 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 3 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 4 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 5 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
Post by Mark Dixon
Any ideas, please?
Thanks,
Mark
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
r***@open-mpi.org
2017-02-23 15:17:11 UTC
Permalink
Just as a fun follow-up: if you wanted to load-balance across nodes as well as within nodes, then you would add the “span” modifier to map-by:

$ mpirun --map-by socket:span,pe=2 --rank-by core --report-bindings -n 8 hostname
[rhc001:162391] SETTING BINDING TO CORE
[rhc001:162391] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]]: [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc001:162391] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]]: [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
[rhc001:162391] MCW rank 2 bound to socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]]: [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
[rhc001:162391] MCW rank 3 bound to socket 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: [../../../../../../../../../../../..][../../BB/BB/../../../../../../../..]


[rhc002.cluster:150295] MCW rank 4 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]]: [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc002.cluster:150295] MCW rank 5 bound to socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]]: [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
[rhc002.cluster:150295] MCW rank 6 bound to socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]]: [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
[rhc002.cluster:150295] MCW rank 7 bound to socket 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: [../../../../../../../../../../../..][../../BB/BB/../../../../../../../..]

“span” causes ORTE to treat all the sockets etc. as being on a single giant node.

HTH
Ralph
Post by r***@open-mpi.org
******************
mapping
Assigns a default location to each process
ranking
Assigns an MPI_COMM_WORLD rank value to each process
binding
Constrains each process to run on specific processors
The mapping step is used to assign a default location to each process based on the mapper being employed. Mapping by slot, node, and sequentially results in the assignment of the processes to the node level. In contrast, mapping by object, allows the mapper to assign the process to an actual object on each node.
Note: the location assigned to the process is independent of where it will be bound - the assignment is used solely as input to the binding algorithm.
mpirun -hostfile myhostfile -mca rmaps seq ./a.out
will launch three processes, one on each of nodes aa, bb, and cc, respectively. The slot counts don’t matter; one process is launched per line on whatever node is listed on the line.
Another way to specify arbitrary mappings is with a rankfile, which gives you detailed control over process binding as well. Rankfiles are discussed below.
The second phase focuses on the ranking of the process within the job’s MPI_COMM_WORLD. Open MPI separates this from the mapping procedure to allow more flexibility in the relative placement of MPI processes.
The binding phase actually binds each process to a given set of processors. This can improve performance if the operating system is placing processes suboptimally. For example, it might oversubscribe some multi-core processor sockets, leaving other sockets idle; this can lead processes to contend unnecessarily for common resources. Or, it might spread processes out too widely; this can be suboptimal if application performance is sensitive to interprocess communication costs. Binding can also keep the operating system from migrating processes excessively, regardless of how optimally those processes were placed to begin with.
********************
So what you probably want is: --map-by socket:pe=N --rank-by core
$ mpirun --map-by socket:pe=2 --rank-by core --report-bindings -n 8 hostname
[rhc001:154283] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]]: [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]]: [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 2 bound to socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]]: [../../../../BB/BB/../../../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 3 bound to socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: [../../../../../../BB/BB/../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 4 bound to socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]]: [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
[rhc001:154283] MCW rank 5 bound to socket 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: [../../../../../../../../../../../..][../../BB/BB/../../../../../../../..]
[rhc001:154283] MCW rank 6 bound to socket 1[core 16[hwt 0-1]], socket 1[core 17[hwt 0-1]]: [../../../../../../../../../../../..][../../../../BB/BB/../../../../../..]
[rhc001:154283] MCW rank 7 bound to socket 1[core 18[hwt 0-1]], socket 1[core 19[hwt 0-1]]: [../../../../../../../../../../../..][../../../../../../BB/BB/../../../..]
HTH
Ralph
Post by g***@rist.or.jp
Mark,
what about
mpirun -np 6 -map-by slot:PE=4 --bind-to core --report-bindings ./prog
it is a fit for 1) and 2) but not 3)
if you use OpenMP and want 2 threads per task, then you can
export OMP_NUM_THREADS=2
not to use 4 threads by default (with most OpenMP runtimes)
Cheers,
Gilles
----- Original Message -----
Post by Mark Dixon
Hi,
I'm still trying to figure out how to express the core binding I want
to
Post by Mark Dixon
openmpi 2.x via the --map-by option. Can anyone help, please?
I bet I'm being dumb, but it's proving tricky to achieve the following
1) Maximise memory bandwidth usage (e.g. load balance ranks across
processor sockets)
2) Optimise for nearest-neighbour comms (in MPI_COMM_WORLD) (e.g. put
neighbouring ranks on the same socket)
3) Have an incantation that's simple to change based on number of
ranks
Post by Mark Dixon
and processes per rank I want.
Considering a 2 socket, 12 cores/socket box and a program with 2
threads
Post by Mark Dixon
per rank...
$ mpirun -np 12 -map-by slot:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:101235] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket
0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket
0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket
0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 6 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 7 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 8 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 9 bound to socket 1[core 18[hwt 0]], socket
1[core 19[hwt 0]]: [./././././././././././.][././././././B/B/./././.]
Post by Mark Dixon
[somehost:101235] MCW rank 10 bound to socket 1[core 20[hwt 0]],
socket 1[core 21[hwt 0]]: [./././././././././././.][././././././././B/B/.
/.]
Post by Mark Dixon
[somehost:101235] MCW rank 11 bound to socket 1[core 22[hwt 0]],
socket 1[core 23[hwt 0]]: [./././././././././././.][././././././././././
B/B]
Post by Mark Dixon
$ mpirun -np 6 -map-by slot:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:102035] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket
0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket
0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket
0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
Post by Mark Dixon
... whereas if I map by socket instead of slot, I achieve aim (1) but
$ mpirun -np 6 -map-by socket:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:105601] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 1 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 2 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 3 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 4 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 5 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
Post by Mark Dixon
Any ideas, please?
Thanks,
Mark
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Mark Dixon
2017-02-23 15:53:31 UTC
Permalink
Hi Ralph, Gilles, Josh,

Adding "--rank-by core" into the mix straightens it all out - brilliant!
Thanks!

I had thought --rank-by was more related to the relationships between
multiple nodes - doh. I think I need to think more about what span does
(along with everything else)...

Many thanks for taking pity on me (again),

Cheers,

Mark
Post by r***@open-mpi.org
$ mpirun --map-by socket:span,pe=2 --rank-by core --report-bindings -n 8 hostname
[rhc001:162391] SETTING BINDING TO CORE
[rhc001:162391] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]]: [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc001:162391] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]]: [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
[rhc001:162391] MCW rank 2 bound to socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]]: [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
[rhc001:162391] MCW rank 3 bound to socket 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: [../../../../../../../../../../../..][../../BB/BB/../../../../../../../..]
[rhc002.cluster:150295] MCW rank 4 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]]: [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc002.cluster:150295] MCW rank 5 bound to socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]]: [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
[rhc002.cluster:150295] MCW rank 6 bound to socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]]: [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
[rhc002.cluster:150295] MCW rank 7 bound to socket 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: [../../../../../../../../../../../..][../../BB/BB/../../../../../../../..]
“span” causes ORTE to treat all the sockets etc. as being on a single giant node.
HTH
Ralph
Post by r***@open-mpi.org
******************
mapping
Assigns a default location to each process
ranking
Assigns an MPI_COMM_WORLD rank value to each process
binding
Constrains each process to run on specific processors
The mapping step is used to assign a default location to each process based on the mapper being employed. Mapping by slot, node, and sequentially results in the assignment of the processes to the node level. In contrast, mapping by object, allows the mapper to assign the process to an actual object on each node.
Note: the location assigned to the process is independent of where it will be bound - the assignment is used solely as input to the binding algorithm.
mpirun -hostfile myhostfile -mca rmaps seq ./a.out
will launch three processes, one on each of nodes aa, bb, and cc, respectively. The slot counts don’t matter; one process is launched per line on whatever node is listed on the line.
Another way to specify arbitrary mappings is with a rankfile, which gives you detailed control over process binding as well. Rankfiles are discussed below.
The second phase focuses on the ranking of the process within the job’s MPI_COMM_WORLD. Open MPI separates this from the mapping procedure to allow more flexibility in the relative placement of MPI processes.
The binding phase actually binds each process to a given set of processors. This can improve performance if the operating system is placing processes suboptimally. For example, it might oversubscribe some multi-core processor sockets, leaving other sockets idle; this can lead processes to contend unnecessarily for common resources. Or, it might spread processes out too widely; this can be suboptimal if application performance is sensitive to interprocess communication costs. Binding can also keep the operating system from migrating processes excessively, regardless of how optimally those processes were placed to begin with.
********************
So what you probably want is: --map-by socket:pe=N --rank-by core
$ mpirun --map-by socket:pe=2 --rank-by core --report-bindings -n 8 hostname
[rhc001:154283] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 1[hwt 0-1]]: [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt 0-1]]: [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 2 bound to socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]]: [../../../../BB/BB/../../../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 3 bound to socket 0[core 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: [../../../../../../BB/BB/../../../..][../../../../../../../../../../../..]
[rhc001:154283] MCW rank 4 bound to socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]]: [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
[rhc001:154283] MCW rank 5 bound to socket 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: [../../../../../../../../../../../..][../../BB/BB/../../../../../../../..]
[rhc001:154283] MCW rank 6 bound to socket 1[core 16[hwt 0-1]], socket 1[core 17[hwt 0-1]]: [../../../../../../../../../../../..][../../../../BB/BB/../../../../../..]
[rhc001:154283] MCW rank 7 bound to socket 1[core 18[hwt 0-1]], socket 1[core 19[hwt 0-1]]: [../../../../../../../../../../../..][../../../../../../BB/BB/../../../..]
HTH
Ralph
Post by g***@rist.or.jp
Mark,
what about
mpirun -np 6 -map-by slot:PE=4 --bind-to core --report-bindings ./prog
it is a fit for 1) and 2) but not 3)
if you use OpenMP and want 2 threads per task, then you can
export OMP_NUM_THREADS=2
not to use 4 threads by default (with most OpenMP runtimes)
Cheers,
Gilles
----- Original Message -----
Post by Mark Dixon
Hi,
I'm still trying to figure out how to express the core binding I want
to
Post by Mark Dixon
openmpi 2.x via the --map-by option. Can anyone help, please?
I bet I'm being dumb, but it's proving tricky to achieve the following
1) Maximise memory bandwidth usage (e.g. load balance ranks across
processor sockets)
2) Optimise for nearest-neighbour comms (in MPI_COMM_WORLD) (e.g. put
neighbouring ranks on the same socket)
3) Have an incantation that's simple to change based on number of
ranks
Post by Mark Dixon
and processes per rank I want.
Considering a 2 socket, 12 cores/socket box and a program with 2
threads
Post by Mark Dixon
per rank...
$ mpirun -np 12 -map-by slot:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:101235] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket
0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket
0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket
0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 6 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 7 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 8 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 9 bound to socket 1[core 18[hwt 0]], socket
1[core 19[hwt 0]]: [./././././././././././.][././././././B/B/./././.]
Post by Mark Dixon
[somehost:101235] MCW rank 10 bound to socket 1[core 20[hwt 0]],
socket 1[core 21[hwt 0]]: [./././././././././././.][././././././././B/B/.
/.]
Post by Mark Dixon
[somehost:101235] MCW rank 11 bound to socket 1[core 22[hwt 0]],
socket 1[core 23[hwt 0]]: [./././././././././././.][././././././././././
B/B]
Post by Mark Dixon
$ mpirun -np 6 -map-by slot:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:102035] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket
0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket
0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket
0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
Post by Mark Dixon
... whereas if I map by socket instead of slot, I achieve aim (1) but
$ mpirun -np 6 -map-by socket:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:105601] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 1 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 2 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 3 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 4 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 5 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
Post by Mark Dixon
Any ideas, please?
Thanks,
Mark
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
--
-------------------------------------------------------------------
Mark Dixon Email : ***@leeds.ac.uk
Advanced Research Computing (ARC) Tel (int): 35429
IT Services building Tel (ext): +44(0)113 343 5429
University of Leeds, LS2 9JT, UK
-------------------------------------------------------------------
Josh Hursey
2017-02-23 14:50:58 UTC
Permalink
One thing that might help is the *--rank-by *argument that allows you to
specify how ranks are assigned separate from mapping/binding (by default we
follow the mapping pattern).

For example - adding *--rank-by* to your last example:
$ mpirun -np 6 -map-by socket:PE=2 --bind-to core *--rank-by core*
--report-bindings ./prog
[somehost:105601] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
[somehost:105601] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
[somehost:105601] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
[somehost:105601] MCW rank 3 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
[somehost:105601] MCW rank 4 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
[somehost:105601] MCW rank 5 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]

Is that what you are looking for?
Post by g***@rist.or.jp
Mark,
what about
mpirun -np 6 -map-by slot:PE=4 --bind-to core --report-bindings ./prog
it is a fit for 1) and 2) but not 3)
if you use OpenMP and want 2 threads per task, then you can
export OMP_NUM_THREADS=2
not to use 4 threads by default (with most OpenMP runtimes)
Cheers,
Gilles
----- Original Message -----
Post by Mark Dixon
Hi,
I'm still trying to figure out how to express the core binding I want
to
Post by Mark Dixon
openmpi 2.x via the --map-by option. Can anyone help, please?
I bet I'm being dumb, but it's proving tricky to achieve the following
1) Maximise memory bandwidth usage (e.g. load balance ranks across
processor sockets)
2) Optimise for nearest-neighbour comms (in MPI_COMM_WORLD) (e.g. put
neighbouring ranks on the same socket)
3) Have an incantation that's simple to change based on number of
ranks
Post by Mark Dixon
and processes per rank I want.
Considering a 2 socket, 12 cores/socket box and a program with 2
threads
Post by Mark Dixon
per rank...
$ mpirun -np 12 -map-by slot:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:101235] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket
0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket
0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket
0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 6 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 7 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 8 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
Post by Mark Dixon
[somehost:101235] MCW rank 9 bound to socket 1[core 18[hwt 0]], socket
1[core 19[hwt 0]]: [./././././././././././.][././././././B/B/./././.]
Post by Mark Dixon
[somehost:101235] MCW rank 10 bound to socket 1[core 20[hwt 0]],
socket 1[core 21[hwt 0]]: [./././././././././././.][././././././././B/B/.
/.]
Post by Mark Dixon
[somehost:101235] MCW rank 11 bound to socket 1[core 22[hwt 0]],
socket 1[core 23[hwt 0]]: [./././././././././././.][././././././././././
B/B]
Post by Mark Dixon
$ mpirun -np 6 -map-by slot:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:102035] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 3 bound to socket 0[core 6[hwt 0]], socket
0[core 7[hwt 0]]: [././././././B/B/./././.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 4 bound to socket 0[core 8[hwt 0]], socket
0[core 9[hwt 0]]: [././././././././B/B/./.][./././././././././././.]
Post by Mark Dixon
[somehost:102035] MCW rank 5 bound to socket 0[core 10[hwt 0]], socket
0[core 11[hwt 0]]: [././././././././././B/B][./././././././././././.]
Post by Mark Dixon
... whereas if I map by socket instead of slot, I achieve aim (1) but
$ mpirun -np 6 -map-by socket:PE=2 --bind-to core --report-bindings ./
prog
Post by Mark Dixon
[somehost:105601] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
0[core 1[hwt 0]]: [B/B/./././././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 1 bound to socket 1[core 12[hwt 0]], socket
1[core 13[hwt 0]]: [./././././././././././.][B/B/./././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 2 bound to socket 0[core 2[hwt 0]], socket
0[core 3[hwt 0]]: [././B/B/./././././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 3 bound to socket 1[core 14[hwt 0]], socket
1[core 15[hwt 0]]: [./././././././././././.][././B/B/./././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 4 bound to socket 0[core 4[hwt 0]], socket
0[core 5[hwt 0]]: [././././B/B/./././././.][./././././././././././.]
Post by Mark Dixon
[somehost:105601] MCW rank 5 bound to socket 1[core 16[hwt 0]], socket
1[core 17[hwt 0]]: [./././././././././././.][././././B/B/./././././.]
Post by Mark Dixon
Any ideas, please?
Thanks,
Mark
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
--
Josh Hursey
IBM Spectrum MPI Developer
Loading...