[OMPI users] Questions about integration with resource distribution systems

[Kulshrestha, Vipul]

I have several questions about integration of openmpi with resource queuing systems.

1.
I understand that openmpi supports integration with various resource distribution systems such as SGE, LSF, torque etc.

I need to build an openmpi application that can interact with variety of different resource distribution systems, since different customers have different systems. Based on my research, it seems that I need to build a different openmpi installation to work, e.g. create an installation of opempi with grid and create a different installation of openmpi with LSF. Is there a way to build a generic installation of openmpi that can be used with more than 1 distribution system by using some generic mechanism?

2.
For integration with LSF/grid, how would I specify the memory (RAM) requirement (or some other parameter) to bsub/qsub, when launching mpirun command? Will something like below work to ensure that each of the 8 copies of a.out have 40 GB memory reserved for them by grid engine?

qsub -pe orte 8 -b y -V -l m_mem_free=40G -cwd mpirun -np 8 a.out

3.
Some of our customers use custom distribution engine (some non-industry-standard distribution engine). How can I integrate my openmpi application with such system? I would think that it should be possible to do that if openmpi launched/managed interaction with the distribution engine using some kind of generic mechanism (say, use a configurable command to launch, monitor, kill a job and then allow specification of a plugin define these operations with commands specific to the distribution engine being in use). Does such integration exist in openmpi?


Thanks,
Vipul

[r***@open-mpi.org]

Post by Kulshrestha, Vipul
I have several questions about integration of openmpi with resource queuing systems.
1.
I understand that openmpi supports integration with various resource distribution systems such as SGE, LSF, torque etc.
I need to build an openmpi application that can interact with variety of different resource distribution systems, since different customers have different systems. Based on my research, it seems that I need to build a different openmpi installation to work, e.g. create an installation of opempi with grid and create a different installation of openmpi with LSF. Is there a way to build a generic installation of openmpi that can be used with more than 1 distribution system by using some generic mechanism?

Just to be clear: your application doesn’t depend on the environment in any way. Only mpirun does - so if you are distributing an _application_, then your question is irrelevant.

If you are distributing OMPI itself, and therefore mpirun, then you can build the various components if you first install the headers for that environment on your system. It means that you need one machine where all those resource managers at least have their headers installed on it. Then configure OMPI --with-xxx pointing to each of the RM’s headers so all the components get built. When the binary hits your customer’s machine, only those components that have active libraries present will execute.

Post by Kulshrestha, Vipul
2.
For integration with LSF/grid, how would I specify the memory (RAM) requirement (or some other parameter) to bsub/qsub, when launching mpirun command? Will something like below work to ensure that each of the 8 copies of a.out have 40 GB memory reserved for them by grid engine?
qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out

You’ll have to provide something that is environment dependent, I’m afraid - there is no standard out there.

Post by Kulshrestha, Vipul
3.
Some of our customers use custom distribution engine (some non-industry-standard distribution engine). How can I integrate my openmpi application with such system? I would think that it should be possible to do that if openmpi launched/managed interaction with the distribution engine using some kind of generic mechanism (say, use a configurable command to launch, monitor, kill a job and then allow specification of a plugin define these operations with commands specific to the distribution engine being in use). Does such integration exist in openmpi?

Easiest solution is to write a script that reads the allocation and dumps it into a file, and then provide that file as your hostfile on the mpirun cmd line (or in the environment). We will then use ssh to perform the launch. Otherwise, you’ll need to write at least an orte/mca/ras component to get the allocation, and possibly an orte/mca/plm component if you want to use the native launch mechanism in place of ssh.

Post by Kulshrestha, Vipul
Thanks,
Vipul
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>

[Dave Love]

Dave,
unless you are doing direct launch (for example, use 'srun' instead of
'mpirun' under SLURM),
this is the way Open MPI is working : mpirun will use whatever the
resource manager provides
in order to spawn the remote orted (tm with PBS, qrsh with SGE, srun
with SLURM, ...).
then mpirun/orted will fork&exec the MPI tasks.

I know quite well how SGE works with openmpi, which isn't special --
I've done enough work on it. SGE tracks the process tree under orted
just like under bash, even if things daemonize. The OP was correct.

I should qualify that by noting that ENABLE_ADDGRP_KILL has apparently
never propagated through remote startup, so killing those orphans after
VASP crashes may fail, though resource reporting works. (I never
installed a fix for want of a test system, but it's not needed with
Linux cpusets.)

[Dave Love]

Post by Dave Love
I should qualify that by noting that ENABLE_ADDGRP_KILL has apparently
never propagated through remote startup,

Isn't it a setting inside SGE which the sge_execd is aware of? I never
exported any environment variable for this purpose.

Yes, but this is surely off-topic, even though
<https://arc.liv.ac.uk/trac/SGE/ticket/674> mentions openmpi.