[OMPI users] MPI the correct solution?

[d***@mail.com]

On Tue, 9 May 2017 00:30:38 +0200

Hi,

Hello,
I originally ported this question at LQ, but the answer I got back
shows rather poor insight on the subject of MPI, so I'm taking the
liberty of posting here also.
https://www.linuxquestions.org/questions/showthread.php?p=5707962
What I'm trying to do is figure out how/what to use to update an osm
file (open street map), in a cross system manner. I know the correct
program osmosis and for de/re-compression lbzip2 but how to do this
across computers is confusing me, even after a few hours of searching
online.

lbzip2 is only thread parallel on a single machine. With pbzip2 you
mention it's the same, but it exists an MPI version MPIBZIP2 -

I can't find the project, do you have a link?

unfortunately it looks unmaintained since 2007. Maybe you can contact
the author about its state. Without an MPI application like this, the
MPI library is nothing on its own which would divide and distribute one
task to several machines automatically.

Well, there might be other ways to cause a program to run on multiple
computers. Perhaps a virtual machine made of of multiple physical
machines?

osmosis itself seems to run in serial only (they don't say any word
whether it uses any parallelism).

Yes, it does run multiple threads, you just start another task (and add a
buffer). I tested this on my machine, I think it is --read-xml
--write-xml and --read-xml-change that start new threads. The question is
whether or not java is naively MPI aware or does the app need special
coding?

For the intended task the only option is to use a single machine with
as many cores as possible AFAICS.

Though about that, and it is doable with respect to memory and disk
constraints, the problem is that it would take a *long* time esp. with the
amount of updates I must do, hence my inquiry.

Thanks,
David

[Reuti]

As I think it's not relevant to Open MPI itself, I answered in PM only.

-- Reuti

Post by d***@mail.com
On Tue, 9 May 2017 00:30:38 +0200

Hi,

Hello,
I originally ported this question at LQ, but the answer I got back
shows rather poor insight on the subject of MPI, so I'm taking the
liberty of posting here also.
https://www.linuxquestions.org/questions/showthread.php?p=5707962
What I'm trying to do is figure out how/what to use to update an osm
file (open street map), in a cross system manner. I know the correct
program osmosis and for de/re-compression lbzip2 but how to do this
across computers is confusing me, even after a few hours of searching
online.

lbzip2 is only thread parallel on a single machine. With pbzip2 you
mention it's the same, but it exists an MPI version MPIBZIP2 -

I can't find the project, do you have a link?

unfortunately it looks unmaintained since 2007. Maybe you can contact
the author about its state. Without an MPI application like this, the
MPI library is nothing on its own which would divide and distribute one
task to several machines automatically.

Well, there might be other ways to cause a program to run on multiple
computers. Perhaps a virtual machine made of of multiple physical
machines?

osmosis itself seems to run in serial only (they don't say any word
whether it uses any parallelism).

Yes, it does run multiple threads, you just start another task (and add a
buffer). I tested this on my machine, I think it is --read-xml
--write-xml and --read-xml-change that start new threads. The question is
whether or not java is naively MPI aware or does the app need special
coding?

For the intended task the only option is to use a single machine with
as many cores as possible AFAICS.

Though about that, and it is doable with respect to memory and disk
constraints, the problem is that it would take a *long* time esp. with the
amount of updates I must do, hence my inquiry.
Thanks,
David
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users