[OMPI users] centos 7.2 openmpi from repo, stdout issue

[Emre Brookes]

$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

$ yum list installed | grep openmpi
openmpi.x86_64 1.10.0-10.el7 @base
openmpi-devel.x86_64 1.10.0-10.el7 @base

(1) When I run
$ mpirun -H myhosts -np myprocs executable
the job runs fine and outputs correctly to stdout

(2) When I run
$ mpirun -H myhosts -np myprocs executable > stdout.log
The stdout.log file prematurely ends (without full output)
... but the mpi executable itself seems to keep running forever until
manually terminated will a "kill".

(3) When I run
$ mpirun -H myhosts -np myprocs executable | cat > stdout.log
the job runs fine and outputs correctly to the stdout.log file

I tried playing with a 'stdbuf' prefix to the command, but this didn't
seem to help
I would like (2) to work, but have resorted to (3).

I tried digging around in the parameters after seeing
https://github.com/open-mpi/ompi/issues/341
and thinking it might be something similar, but didn't see any poll or
epoll in .conf
I am hesitant to try to compile from scratch and get away from the repo
release cycle.

Is this a known bug?
If so, and if it has been fixed, would you recommend I install the
latest stable rpm of 1.10.4-1 from
https://www.open-mpi.org/software/ompi/v1.10/ ?

Thanks,
Emre

[Jeff Squyres (jsquyres)]

We did have some kind of stdout/stderr truncation issue a little while ago, but I don't remember what version it specifically affected.

I would definitely update to at least Open MPI 1.10.4 (lots of bug fixes since 1.10.0). Better would be to update to Open MPI 2.0.1 -- that's the current generation and where all of our work is going these days.

Post by Emre Brookes
$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
$ yum list installed | grep openmpi
(1) When I run
$ mpirun -H myhosts -np myprocs executable
the job runs fine and outputs correctly to stdout
(2) When I run
$ mpirun -H myhosts -np myprocs executable > stdout.log
The stdout.log file prematurely ends (without full output)
... but the mpi executable itself seems to keep running forever until manually terminated will a "kill".
(3) When I run
$ mpirun -H myhosts -np myprocs executable | cat > stdout.log
the job runs fine and outputs correctly to the stdout.log file
I tried playing with a 'stdbuf' prefix to the command, but this didn't seem to help
I would like (2) to work, but have resorted to (3).
I tried digging around in the parameters after seeing https://github.com/open-mpi/ompi/issues/341
and thinking it might be something similar, but didn't see any poll or epoll in .conf
I am hesitant to try to compile from scratch and get away from the repo release cycle.
Is this a known bug?
If so, and if it has been fixed, would you recommend I install the latest stable rpm of 1.10.4-1 from https://www.open-mpi.org/software/ompi/v1.10/ ?
Thanks,
Emre
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Emre Brookes]

Thank you for the sanity check and recommendations.
I will post my results here when resolved.

Post by Jeff Squyres (jsquyres)
We did have some kind of stdout/stderr truncation issue a little while ago, but I don't remember what version it specifically affected.
I would definitely update to at least Open MPI 1.10.4 (lots of bug fixes since 1.10.0). Better would be to update to Open MPI 2.0.1 -- that's the current generation and where all of our work is going these days.

Post by Emre Brookes
$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
$ yum list installed | grep openmpi
(1) When I run
$ mpirun -H myhosts -np myprocs executable
the job runs fine and outputs correctly to stdout
(2) When I run
$ mpirun -H myhosts -np myprocs executable > stdout.log
The stdout.log file prematurely ends (without full output)
... but the mpi executable itself seems to keep running forever until manually terminated will a "kill".
(3) When I run
$ mpirun -H myhosts -np myprocs executable | cat > stdout.log
the job runs fine and outputs correctly to the stdout.log file
I tried playing with a 'stdbuf' prefix to the command, but this didn't seem to help
I would like (2) to work, but have resorted to (3).
I tried digging around in the parameters after seeing https://github.com/open-mpi/ompi/issues/341
and thinking it might be something similar, but didn't see any poll or epoll in .conf
I am hesitant to try to compile from scratch and get away from the repo release cycle.
Is this a known bug?
If so, and if it has been fixed, would you recommend I install the latest stable rpm of 1.10.4-1 from https://www.open-mpi.org/software/ompi/v1.10/ ?
Thanks,
Emre
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Emre Brookes]

FYI -

We upgraded to Open MPI 2.0.1 and this resolved the issue.

Of course, it was not so simple to get there, as the Centos 7.2 default
gcc (4.8.4) produced "internal compiler error" when recompiling NAMD
with OMPI 2.0.1 and 1.10.4. So we had to install a newer compiler. One
interesting result was the OMPI + NAMD performance comparisons with
various gcc versions. gcc 5.4.0 significantly outperformed gcc 6.2.0 on
our tests...

Post by Jeff Squyres (jsquyres)
We did have some kind of stdout/stderr truncation issue a little while ago, but I don't remember what version it specifically affected.
I would definitely update to at least Open MPI 1.10.4 (lots of bug fixes since 1.10.0). Better would be to update to Open MPI 2.0.1 -- that's the current generation and where all of our work is going these days.

Post by Emre Brookes
$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
$ yum list installed | grep openmpi
(1) When I run
$ mpirun -H myhosts -np myprocs executable
the job runs fine and outputs correctly to stdout
(2) When I run
$ mpirun -H myhosts -np myprocs executable > stdout.log
The stdout.log file prematurely ends (without full output)
... but the mpi executable itself seems to keep running forever until manually terminated will a "kill".
(3) When I run
$ mpirun -H myhosts -np myprocs executable | cat > stdout.log
the job runs fine and outputs correctly to the stdout.log file
I tried playing with a 'stdbuf' prefix to the command, but this didn't seem to help
I would like (2) to work, but have resorted to (3).
I tried digging around in the parameters after seeing https://github.com/open-mpi/ompi/issues/341
and thinking it might be something similar, but didn't see any poll or epoll in .conf
I am hesitant to try to compile from scratch and get away from the repo release cycle.
Is this a known bug?
If so, and if it has been fixed, would you recommend I install the latest stable rpm of 1.10.4-1 from https://www.open-mpi.org/software/ompi/v1.10/ ?
Thanks,
Emre
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users