[OMPI users] MPi Abort verbosity

Discussion:

Gabriele Fatigati

2010-02-24 08:55:28 UTC

Dear Openmpi users and developers,

i have a question about MPI_Abort error message. I have a program written in
C++. Is there a way to decrease a verbosity of this error? When this
function is called, openmpi prints many information like stack trace, rank
of processor who called MPI_Abort ecc.. But i'm interesting just called
rank. Is it possible?

Thanks in advance.

I'm using openmpi 1.2.2
--
Ing. Gabriele Fatigati

Parallel programmer

CINECA Systems & Tecnologies Department

Supercomputing Group

Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy

www.cineca.it Tel: +39 051 6171722

g.fatigati [AT] cineca.it

Nadia Derbey

2010-02-24 09:37:35 UTC

Permalink

Post by Gabriele Fatigati
Dear Openmpi users and developers,
i have a question about MPI_Abort error message. I have a program
written in C++. Is there a way to decrease a verbosity of this error?
When this function is called, openmpi prints many information like
stack trace, rank of processor who called MPI_Abort ecc.. But i'm
interesting just called rank. Is it possible?

Hi,

Setting the mca parameter "mpi_abort_print_stack" to 0 makes the stack
not printed out.

Post by Gabriele Fatigati
Thanks in advance.
I'm using openmpi 1.2.2

... well, don't know if it's available in that release...

Regards,
Nadia

Post by Gabriele Fatigati
--
Ing. Gabriele Fatigati
Parallel programmer
CINECA Systems & Tecnologies Department
Supercomputing Group
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati [AT] cineca.it
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

--
Nadia Derbey <***@bull.net>

Gabriele Fatigati

2010-02-24 10:28:08 UTC

Permalink

Hi Nadia,

thanks for quick reply.

But i suppose that parameter is 0 by default. Suppose i have the follw
output:

- ------------------------------
--------------------------------------------
- --> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 4. <--

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
- --------------------------------------------------------------------------
Inside my_mpi_err_handler
Inside my_mpi_err_handler
I am 0 and we are in 2
I am 1 and we are in 2
- --------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 3773 on
node nb-user exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
- --------------------------------------------------------------------------
- --

I would like to see only this:

- --> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 4. <--

And nothing else. Is it possible?

I can upgrade my OpenMPI if necessary.

Thanks.

Post by Nadia Derbey

Hi,
Setting the mca parameter "mpi_abort_print_stack" to 0 makes the stack
not printed out.

Post by Gabriele Fatigati
Thanks in advance.
I'm using openmpi 1.2.2

... well, don't know if it's available in that release...
Regards,
Nadia

--
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

jody

2010-02-24 12:22:58 UTC

Permalink

Hi Gabriele
you could always pipe your output through grep

my_app | grep "MPI_ABORT was invoked"

jody

On Wed, Feb 24, 2010 at 11:28 AM, Gabriele Fatigati

Post by Gabriele Fatigati
Hi Nadia,
thanks for quick reply.
But i suppose that parameter is 0 by default. Suppose i have the follw
- ------------------------------
--------------------------------------------
- --> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 4. <--
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
- --------------------------------------------------------------------------
Inside my_mpi_err_handler
Inside my_mpi_err_handler
I am 0 and we are in 2
I am 1 and we are in 2
- --------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 3773 on
node nb-user exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
- --------------------------------------------------------------------------
- --
- --> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 4. <--
And nothing else. Is it possible?
I can upgrade my OpenMPI if necessary.
Thanks.

Post by Nadia Derbey

Hi,
Setting the mca parameter "mpi_abort_print_stack" to 0 makes the stack
not printed out.

Post by Gabriele Fatigati
Thanks in advance.
I'm using openmpi 1.2.2

... well, don't know if it's available in that release...
Regards,
Nadia

--
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

--
Ing. Gabriele Fatigati
Parallel programmer
CINECA Systems & Tecnologies Department
Supercomputing Group
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati [AT] cineca.it
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Gabriele Fatigati

2010-02-24 13:21:02 UTC

Permalink

Yes, of course,

but i would like to know if there is any way to do that with openmpi

Post by jody
Hi Gabriele
you could always pipe your output through grep
my_app | grep "MPI_ABORT was invoked"
jody
On Wed, Feb 24, 2010 at 11:28 AM, Gabriele Fatigati

--------------------------------------------------------------------------

Post by Gabriele Fatigati
Inside my_mpi_err_handler
Inside my_mpi_err_handler
I am 0 and we are in 2
I am 1 and we are in 2
-

--------------------------------------------------------------------------

Post by Gabriele Fatigati
mpirun has exited due to process rank 0 with PID 3773 on
node nb-user exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
-

--------------------------------------------------------------------------

Post by Gabriele Fatigati
- --
- --> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 4. <--
And nothing else. Is it possible?
I can upgrade my OpenMPI if necessary.
Thanks.

Post by Nadia Derbey

Hi,
Setting the mca parameter "mpi_abort_print_stack" to 0 makes the stack
not printed out.

Post by Gabriele Fatigati
Thanks in advance.
I'm using openmpi 1.2.2

... well, don't know if it's available in that release...
Regards,
Nadia

--
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Jed Brown

2010-02-24 13:36:13 UTC

Permalink

Post by Gabriele Fatigati
Yes, of course,
but i would like to know if there is any way to do that with openmpi

See the error handler docs, e.g. MPI_Comm_set_errhandler.

Jed

Ralph Castain

2010-02-24 13:50:05 UTC

Permalink

I don't believe the error handler will help suppress the messages you are trying to avoid as they don't originate in the MPI layer. They are actually generated in the RTE layer as mpirun is exiting.

You could try adding the --quiet option to your mpirun cmd line. This will help eliminate some (maybe not all) of the verbage.

Post by Jed Brown

Post by Gabriele Fatigati
Yes, of course,
but i would like to know if there is any way to do that with openmpi

See the error handler docs, e.g. MPI_Comm_set_errhandler.
Jed
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Gabriele Fatigati

2010-02-24 14:17:53 UTC

Permalink

Mm,
i'm trying to explain better.

My target is, when a MPI process dead for some reason, after launched
MPI_Abort i would like to control this behaviour. Example:

rank 0 died and launc MPI_Abort

i would like to do something before other process died. So i want to control
shutdown of my MPI application. Is it possible?

Post by Ralph Castain
I don't believe the error handler will help suppress the messages you are
trying to avoid as they don't originate in the MPI layer. They are actually
generated in the RTE layer as mpirun is exiting.
You could try adding the --quiet option to your mpirun cmd line. This will
help eliminate some (maybe not all) of the verbage.

On Wed, 24 Feb 2010 14:21:02 +0100, Gabriele Fatigati <

Post by Gabriele Fatigati
Yes, of course,
but i would like to know if there is any way to do that with openmpi

See the error handler docs, e.g. MPI_Comm_set_errhandler.
Jed
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Ralph Castain

2010-02-24 14:36:28 UTC

Permalink

I'm afraid not. We are working on alternative error response mechanisms, but nothing is released at this time.

Post by Gabriele Fatigati
Mm,
i'm trying to explain better.
rank 0 died and launc MPI_Abort
i would like to do something before other process died. So i want to control shutdown of my MPI application. Is it possible?
I don't believe the error handler will help suppress the messages you are trying to avoid as they don't originate in the MPI layer. They are actually generated in the RTE layer as mpirun is exiting.
You could try adding the --quiet option to your mpirun cmd line. This will help eliminate some (maybe not all) of the verbage.

Post by Jed Brown

Post by Gabriele Fatigati
Yes, of course,
but i would like to know if there is any way to do that with openmpi

See the error handler docs, e.g. MPI_Comm_set_errhandler.
Jed
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Ing. Gabriele Fatigati
Parallel programmer
CINECA Systems & Tecnologies Department
Supercomputing Group
Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy
www.cineca.it Tel: +39 051 6171722
g.fatigati [AT] cineca.it
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

Nadia Derbey

2010-02-24 14:44:04 UTC

Permalink

Post by Ralph Castain
I'm afraid not. We are working on alternative error response
mechanisms, but nothing is released at this time.

Don't know if this would work, but why not doing what follows:
1. set a signal handler in your application. This where you would do
your cleanup.
2. call your application with an mpi_abort_delay set to a value > 0 (0
is the default value). This should delay the actual procs abort.
3. During that delay, and after you see the message about process going
to abort, send the signal that should be caught in #1.

But may be I'm wrong, as I told you I've never tested that...

Regards,
Nadia

Post by Ralph Castain

Post by Gabriele Fatigati
Mm,
i'm trying to explain better.
My target is, when a MPI process dead for some reason, after
rank 0 died and launc MPI_Abort
i would like to do something before other process died. So i want to
control shutdown of my MPI application. Is it possible?
I don't believe the error handler will help suppress the
messages you are trying to avoid as they don't originate in
the MPI layer. They are actually generated in the RTE layer
as mpirun is exiting.
You could try adding the --quiet option to your mpirun cmd
line. This will help eliminate some (maybe not all) of the
verbage.

On Wed, 24 Feb 2010 14:21:02 +0100, Gabriele Fatigati

Post by Gabriele Fatigati
Yes, of course,
but i would like to know if there is any way to do that

with openmpi

See the error handler docs, e.g. MPI_Comm_set_errhandler.
Jed
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

--
Nadia Derbey <***@bull.net>

Rodolfo Chua

2010-02-24 16:04:50 UTC

Permalink

I've successfully installed openMPI on other PC. But when I tried to install it
on my laptop and typed 'mpicc' , the response was:

The program 'mpicc' can be found in the following packages:
* lam4-dev
* libmpich-mpd1.0-dev
* libmpich-shmem1.0-dev
* libmpich1.0-dev
* libopenmpi-dev
* mpich2
Try: sudo apt-get install <selected package>
mpicc: command not found
***@ubuntu:~$ sudo apt-get install libopenmpi-dev
[sudo] password for chao:
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package liopenmpi-dev

Of course I'm connected to the internet. Please help..

________________________________
From: Nadia Derbey <***@bull.net>
To: Open MPI Users <***@open-mpi.org>
Sent: Wed, February 24, 2010 10:44:04 PM
Subject: Re: [OMPI users] MPi Abort verbosity

Post by Ralph Castain
I'm afraid not. We are working on alternative error response
mechanisms, but nothing is released at this time.

Post by Ralph Castain

On Wed, 24 Feb 2010 14:21:02 +0100, Gabriele Fatigati

Post by Gabriele Fatigati
Yes, of course,
but i would like to know if there is any way to do that

with openmpi

See the error handler docs, e.g. MPI_Comm_set_errhandler.
Jed
_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

_______________________________________________
users mailing list
http://www.open-mpi.org/mailman/listinfo.cgi/users

--
Nadia Derbey <***@bull.net>

Jeff Squyres

2010-02-24 16:34:11 UTC

Permalink

Post by Rodolfo Chua
I've successfully installed openMPI on other PC. But when I tried to install it

Please do not reply off-topic -- please start a new thread with a different subject if you have an unrelated question. Otherwise, the threading in both the web archives and people's email clients will be incorrect. Thanks.

Post by Rodolfo Chua
* lam4-dev
* libmpich-mpd1.0-dev
* libmpich-shmem1.0-dev
* libmpich1.0-dev
* libopenmpi-dev
* mpich2
Try: sudo apt-get install <selected package>
mpicc: command not found
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package liopenmpi-dev

You'll likely need to check with the apt-get packagers to see what package you need to install.

FWIW, I see that you said install "libopenmpi-dev", but it said it couldn't find "liopenmpi-dev". Was there a typo in there somewhere?

--
Jeff Squyres
***@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/