[OMPI users] check for CUDA support

Discussion:

Andrei Berceanu

2018-10-30 18:33:46 UTC

Permalink

Hi all,

I am using ***@3.1.2 on an ***@16.04.5 box, how can I check if it
has CUDA support or not?

Best,
Andrei

Akshay Venkatesh

2018-10-30 18:47:29 UTC

Permalink

Andrei,

I generally check with one of these two:

$ ompi_info -a | grep "\-with\-cuda"
Configure command line: '--prefix=$HOME/ompi/build-cuda'
'--enable-mpirun-prefix-by-default' '--with-cuda=/usr/local/cuda'
'--with-ucx=$HOME/ucx-github/build'
'--with-ucx-libdir=$HOME/ucx-github/build/lib' '--enable-debug'
'--enable-mem-debug' '--enable-mpi-fortran=no' '--disable-oshmem'
'--enable-install-libpmix' '--with-ompi-pmix-rte'

$ ompi_info -a | grep "xtensions"
MPI extensions: affinity, cuda, pcollreq

Post by Andrei Berceanu
Hi all,
has CUDA support or not?
Best,
Andrei
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
-Akshay
NVIDIA

Andrei Berceanu

2018-10-30 19:07:32 UTC

Permalink

Thanks! I tried both and
ompi_info -a | grep "\-with\-cuda" returns nothing, while
ompi_info -a | grep "xtensions" returns
MPI extensions: affinity, cuda

It seems the two outputs are in conflict, what does that mean?

Post by Akshay Venkatesh
Andrei,
$ ompi_info -a | grep "\-with\-cuda"
Configure command line: '--prefix=$HOME/ompi/build-cuda'
'--enable-mpirun-prefix-by-default' '--with-cuda=/usr/local/cuda'
'--with-ucx=$HOME/ucx-github/build'
'--with-ucx-libdir=$HOME/ucx-github/build/lib' '--enable-debug'
'--enable-mem-debug' '--enable-mpi-fortran=no' '--disable-oshmem'
'--enable-install-libpmix' '--with-ompi-pmix-rte'
$ ompi_info -a | grep "xtensions"
MPI extensions: affinity, cuda, pcollreq
On Tue, Oct 30, 2018 at 11:40 AM Andrei Berceanu <

Post by Andrei Berceanu
Hi all,
has CUDA support or not?
Best,
Andrei
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
-Akshay
NVIDIA
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

Akshay Venkatesh

2018-10-30 19:14:04 UTC

Permalink

The first one is the critical one. If the build was not configured
--with-cuda, then it's unlikely cuda-aware MPI is supported.

Post by Andrei Berceanu
Thanks! I tried both and
ompi_info -a | grep "\-with\-cuda" returns nothing, while
ompi_info -a | grep "xtensions" returns
MPI extensions: affinity, cuda
It seems the two outputs are in conflict, what does that mean?

Post by Andrei Berceanu
Hi all,
it has CUDA support or not?
Best,
Andrei
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
-Akshay
NVIDIA
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

Jeff Squyres (jsquyres) via users

2018-10-30 19:43:33 UTC

Permalink

The "Configure command line" shows you the command line that was given to "configure" when building Open MPI.

The "MPI extensions" line just indicates which Open MPI "extensions" were built.

CUDA is one of the possible extensions that can get built.

The CUDA Open MPI extension is actually an API call that will tell you if your Open MPI has CUDA support:

https://www.open-mpi.org/doc/v3.1/man3/MPIX_Query_cuda_support.3.php

The first one is the critical one. If the build was not configured --with-cuda, then it's unlikely cuda-aware MPI is supported.
Thanks! I tried both and
ompi_info -a | grep "\-with\-cuda" returns nothing, while
ompi_info -a | grep "xtensions" returns
MPI extensions: affinity, cuda
It seems the two outputs are in conflict, what does that mean?
Andrei,
$ ompi_info -a | grep "\-with\-cuda"
Configure command line: '--prefix=$HOME/ompi/build-cuda' '--enable-mpirun-prefix-by-default' '--with-cuda=/usr/local/cuda' '--with-ucx=$HOME/ucx-github/build' '--with-ucx-libdir=$HOME/ucx-github/build/lib' '--enable-debug' '--enable-mem-debug' '--enable-mpi-fortran=no' '--disable-oshmem' '--enable-install-libpmix' '--with-ompi-pmix-rte'
$ ompi_info -a | grep "xtensions"
MPI extensions: affinity, cuda, pcollreq
Hi all,
Best,
Andrei
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
--
-Akshay
NVIDIA
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
Jeff Squyres
***@cisco.com

Akshay Venkatesh

2018-10-30 20:29:50 UTC

Permalink

+1 to what Jeff said.

So you would need --with-cuda pointing to a cuda installation to have
cuda-awareness in OpenMPI.

On Tue, Oct 30, 2018 at 12:47 PM Jeff Squyres (jsquyres) via users <

Post by Jeff Squyres (jsquyres) via users
The "Configure command line" shows you the command line that was given to
"configure" when building Open MPI.
The "MPI extensions" line just indicates which Open MPI "extensions" were built.
CUDA is one of the possible extensions that can get built.
The CUDA Open MPI extension is actually an API call that will tell you if
https://www.open-mpi.org/doc/v3.1/man3/MPIX_Query_cuda_support.3.php

Post by Akshay Venkatesh
The first one is the critical one. If the build was not configured

--with-cuda, then it's unlikely cuda-aware MPI is supported.

Post by Akshay Venkatesh
On Tue, Oct 30, 2018, 12:10 PM Andrei Berceanu <
Thanks! I tried both and
ompi_info -a | grep "\-with\-cuda" returns nothing, while
ompi_info -a | grep "xtensions" returns
MPI extensions: affinity, cuda
It seems the two outputs are in conflict, what does that mean?
On Tue, Oct 30, 2018 at 8:50 PM Akshay Venkatesh <
Andrei,
$ ompi_info -a | grep "\-with\-cuda"
Configure command line: '--prefix=$HOME/ompi/build-cuda'

'--enable-mpirun-prefix-by-default' '--with-cuda=/usr/local/cuda'
'--with-ucx=$HOME/ucx-github/build'
'--with-ucx-libdir=$HOME/ucx-github/build/lib' '--enable-debug'
'--enable-mem-debug' '--enable-mpi-fortran=no' '--disable-oshmem'
'--enable-install-libpmix' '--with-ompi-pmix-rte'

Post by Akshay Venkatesh
$ ompi_info -a | grep "xtensions"
MPI extensions: affinity, cuda, pcollreq
On Tue, Oct 30, 2018 at 11:40 AM Andrei Berceanu <
Hi all,

it has CUDA support or not?

Post by Akshay Venkatesh
Best,
Andrei
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
--
-Akshay
NVIDIA
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
Jeff Squyres
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
-Akshay
NVIDIA

Jeff Squyres (jsquyres) via users

2018-10-30 20:42:10 UTC

Permalink

Oh, I should clarify: that MPIX_Query_cuda_support() function is always present in Open MPI -- even if you don't compile with CUDA support.

That's kinda the point / it's probably obvious, but I thought I would clarify, anyway. :-)

Post by Akshay Venkatesh
+1 to what Jeff said.
So you would need --with-cuda pointing to a cuda installation to have cuda-awareness in OpenMPI.
The "Configure command line" shows you the command line that was given to "configure" when building Open MPI.
The "MPI extensions" line just indicates which Open MPI "extensions" were built.
CUDA is one of the possible extensions that can get built.
https://www.open-mpi.org/doc/v3.1/man3/MPIX_Query_cuda_support.3.php

--
Jeff Squyres
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
--
-Akshay
NVIDIA

--
Jeff Squyres
***@cisco.com

Akshay Venkatesh

2018-10-30 21:01:58 UTC

Permalink

Thanks for the clarification, Jeff.

Andrei,
MPIX_Query_cuda_support indeed just returns the value of OPAL_CUDA_SUPPORT
which is set to one if configure included --with-cuda.

/* If CUDA-aware support is configured in, return 1. Otherwise, return 0.

* This API may be extended to return more features in the future. */
int MPIX_Query_cuda_support(void)
{
return OPAL_CUDA_SUPPORT;
}

You can see the details of how OPAL_CUDA_SUPPORT is set in
root_dir/config/opal_check_cuda.m4

In summary, you could use MPIX_Query_cuda_support in your application to
check during runtime or you could check ompi_info to see if the build was
configured correctly --with-cuda before running a cuda-aware MPI
application.

Post by Jeff Squyres (jsquyres) via users
Oh, I should clarify: that MPIX_Query_cuda_support() function is always
present in Open MPI -- even if you don't compile with CUDA support.
That's kinda the point / it's probably obvious, but I thought I would clarify, anyway. :-)

Post by Akshay Venkatesh
+1 to what Jeff said.
So you would need --with-cuda pointing to a cuda installation to have

cuda-awareness in OpenMPI.

Post by Akshay Venkatesh
On Tue, Oct 30, 2018 at 12:47 PM Jeff Squyres (jsquyres) via users <
The "Configure command line" shows you the command line that was given

to "configure" when building Open MPI.

Post by Akshay Venkatesh
The "MPI extensions" line just indicates which Open MPI "extensions"

were built.

Post by Akshay Venkatesh
CUDA is one of the possible extensions that can get built.
The CUDA Open MPI extension is actually an API call that will tell you
https://www.open-mpi.org/doc/v3.1/man3/MPIX_Query_cuda_support.3.php

Post by Akshay Venkatesh
The first one is the critical one. If the build was not configured

--with-cuda, then it's unlikely cuda-aware MPI is supported.

Post by Akshay Venkatesh

Post by Akshay Venkatesh
$ ompi_info -a | grep "xtensions"
MPI extensions: affinity, cuda, pcollreq
On Tue, Oct 30, 2018 at 11:40 AM Andrei Berceanu <
Hi all,

it has CUDA support or not?

Post by Akshay Venkatesh

--
Jeff Squyres
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users
--
-Akshay
NVIDIA

--
Jeff Squyres

--
-Akshay
NVIDIA

Andrei Berceanu

2018-10-30 19:18:40 UTC

Permalink

In fact, I have "'--without-cuda":

Configure command line:
'--prefix=/home/andrei/src/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.3.0/openmpi-3.1.2-ocjvxwsaq3qfm6342xryaxgcesjnn4a6'
'--enable-shared' '--with-wrapper-ldflags=' '--enable-static'
'--without-pmi' '--enable-mpi-cxx'
'--with-zlib=/home/andrei/src/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.3.0/zlib-1.2.11-m5um2ez3iajeehvt4sp7cgqnhjaybyjy'
'--without-psm' '--without-psm2' '--without-verbs' '--without-mxm'
'--without-ucx' '--without-libfabric' '--without-alps' '--without-lsf'
'--without-tm' '--without-slurm' '--without-sge' '--without-loadleveler'
'--disable-memchecker'
'--with-hwloc=/home/andrei/src/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.3.0/hwloc-1.11.9-izyltij6nzgb7gmknxufz5znfg763zgq'
'--disable-java' '--disable-mpi-java' '--without-cuda'
'--enable-cxx-exceptions'

Post by Andrei Berceanu
Hi all,
it has CUDA support or not?
Best,
Andrei
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users

--
-Akshay
NVIDIA
_______________________________________________
users mailing list
https://lists.open-mpi.org/mailman/listinfo/users