[OMPI users] OpenMPI 2.1.0 + PGI 17.3 = asm test failures

Discussion:

Prentice Bisbal

2017-04-27 22:00:11 UTC

I'm building Open MPI 2.1.0 with PGI 17.3, and now I'm getting 'illegal
instruction' errors during 'make check':

../../config/test-driver: line 107: 65169 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math
- 1 threads: Passed

That's just one example of the error output. See all relevant error
output below.

Usually, I see these errors when trying to run an executable on a
processor that doesn't support the instruction set of the executable. I
used to see this all the time when I supported an IBM Blue Gene/P
system. I don't think I've ever seen it on an x86 system.

I'm passing the argument '-tp=x64' to pgcc to build a unified binary,
so that might be part of the problem, but I've used this exact same
process to build 2.1.0 with PGI 16.5 just a couple hours ago. I also
built 1.10.3 with the same compiler flags with PGI 16.5 and 17.3 without
this error.

Any ideas?

The relevant output from 'make check':

make check-TESTS
make[3]: Entering directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[4]: Entering directory `/local/pbisbal/openmpi-2.1.0/test/asm'
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_barrier
- 1 threads: Passed
PASS: atomic_barrier
- 2 threads: Passed
PASS: atomic_barrier
- 4 threads: Passed
PASS: atomic_barrier
- 5 threads: Passed
PASS: atomic_barrier
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_barrier_noinline
- 1 threads: Passed
PASS: atomic_barrier_noinline
- 2 threads: Passed
PASS: atomic_barrier_noinline
- 4 threads: Passed
PASS: atomic_barrier_noinline
- 5 threads: Passed
PASS: atomic_barrier_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_spinlock
- 1 threads: Passed
PASS: atomic_spinlock
- 2 threads: Passed
PASS: atomic_spinlock
- 4 threads: Passed
PASS: atomic_spinlock
- 5 threads: Passed
PASS: atomic_spinlock
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_spinlock_noinline
- 1 threads: Passed
PASS: atomic_spinlock_noinline
- 2 threads: Passed
PASS: atomic_spinlock_noinline
- 4 threads: Passed
PASS: atomic_spinlock_noinline
- 5 threads: Passed
PASS: atomic_spinlock_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65169 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math
- 1 threads: Passed
../../config/test-driver: line 107: 65172 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math
- 2 threads: Passed
../../config/test-driver: line 107: 65176 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math
- 4 threads: Passed
../../config/test-driver: line 107: 65180 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math
- 5 threads: Passed
../../config/test-driver: line 107: 65185 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65195 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math_noinline
- 1 threads: Passed
../../config/test-driver: line 107: 65198 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math_noinline
- 2 threads: Passed
../../config/test-driver: line 107: 65202 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math_noinline
- 4 threads: Passed
../../config/test-driver: line 107: 65206 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math_noinline
- 5 threads: Passed
../../config/test-driver: line 107: 65210 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_math_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65220 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_cmpset
- 1 threads: Passed
../../config/test-driver: line 107: 65223 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_cmpset
- 2 threads: Passed
../../config/test-driver: line 107: 65227 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_cmpset
- 4 threads: Passed
../../config/test-driver: line 107: 65231 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_cmpset
- 5 threads: Passed
../../config/test-driver: line 107: 65235 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_cmpset
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65245 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_cmpset_noinline
- 1 threads: Passed
../../config/test-driver: line 107: 65248 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_cmpset_noinline
- 2 threads: Passed
../../config/test-driver: line 107: 65252 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_cmpset_noinline
- 4 threads: Passed
../../config/test-driver: line 107: 65256 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_cmpset_noinline
- 5 threads: Passed
../../config/test-driver: line 107: 65260 Illegal instruction "$@" >
$log_file 2>&1
FAIL: atomic_cmpset_noinline
- 8 threads: Passed
============================================================================
Testsuite summary for Open MPI 2.1.0
============================================================================
# TOTAL: 8
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 4
# XPASS: 0
# ERROR: 0
============================================================================
See test/asm/test-suite.log
Please report to http://www.open-mpi.org/community/help/
============================================================================
make[4]: *** [test-suite.log] Error 1
make[4]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[3]: *** [check-TESTS] Error 2
make[3]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test'
make: *** [check-recursive] Error 1

The contents of test/asm/test-suite.log:

$ cat test/asm/test-suite.log
=============================================
Open MPI 2.1.0: test/asm/test-suite.log
=============================================

# TOTAL: 8
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 4
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: atomic_math
=================

FAIL atomic_math (exit status: 132)

FAIL: atomic_math_noinline
==========================

FAIL atomic_math_noinline (exit status: 132)

FAIL: atomic_cmpset
===================

FAIL atomic_cmpset (exit status: 132)

FAIL: atomic_cmpset_noinline
============================

FAIL atomic_cmpset_noinline (exit status: 132)

And my configure command:

./configure \
--prefix=/usr/pppl/pgi/17.3-pkgs/openmpi-2.1.0 \
--disable-silent-rules \
--enable-mpi-fortran \
--enable-mpi-cxx \
--enable-shared \
--enable-static \
--enable-mpi-thread-multiple \
--with-cuda=/usr/pppl/cuda/cudatoolkit/6.5.14 \
--with-pmix \
--with-verbs \
--with-hwloc \
--with-pmi=/usr/pppl/slurm/15.08.8 \
--with-slurm \
--with-psm \
CPP=cpp \
CC=pgcc \
CFLAGS="-fpic -tp=x64 -fast" \
CXX=pgc++ \
CXXFLAGS="-fpic -tp=x64 -fast" \
FC=pgfortran \
FCFLAGS="-fpic -tp=x64 -fast" \
2>&1 | tee configure.log

--
Prentice

Prentice Bisbal

2017-04-28 21:09:12 UTC

Permalink

Update: removing the -fast switch caused this error to go away.

Prentice

Post by Prentice Bisbal
I'm building Open MPI 2.1.0 with PGI 17.3, and now I'm getting
$log_file 2>&1
FAIL: atomic_math
- 1 threads: Passed
That's just one example of the error output. See all relevant error
output below.
Usually, I see these errors when trying to run an executable on a
processor that doesn't support the instruction set of the executable.
I used to see this all the time when I supported an IBM Blue Gene/P
system. I don't think I've ever seen it on an x86 system.
I'm passing the argument '-tp=x64' to pgcc to build a unified binary,
so that might be part of the problem, but I've used this exact same
process to build 2.1.0 with PGI 16.5 just a couple hours ago. I also
built 1.10.3 with the same compiler flags with PGI 16.5 and 17.3
without this error.
Any ideas?
make check-TESTS
make[3]: Entering directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[4]: Entering directory `/local/pbisbal/openmpi-2.1.0/test/asm'
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_barrier
- 1 threads: Passed
PASS: atomic_barrier
- 2 threads: Passed
PASS: atomic_barrier
- 4 threads: Passed
PASS: atomic_barrier
- 5 threads: Passed
PASS: atomic_barrier
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_barrier_noinline
- 1 threads: Passed
PASS: atomic_barrier_noinline
- 2 threads: Passed
PASS: atomic_barrier_noinline
- 4 threads: Passed
PASS: atomic_barrier_noinline
- 5 threads: Passed
PASS: atomic_barrier_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_spinlock
- 1 threads: Passed
PASS: atomic_spinlock
- 2 threads: Passed
PASS: atomic_spinlock
- 4 threads: Passed
PASS: atomic_spinlock
- 5 threads: Passed
PASS: atomic_spinlock
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_spinlock_noinline
- 1 threads: Passed
PASS: atomic_spinlock_noinline
- 2 threads: Passed
PASS: atomic_spinlock_noinline
- 4 threads: Passed
PASS: atomic_spinlock_noinline
- 5 threads: Passed
PASS: atomic_spinlock_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
$log_file 2>&1
FAIL: atomic_math
- 1 threads: Passed
$log_file 2>&1
FAIL: atomic_math
- 2 threads: Passed
$log_file 2>&1
FAIL: atomic_math
- 4 threads: Passed
$log_file 2>&1
FAIL: atomic_math
- 5 threads: Passed
$log_file 2>&1
FAIL: atomic_math
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
$log_file 2>&1
FAIL: atomic_math_noinline
- 1 threads: Passed
$log_file 2>&1
FAIL: atomic_math_noinline
- 2 threads: Passed
$log_file 2>&1
FAIL: atomic_math_noinline
- 4 threads: Passed
$log_file 2>&1
FAIL: atomic_math_noinline
- 5 threads: Passed
$log_file 2>&1
FAIL: atomic_math_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
$log_file 2>&1
FAIL: atomic_cmpset
- 1 threads: Passed
$log_file 2>&1
FAIL: atomic_cmpset
- 2 threads: Passed
$log_file 2>&1
FAIL: atomic_cmpset
- 4 threads: Passed
$log_file 2>&1
FAIL: atomic_cmpset
- 5 threads: Passed
$log_file 2>&1
FAIL: atomic_cmpset
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
$log_file 2>&1
FAIL: atomic_cmpset_noinline
- 1 threads: Passed
$log_file 2>&1
FAIL: atomic_cmpset_noinline
- 2 threads: Passed
$log_file 2>&1
FAIL: atomic_cmpset_noinline
- 4 threads: Passed
$log_file 2>&1
FAIL: atomic_cmpset_noinline
- 5 threads: Passed
$log_file 2>&1
FAIL: atomic_cmpset_noinline
- 8 threads: Passed
============================================================================
Testsuite summary for Open MPI 2.1.0
============================================================================
# TOTAL: 8
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 4
# XPASS: 0
# ERROR: 0
============================================================================
See test/asm/test-suite.log
Please report to http://www.open-mpi.org/community/help/
============================================================================
make[4]: *** [test-suite.log] Error 1
make[4]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[3]: *** [check-TESTS] Error 2
make[3]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test'
make: *** [check-recursive] Error 1
$ cat test/asm/test-suite.log
=============================================
Open MPI 2.1.0: test/asm/test-suite.log
=============================================
# TOTAL: 8
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 4
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: atomic_math
=================
FAIL atomic_math (exit status: 132)
FAIL: atomic_math_noinline
==========================
FAIL atomic_math_noinline (exit status: 132)
FAIL: atomic_cmpset
===================
FAIL atomic_cmpset (exit status: 132)
FAIL: atomic_cmpset_noinline
============================
FAIL atomic_cmpset_noinline (exit status: 132)
./configure \
--prefix=/usr/pppl/pgi/17.3-pkgs/openmpi-2.1.0 \
--disable-silent-rules \
--enable-mpi-fortran \
--enable-mpi-cxx \
--enable-shared \
--enable-static \
--enable-mpi-thread-multiple \
--with-cuda=/usr/pppl/cuda/cudatoolkit/6.5.14 \
--with-pmix \
--with-verbs \
--with-hwloc \
--with-pmi=/usr/pppl/slurm/15.08.8 \
--with-slurm \
--with-psm \
CPP=cpp \
CC=pgcc \
CFLAGS="-fpic -tp=x64 -fast" \
CXX=pgc++ \
CXXFLAGS="-fpic -tp=x64 -fast" \
FC=pgfortran \
FCFLAGS="-fpic -tp=x64 -fast" \
2>&1 | tee configure.log

Jeff Squyres (jsquyres)

2017-04-29 11:37:13 UTC

Permalink

IBM: can someone check to see if this is a compiler error?

Post by Prentice Bisbal
Update: removing the -fast switch caused this error to go away.
Prentice

Post by Prentice Bisbal
FAIL: atomic_math
- 1 threads: Passed
That's just one example of the error output. See all relevant error output below.
Usually, I see these errors when trying to run an executable on a processor that doesn't support the instruction set of the executable. I used to see this all the time when I supported an IBM Blue Gene/P system. I don't think I've ever seen it on an x86 system.
I'm passing the argument '-tp=x64' to pgcc to build a unified binary, so that might be part of the problem, but I've used this exact same process to build 2.1.0 with PGI 16.5 just a couple hours ago. I also built 1.10.3 with the same compiler flags with PGI 16.5 and 17.3 without this error.
Any ideas?
make check-TESTS
make[3]: Entering directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[4]: Entering directory `/local/pbisbal/openmpi-2.1.0/test/asm'
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_barrier
- 1 threads: Passed
PASS: atomic_barrier
- 2 threads: Passed
PASS: atomic_barrier
- 4 threads: Passed
PASS: atomic_barrier
- 5 threads: Passed
PASS: atomic_barrier
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_barrier_noinline
- 1 threads: Passed
PASS: atomic_barrier_noinline
- 2 threads: Passed
PASS: atomic_barrier_noinline
- 4 threads: Passed
PASS: atomic_barrier_noinline
- 5 threads: Passed
PASS: atomic_barrier_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_spinlock
- 1 threads: Passed
PASS: atomic_spinlock
- 2 threads: Passed
PASS: atomic_spinlock
- 4 threads: Passed
PASS: atomic_spinlock
- 5 threads: Passed
PASS: atomic_spinlock
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_spinlock_noinline
- 1 threads: Passed
PASS: atomic_spinlock_noinline
- 2 threads: Passed
PASS: atomic_spinlock_noinline
- 4 threads: Passed
PASS: atomic_spinlock_noinline
- 5 threads: Passed
PASS: atomic_spinlock_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
FAIL: atomic_math
- 1 threads: Passed
FAIL: atomic_math
- 2 threads: Passed
FAIL: atomic_math
- 4 threads: Passed
FAIL: atomic_math
- 5 threads: Passed
FAIL: atomic_math
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
FAIL: atomic_math_noinline
- 1 threads: Passed
FAIL: atomic_math_noinline
- 2 threads: Passed
FAIL: atomic_math_noinline
- 4 threads: Passed
FAIL: atomic_math_noinline
- 5 threads: Passed
FAIL: atomic_math_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
FAIL: atomic_cmpset
- 1 threads: Passed
FAIL: atomic_cmpset
- 2 threads: Passed
FAIL: atomic_cmpset
- 4 threads: Passed
FAIL: atomic_cmpset
- 5 threads: Passed
FAIL: atomic_cmpset
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
FAIL: atomic_cmpset_noinline
- 1 threads: Passed
FAIL: atomic_cmpset_noinline
- 2 threads: Passed
FAIL: atomic_cmpset_noinline
- 4 threads: Passed
FAIL: atomic_cmpset_noinline
- 5 threads: Passed
FAIL: atomic_cmpset_noinline
- 8 threads: Passed
============================================================================
Testsuite summary for Open MPI 2.1.0
============================================================================
# TOTAL: 8
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 4
# XPASS: 0
# ERROR: 0
============================================================================
See test/asm/test-suite.log
Please report to http://www.open-mpi.org/community/help/
============================================================================
make[4]: *** [test-suite.log] Error 1
make[4]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[3]: *** [check-TESTS] Error 2
make[3]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test'
make: *** [check-recursive] Error 1
$ cat test/asm/test-suite.log
=============================================
Open MPI 2.1.0: test/asm/test-suite.log
=============================================
# TOTAL: 8
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 4
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: atomic_math
=================
FAIL atomic_math (exit status: 132)
FAIL: atomic_math_noinline
==========================
FAIL atomic_math_noinline (exit status: 132)
FAIL: atomic_cmpset
===================
FAIL atomic_cmpset (exit status: 132)
FAIL: atomic_cmpset_noinline
============================
FAIL atomic_cmpset_noinline (exit status: 132)
./configure \
--prefix=/usr/pppl/pgi/17.3-pkgs/openmpi-2.1.0 \
--disable-silent-rules \
--enable-mpi-fortran \
--enable-mpi-cxx \
--enable-shared \
--enable-static \
--enable-mpi-thread-multiple \
--with-cuda=/usr/pppl/cuda/cudatoolkit/6.5.14 \
--with-pmix \
--with-verbs \
--with-hwloc \
--with-pmi=/usr/pppl/slurm/15.08.8 \
--with-slurm \
--with-psm \
CPP=cpp \
CC=pgcc \
CFLAGS="-fpic -tp=x64 -fast" \
CXX=pgc++ \
CXXFLAGS="-fpic -tp=x64 -fast" \
FC=pgfortran \
FCFLAGS="-fpic -tp=x64 -fast" \
2>&1 | tee configure.log

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

--
Jeff Squyres
***@cisco.com

Prentice Bisbal

2017-05-01 15:21:22 UTC

Permalink

Jeff,

Why IBM? This problem is caused by the PGI compilers, so shouldn't this
be directed towards NVidia, which now owns PGI?

Prentice

Post by Jeff Squyres (jsquyres)
IBM: can someone check to see if this is a compiler error?

Post by Prentice Bisbal
Update: removing the -fast switch caused this error to go away.
Prentice

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Jeff Squyres (jsquyres)

2017-05-01 18:20:28 UTC

Permalink

Er... right. Duh.

Jeff,
Why IBM? This problem is caused by the PGI compilers, so shouldn't this be directed towards NVidia, which now owns PGI?
Prentice

Post by Jeff Squyres (jsquyres)
IBM: can someone check to see if this is a compiler error?

Post by Prentice Bisbal
Update: removing the -fast switch caused this error to go away.
Prentice

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

--
Jeff Squyres
***@cisco.com

Prentice Bisbal

2017-05-01 18:49:59 UTC

Permalink

Jeff,

You probably were thrown off when I said I've only really seen this
problem when people didn't cross-compile correctly on the Blue Gene/P I
used to support. Also, PGI, and IBM both have 3 letters...;)

Prentice

Post by Jeff Squyres (jsquyres)
Er... right. Duh.

Jeff,
Why IBM? This problem is caused by the PGI compilers, so shouldn't this be directed towards NVidia, which now owns PGI?
Prentice

Post by Jeff Squyres (jsquyres)
IBM: can someone check to see if this is a compiler error?

Post by Prentice Bisbal
Update: removing the -fast switch caused this error to go away.
Prentice

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Sylvain Jeaugey

2017-05-01 18:58:19 UTC

Permalink

I also saw IBM and ignored the email :-)

Thanks for reporting the issue, I passed it to the PGI team.

Post by Prentice Bisbal
Jeff,
You probably were thrown off when I said I've only really seen this
problem when people didn't cross-compile correctly on the Blue Gene/P
I used to support. Also, PGI, and IBM both have 3 letters...;)
Prentice

Post by Jeff Squyres (jsquyres)
Er... right. Duh.

Post by Prentice Bisbal
Jeff,
Why IBM? This problem is caused by the PGI compilers, so shouldn't
this be directed towards NVidia, which now owns PGI?
Prentice

Post by Jeff Squyres (jsquyres)
IBM: can someone check to see if this is a compiler error?

Post by Prentice Bisbal
Update: removing the -fast switch caused this error to go away.
Prentice

Post by Prentice Bisbal
I'm building Open MPI 2.1.0 with PGI 17.3, and now I'm getting
../../config/test-driver: line 107: 65169 Illegal instruction
FAIL: atomic_math
- 1 threads: Passed
That's just one example of the error output. See all relevant error output below.
Usually, I see these errors when trying to run an executable on a
processor that doesn't support the instruction set of the
executable. I used to see this all the time when I supported an
IBM Blue Gene/P system. I don't think I've ever seen it on an x86
system.
I'm passing the argument '-tp=x64' to pgcc to build a unified
binary, so that might be part of the problem, but I've used this
exact same process to build 2.1.0 with PGI 16.5 just a couple
hours ago. I also built 1.10.3 with the same compiler flags with
PGI 16.5 and 17.3 without this error.
Any ideas?
make check-TESTS
make[3]: Entering directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[4]: Entering directory `/local/pbisbal/openmpi-2.1.0/test/asm'
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_barrier
- 1 threads: Passed
PASS: atomic_barrier
- 2 threads: Passed
PASS: atomic_barrier
- 4 threads: Passed
PASS: atomic_barrier
- 5 threads: Passed
PASS: atomic_barrier
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_barrier_noinline
- 1 threads: Passed
PASS: atomic_barrier_noinline
- 2 threads: Passed
PASS: atomic_barrier_noinline
- 4 threads: Passed
PASS: atomic_barrier_noinline
- 5 threads: Passed
PASS: atomic_barrier_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_spinlock
- 1 threads: Passed
PASS: atomic_spinlock
- 2 threads: Passed
PASS: atomic_spinlock
- 4 threads: Passed
PASS: atomic_spinlock
- 5 threads: Passed
PASS: atomic_spinlock
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_spinlock_noinline
- 1 threads: Passed
PASS: atomic_spinlock_noinline
- 2 threads: Passed
PASS: atomic_spinlock_noinline
- 4 threads: Passed
PASS: atomic_spinlock_noinline
- 5 threads: Passed
PASS: atomic_spinlock_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65169 Illegal instruction
FAIL: atomic_math
- 1 threads: Passed
../../config/test-driver: line 107: 65172 Illegal instruction
FAIL: atomic_math
- 2 threads: Passed
../../config/test-driver: line 107: 65176 Illegal instruction
FAIL: atomic_math
- 4 threads: Passed
../../config/test-driver: line 107: 65180 Illegal instruction
FAIL: atomic_math
- 5 threads: Passed
../../config/test-driver: line 107: 65185 Illegal instruction
FAIL: atomic_math
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65195 Illegal instruction
FAIL: atomic_math_noinline
- 1 threads: Passed
../../config/test-driver: line 107: 65198 Illegal instruction
FAIL: atomic_math_noinline
- 2 threads: Passed
../../config/test-driver: line 107: 65202 Illegal instruction
FAIL: atomic_math_noinline
- 4 threads: Passed
../../config/test-driver: line 107: 65206 Illegal instruction
FAIL: atomic_math_noinline
- 5 threads: Passed
../../config/test-driver: line 107: 65210 Illegal instruction
FAIL: atomic_math_noinline
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65220 Illegal instruction
FAIL: atomic_cmpset
- 1 threads: Passed
../../config/test-driver: line 107: 65223 Illegal instruction
FAIL: atomic_cmpset
- 2 threads: Passed
../../config/test-driver: line 107: 65227 Illegal instruction
FAIL: atomic_cmpset
- 4 threads: Passed
../../config/test-driver: line 107: 65231 Illegal instruction
FAIL: atomic_cmpset
- 5 threads: Passed
../../config/test-driver: line 107: 65235 Illegal instruction
FAIL: atomic_cmpset
- 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65245 Illegal instruction
FAIL: atomic_cmpset_noinline
- 1 threads: Passed
../../config/test-driver: line 107: 65248 Illegal instruction
FAIL: atomic_cmpset_noinline
- 2 threads: Passed
../../config/test-driver: line 107: 65252 Illegal instruction
FAIL: atomic_cmpset_noinline
- 4 threads: Passed
../../config/test-driver: line 107: 65256 Illegal instruction
FAIL: atomic_cmpset_noinline
- 5 threads: Passed
../../config/test-driver: line 107: 65260 Illegal instruction
FAIL: atomic_cmpset_noinline
- 8 threads: Passed
============================================================================
Testsuite summary for Open MPI 2.1.0
============================================================================
# TOTAL: 8
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 4
# XPASS: 0
# ERROR: 0
============================================================================
See test/asm/test-suite.log
Please report to http://www.open-mpi.org/community/help/
============================================================================
make[4]: *** [test-suite.log] Error 1
make[4]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[3]: *** [check-TESTS] Error 2
make[3]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test'
make: *** [check-recursive] Error 1
$ cat test/asm/test-suite.log
=============================================
Open MPI 2.1.0: test/asm/test-suite.log
=============================================
# TOTAL: 8
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 4
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: atomic_math
=================
FAIL atomic_math (exit status: 132)
FAIL: atomic_math_noinline
==========================
FAIL atomic_math_noinline (exit status: 132)
FAIL: atomic_cmpset
===================
FAIL atomic_cmpset (exit status: 132)
FAIL: atomic_cmpset_noinline
============================
FAIL atomic_cmpset_noinline (exit status: 132)
./configure \
--prefix=/usr/pppl/pgi/17.3-pkgs/openmpi-2.1.0 \
--disable-silent-rules \
--enable-mpi-fortran \
--enable-mpi-cxx \
--enable-shared \
--enable-static \
--enable-mpi-thread-multiple \
--with-cuda=/usr/pppl/cuda/cudatoolkit/6.5.14 \
--with-pmix \
--with-verbs \
--with-hwloc \
--with-pmi=/usr/pppl/slurm/15.08.8 \
--with-slurm \
--with-psm \
CPP=cpp \
CC=pgcc \
CFLAGS="-fpic -tp=x64 -fast" \
CXX=pgc++ \
CXXFLAGS="-fpic -tp=x64 -fast" \
FC=pgfortran \
FCFLAGS="-fpic -tp=x64 -fast" \
2>&1 | tee configure.log

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------

Prentice Bisbal via users

2018-11-28 15:01:44 UTC

Permalink

Sylvain,

I just ran into the same exact errors when compiling OpenMPI 3.0.0 with
PGI 18.3

Prentice

Post by Sylvain Jeaugey
I also saw IBM and ignored the email :-)
Thanks for reporting the issue, I passed it to the PGI team.

Er... right. Duh.

Post by Prentice Bisbal
Jeff,
Why IBM? This problem is caused by the PGI compilers, so shouldn't
this be directed towards NVidia, which now owns PGI?
Prentice

Post by Jeff Squyres (jsquyres)
IBM: can someone check to see if this is a compiler error?

Post by Prentice Bisbal
Update: removing the -fast switch caused this error to go away.
Prentice

Post by Prentice Bisbal
I'm building Open MPI 2.1.0 with PGI 17.3, and now I'm getting
../../config/test-driver: line 107: 65169 Illegal instruction
FAIL: atomic_math
    - 1 threads: Passed
That's just one example of the error output. See all relevant
error output below.
Usually, I see these errors when trying to run an executable on
a processor that doesn't support the instruction set of the
executable. I used to see this all the time when I supported an
IBM Blue Gene/P system. I don't think I've ever seen it on an
x86 system.
I'm passing the argument '-tp=x64' to pgcc to build a unified
binary, so that might be part of the problem, but I've used this
exact same process to build 2.1.0 with PGI 16.5 just a couple
hours ago. I also built 1.10.3 with the same compiler flags with
PGI 16.5 and 17.3 without this error.
Any ideas?
make check-TESTS
make[3]: Entering directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[4]: Entering directory `/local/pbisbal/openmpi-2.1.0/test/asm'
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_barrier
    - 1 threads: Passed
PASS: atomic_barrier
    - 2 threads: Passed
PASS: atomic_barrier
    - 4 threads: Passed
PASS: atomic_barrier
    - 5 threads: Passed
PASS: atomic_barrier
    - 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_barrier_noinline
    - 1 threads: Passed
PASS: atomic_barrier_noinline
    - 2 threads: Passed
PASS: atomic_barrier_noinline
    - 4 threads: Passed
PASS: atomic_barrier_noinline
    - 5 threads: Passed
PASS: atomic_barrier_noinline
    - 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_spinlock
    - 1 threads: Passed
PASS: atomic_spinlock
    - 2 threads: Passed
PASS: atomic_spinlock
    - 4 threads: Passed
PASS: atomic_spinlock
    - 5 threads: Passed
PASS: atomic_spinlock
    - 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
PASS: atomic_spinlock_noinline
    - 1 threads: Passed
PASS: atomic_spinlock_noinline
    - 2 threads: Passed
PASS: atomic_spinlock_noinline
    - 4 threads: Passed
PASS: atomic_spinlock_noinline
    - 5 threads: Passed
PASS: atomic_spinlock_noinline
    - 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65169 Illegal instruction
FAIL: atomic_math
    - 1 threads: Passed
../../config/test-driver: line 107: 65172 Illegal instruction
FAIL: atomic_math
    - 2 threads: Passed
../../config/test-driver: line 107: 65176 Illegal instruction
FAIL: atomic_math
    - 4 threads: Passed
../../config/test-driver: line 107: 65180 Illegal instruction
FAIL: atomic_math
    - 5 threads: Passed
../../config/test-driver: line 107: 65185 Illegal instruction
FAIL: atomic_math
    - 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65195 Illegal instruction
FAIL: atomic_math_noinline
    - 1 threads: Passed
../../config/test-driver: line 107: 65198 Illegal instruction
FAIL: atomic_math_noinline
    - 2 threads: Passed
../../config/test-driver: line 107: 65202 Illegal instruction
FAIL: atomic_math_noinline
    - 4 threads: Passed
../../config/test-driver: line 107: 65206 Illegal instruction
FAIL: atomic_math_noinline
    - 5 threads: Passed
../../config/test-driver: line 107: 65210 Illegal instruction
FAIL: atomic_math_noinline
    - 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65220 Illegal instruction
FAIL: atomic_cmpset
    - 1 threads: Passed
../../config/test-driver: line 107: 65223 Illegal instruction
FAIL: atomic_cmpset
    - 2 threads: Passed
../../config/test-driver: line 107: 65227 Illegal instruction
FAIL: atomic_cmpset
    - 4 threads: Passed
../../config/test-driver: line 107: 65231 Illegal instruction
FAIL: atomic_cmpset
    - 5 threads: Passed
../../config/test-driver: line 107: 65235 Illegal instruction
FAIL: atomic_cmpset
    - 8 threads: Passed
basename: extra operand `--test-name'
Try `basename --help' for more information.
--> Testing
../../config/test-driver: line 107: 65245 Illegal instruction
FAIL: atomic_cmpset_noinline
    - 1 threads: Passed
../../config/test-driver: line 107: 65248 Illegal instruction
FAIL: atomic_cmpset_noinline
    - 2 threads: Passed
../../config/test-driver: line 107: 65252 Illegal instruction
FAIL: atomic_cmpset_noinline
    - 4 threads: Passed
../../config/test-driver: line 107: 65256 Illegal instruction
FAIL: atomic_cmpset_noinline
    - 5 threads: Passed
../../config/test-driver: line 107: 65260 Illegal instruction
FAIL: atomic_cmpset_noinline
    - 8 threads: Passed
============================================================================
Testsuite summary for Open MPI 2.1.0
============================================================================
# TOTAL: 8
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 4
# XPASS: 0
# ERROR: 0
============================================================================
See test/asm/test-suite.log
Please report to http://www.open-mpi.org/community/help/
============================================================================
make[4]: *** [test-suite.log] Error 1
make[4]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[3]: *** [check-TESTS] Error 2
make[3]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[2]: *** [check-am] Error 2
make[2]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test/asm'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/local/pbisbal/openmpi-2.1.0/test'
make: *** [check-recursive] Error 1
$ cat test/asm/test-suite.log
=============================================
   Open MPI 2.1.0: test/asm/test-suite.log
=============================================
# TOTAL: 8
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 4
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: atomic_math
=================
FAIL atomic_math (exit status: 132)
FAIL: atomic_math_noinline
==========================
FAIL atomic_math_noinline (exit status: 132)
FAIL: atomic_cmpset
===================
FAIL atomic_cmpset (exit status: 132)
FAIL: atomic_cmpset_noinline
============================
FAIL atomic_cmpset_noinline (exit status: 132)
./configure \
--prefix=/usr/pppl/pgi/17.3-pkgs/openmpi-2.1.0 \
--disable-silent-rules \
--enable-mpi-fortran \
--enable-mpi-cxx \
--enable-shared \
--enable-static \
--enable-mpi-thread-multiple \
--with-cuda=/usr/pppl/cuda/cudatoolkit/6.5.14 \
--with-pmix \
--with-verbs \
--with-hwloc \
--with-pmi=/usr/pppl/slurm/15.08.8 \
--with-slurm \
--with-psm \
CPP=cpp \
CC=pgcc \
CFLAGS="-fpic -tp=x64 -fast" \
CXX=pgc++ \
CXXFLAGS="-fpic -tp=x64 -fast" \
FC=pgfortran \
FCFLAGS="-fpic -tp=x64 -fast" \
2>&1 | tee configure.log

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or
distribution
is prohibited. If you are not the intended recipient, please contact
the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Jeff Squyres (jsquyres)

2017-05-01 18:20:47 UTC

Permalink

What I should have said was: NVIDIA -- can someone check to see if this is a PGI compiler error?

Post by Jeff Squyres (jsquyres)
IBM: can someone check to see if this is a compiler error?

Post by Prentice Bisbal
Update: removing the -fast switch caused this error to go away.
Prentice

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

--
Jeff Squyres
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

--
Jeff Squyres
***@cisco.com