[OMPI users] Issues with PGI 16.10, OpenMPI 2.1.0 on macOS: Fortran issues with hello world (running and dylib)

[Matt Thompson]

All,

Aside from the issues I'm having with pgc++ on macOS, I'm also having an
issue not related to that configure issue.

Namely, if I build Open MPI using the siterc figured out by PGI Help:

http://www.pgroup.com/userforum/viewtopic.php?p=21105#21105

note I used the siterc, but I did not enable-mpi-cxx for this:

./configure --disable-wrapper-rpath CC=pgcc CXX=pgc++ FC=pgfortran
FCFLAGS='-Mpreprocess -Mbackslash' --prefix=...

I can successfully configure and make Open MPI 2.1.0. However, it can't
seem to do things that Open MPI + other compilers can as I've figured out
trying to build various libraries like ESMF, etc.

For example, I can't seem to make a dynamic lib with PGI, yet I can with
gfortran. To wit, using Open MPI 2.1.0 and GCC 6.3.0 (built from the source
tarball from GNU):

(975) $ mpifort --version

GNU Fortran (GCC) 6.3.0
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
(976) $ lt
total 4.0K
-rw-r--r-- 1 mathomp4 1286109195 1.1K Mar 22 14:25 hello_usempif08.f90
drwxr-xr-x 4 mathomp4 1286109195 136 Mar 24 08:46 Simple/
(977) $ mpifort --version
GNU Fortran (GCC) 6.3.0
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
(978) $ mpifort -c hello_usempif08.f90
(979) $ mpifort -dynamiclib -o hello_usempif08.dylib hello_usempif08.o
(980) $ ls -ltr hello_usempif08.*
-rw-r--r-- 1 mathomp4 1286109195 1031 Mar 22 14:25 hello_usempif08.f90
-rw-r--r-- 1 mathomp4 1286109195 1968 Mar 24 08:47 hello_usempif08.o
-rwxr-xr-x 1 mathomp4 1286109195 9308 Mar 24 08:47 hello_usempif08.dylib*

Now for Open MPI 2.1.0 and PGI 16.10:

(827) $ mpifort --version

pgfortran 16.10-0 64-bit target on Apple OS/X -tp sandybridge
The Portland Group - PGI Compilers and Tools
Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
(828) $ mpifort -c hello_usempif08.f90
(829) $ mpifort -dynamiclib -o hello_usempif08.dylib hello_usempif08.o
unknown option character `o' in: -commons
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool
-static [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch]
[-sacLT] [-no_warning_for_no_symbols]
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool
-dynamic [-] file [...] [-filelist listfile[,dirname]] [-arch_only arch]
[-o output] [-install_name name] [-compatibility_version #]
[-current_version #] [-seg1addr 0x#] [-segs_read_only_addr 0x#]
[-segs_read_write_addr 0x#] [-seg_addr_table <filename>]
[-seg_addr_table_filename <file_system_path>] [-all_load] [-noall_load]

Looking at showme, they are roughly the same:

(981) $ mpifort -showme

gfortran
-I/Users/mathomp4/installed/Compiler/gcc-gfortran-6.3.0/openmpi/2.1.0/include
-Wl,-flat_namespace -Wl,-commons,use_dylibs
-I/Users/mathomp4/installed/Compiler/gcc-gfortran-6.3.0/openmpi/2.1.0/lib
-L/Users/mathomp4/installed/Compiler/gcc-gfortran-6.3.0/openmpi/2.1.0/lib
-lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi

(830) $ mpifort -showme

pgfortran
-I/Users/mathomp4/installed/Compiler/pgi-16.10/openmpi/2.1.0/include
-Wl,-flat_namespace -Wl,-commons,use_dylibs
-I/Users/mathomp4/installed/Compiler/pgi-16.10/openmpi/2.1.0/lib
-L/Users/mathomp4/installed/Compiler/pgi-16.10/openmpi/2.1.0/lib
-lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lopen-rte
-lopen-pal -lm

My first thought was maybe the all-GNU was avoiding something (like using
GNU libtool instead of Apple's?). So, I built Open MPI 2.1.0 using built-in
clang as the C compiler and gfortran from
https://gcc.gnu.org/wiki/GFortranBinaries (6.1 as I'm on El Capitan), but
doing that works just fine like the all-GNU and I can't imagine it would
use the GNU libtool (though...maybe? somehow? Any way to figure this out?)

My other test was, hey, let's just try *running* hello world. All GNU 6.3
(and the clang/gfortran 6.1 looks similar):

(982) $ mpifort -o hello_usempif08.exe hello_usempif08.f90

(983) $ mpirun -np 4 ./hello_usempif08.exe
Hello, world, I am 1 of 4: Open MPI v2.1.0, package: Open MPI
v2.0.1-696-g1cd1edf, Mar 20, 2017
Hello, world, I am 3 of 4: Open MPI v2.1.0, package: Open MPI
v2.0.1-696-g1cd1edf, Mar 20, 2017
Hello, world, I am 0 of 4: Open MPI v2.1.0, package: Open MPI
v2.0.1-696-g1cd1edf, Mar 20, 2017
Hello, world, I am 2 of 4: Open MPI v2.1.0, package: Open MPI
v2.0.1-696-g1cd1edf, Mar 20, 2017

PGI 16.10:

(831) $ mpifort -o hello_usempif08.exe hello_usempif08.f90

(832) $ mpirun -np 4 ./hello_usempif08.exe
[computer.concealed:87920] mca_base_component_repository_open: unable to
open mca_patcher_overwrite: File not found (ignored)
[computer.concealed:87920] mca_base_component_repository_open: unable to
open mca_shmem_mmap: File not found (ignored)
[computer.concealed:87920] mca_base_component_repository_open: unable to
open mca_shmem_posix: File not found (ignored)
[computer.concealed:87920] mca_base_component_repository_open: unable to
open mca_shmem_sysv: File not found (ignored)
--------------------------------------------------------------------------
It looks like opal_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
opal_shmem_base_select failed
--> Returned value -1 instead of OPAL_SUCCESS
--------------------------------------------------------------------------

Well okay then. I didn't see any errors during the build for PGI:

(835) $ grep Error make.pgi-16.10.log

SED mpi/man/man3/MPI_Error_class.3
SED mpi/man/man3/MPI_Error_string.3

Any ideas/help?