[OMPI users] Fortran and MPI-3 shared memory

[Tom Rosmond]

All:

I am trying to understand the use of the shared memory features of MPI-3
that allow direct sharing of the memory space of on-node processes.
Attached are 2 small test programs, one written in C (testmpi3.c), the
other F95 (testmpi3.f90) . They are solving the identical 'halo'
exchange problem. 'testmpi3.c' is a simplified version of an example
program from a presentation by Mark Lubin of Intel. I wrote
'testmpi3.f90' to mimic the C version.

Also attached are 2 text files of the compile, execution, and output
of the respective programs:

CC_testmpi3.txt
F95_testmpi3.txt

Note: All 4 files are contained in the attached 'testmpi3.tar.gz'.

Comparing the outputs of each version, it is clear that the shared
memory copies in 'testmpi3.c' are working correctly, but not in
'testmpi3.f90'. As far as I can tell, the 2 programs are equivalent up
to line 134 of 'testmpi3.c' and lines 97-101 of 'testmpi3.f90'. I
thought the calls to 'c_f_pointer' would produce Fortran pointers that
would access the correct shared memory addresses as the C-pointers do in
'testmpi3.c', but clearly that isn't happening. Can anyone explain why
not, and what is needed to make this happen. Any suggestions are welcome.

My environment:
Scientific Linux 6.8
INTEL FORTRAN and ICC version 15.0.2.164
OPEN-MPI 2.0.1


T. Rosmond

[Jeff Hammond]

I can reproduce this with Intel 17 and MPICH on Mac OSX so it's not an
Open-MPI issue. I added VOLATILE to the shared memory objects to prevent
Fortran compiler optimizations as well as a bunch of MPI_Win_sync calls
(after replacing fence with lock_all/unlock_all), but neither changed the
outcome.

While I claim a modest understanding of MPI-3 RMA and Fortran 2008,
unfortunately, I have never figured out how to use MPI-3 shared memory from
Fortran, which is especially unfortunate since it seems to be a fantastic
source of frustration to both real users such as yourself and MPI+Fortran
standard experts (Rolf).

Sorry for the unsatisfying response, but my suspicion is that this is a
program correctness issue. I can't point to any error, but I've ruled out
the obvious alternatives.

Jeff

Post by Tom Rosmond
I am trying to understand the use of the shared memory features of MPI-3
that allow direct sharing of the memory space of on-node processes.
Attached are 2 small test programs, one written in C (testmpi3.c), the
other F95 (testmpi3.f90) . They are solving the identical 'halo' exchange
problem. 'testmpi3.c' is a simplified version of an example program from a
presentation by Mark Lubin of Intel. I wrote 'testmpi3.f90' to mimic the C
version.
Also attached are 2 text files of the compile, execution, and output of
CC_testmpi3.txt
F95_testmpi3.txt
Note: All 4 files are contained in the attached 'testmpi3.tar.gz'.
Comparing the outputs of each version, it is clear that the shared memory
copies in 'testmpi3.c' are working correctly, but not in 'testmpi3.f90'.
As far as I can tell, the 2 programs are equivalent up to line 134 of
'testmpi3.c' and lines 97-101 of 'testmpi3.f90'. I thought the calls to
'c_f_pointer' would produce Fortran pointers that would access the correct
shared memory addresses as the C-pointers do in 'testmpi3.c', but clearly
that isn't happening. Can anyone explain why not, and what is needed to
make this happen. Any suggestions are welcome.
Scientific Linux 6.8
INTEL FORTRAN and ICC version 15.0.2.164
OPEN-MPI 2.0.1
T. Rosmond
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Tom Rosmond]

Jeff,

Thanks for looking at this. I know it isn't specific to Open-MPI, but
it is a frustrating issue vis-a-vis MPI and Fortran. There are many
very large MPI applications around the world written in Fortran that
could benefit greatly from this MPI-3 capability. My own background is
in numerical weather prediction, and I know it would be welcome in that
community. Someone knowledgeable in both C and Fortran should be able
to get to bottom of it.

T. Rosmond

Post by Jeff Hammond
I can reproduce this with Intel 17 and MPICH on Mac OSX so it's not an
Open-MPI issue. I added VOLATILE to the shared memory objects to
prevent Fortran compiler optimizations as well as a bunch of
MPI_Win_sync calls (after replacing fence with lock_all/unlock_all),
but neither changed the outcome.
While I claim a modest understanding of MPI-3 RMA and Fortran 2008,
unfortunately, I have never figured out how to use MPI-3 shared memory
from Fortran, which is especially unfortunate since it seems to be a
fantastic source of frustration to both real users such as yourself
and MPI+Fortran standard experts (Rolf).
Sorry for the unsatisfying response, but my suspicion is that this is
a program correctness issue. I can't point to any error, but I've
ruled out the obvious alternatives.
Jeff
I am trying to understand the use of the shared memory features of
MPI-3 that allow direct sharing of the memory space of on-node
processes. Attached are 2 small test programs, one written in C
(testmpi3.c), the other F95 (testmpi3.f90) . They are solving the
identical 'halo' exchange problem. 'testmpi3.c' is a simplified
version of an example program from a presentation by Mark Lubin of
Intel. I wrote 'testmpi3.f90' to mimic the C version.
Also attached are 2 text files of the compile, execution, and
CC_testmpi3.txt
F95_testmpi3.txt
Note: All 4 files are contained in the attached 'testmpi3.tar.gz'.
Comparing the outputs of each version, it is clear that the shared
memory copies in 'testmpi3.c' are working correctly, but not in
'testmpi3.f90'. As far as I can tell, the 2 programs are
equivalent up to line 134 of 'testmpi3.c' and lines 97-101 of
'testmpi3.f90'. I thought the calls to 'c_f_pointer' would produce
Fortran pointers that would access the correct shared memory
addresses as the C-pointers do in 'testmpi3.c', but clearly that
isn't happening. Can anyone explain why not, and what is needed to
make this happen. Any suggestions are welcome.
Scientific Linux 6.8
INTEL FORTRAN and ICC version 15.0.2.164
OPEN-MPI 2.0.1
T. Rosmond
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
<https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
--
Jeff Hammond
http://jeffhammond.github.io/
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Bader, Reinhold]

I've had a look at the OpenMPI 1.10.3 sources, and the trouble appears to me to be that the MPI wrappers declare
the argument

TYPE(C_PTR), INTENT(OUT) :: baseptr

inside the BIND(C) interface on the Fortran side (for OpenMPI this would, for example be ompi_win_allocate_f), and
the C implementation has "char *baseptr" as parameter. This is incorrect, it needs to be

char **baseptr

I believe this actually points to a design weakness of the C-side interface. For establishment of a C pointer, usually
functions that return such a pointer are defined (e.g. malloc()). If a formal parameter is desired, this should be
a pointer-to-pointer entity, i.e. it would have been better to define the MPI_Win_allocate prototype as

int MPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info, MPI_comm comm, void **baseptr, MPI_Win win);

Note that the C example pretends that this is the case - it passes the address of the pointer to
MPI_Win_allocate, not the pointer itself.

Cheers


--
Dr. Reinhold Bader
Leibniz Supercomputing Centre (http://www.lrz.de) / HPC Systems and Services
Tel.: [phone] Fax: 8625

[Gilles Gouaillardet]

From the MPI 3.1 standard (page 338)

Rationale. The C bindings of MPI_ALLOC_MEM and MPI_FREE_MEM are similar
to the bindings for the malloc and free C library calls: a call to
MPI_Alloc_mem(: : :, &base) should be paired with a call to
MPI_Free_mem(base) (one
less level of indirection). Both arguments are declared to be of same type
void* so as to facilitate type casting. The Fortran binding is
consistent with the C
bindings: the Fortran MPI_ALLOC_MEM call returns in baseptr the
TYPE(C_PTR)
pointer or the (integer valued) address of the allocated memory. The
base argument
of MPI_FREE_MEM is a choice argument, which passes (a reference to)
the variable
stored at that location. (End of rationale.)

at first glance, i also thought type should have been char ** instead of
char *, but it is not a typo nor a mistake


Cheers,


Gilles

I've had a look at the OpenMPI 1.10.3 sources, and the trouble appears
to me to be that the MPI wrappers declare
the argument
TYPE(C_PTR), INTENT(OUT) :: baseptr
inside the BIND(C) interface on the Fortran side (for OpenMPI this
would, for example be ompi_win_allocate_f), and
the C implementation has "char *baseptr" as parameter. This is
incorrect, it needs to be
char **baseptr
I believe this actually points to a design weakness of the C-side
interface. For establishment of a C pointer, usually
functions that return such a pointer are defined (e.g. malloc()). If a
formal parameter is desired, this should be
a pointer-to-pointer entity, i.e. it would have been better to define
the MPI_Win_allocate prototype as
int MPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info,
MPI_comm comm, *void **baseptr*, MPI_Win win);
Note that the C example pretends that this is the case – it passes the
address of the pointer to
MPI_Win_allocate, not the pointer itself.
Cheers
--
Dr. Reinhold Bader
Leibniz Supercomputing Centre (http://www.lrz.de) / HPC Systems and Services
Tel.: [phone] Fax: 8625
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Bader, Reinhold]

Rationale. The C bindings of MPI_ALLOC_MEM and MPI_FREE_MEM are similar
to the bindings for the malloc and free C library calls: a call to
MPI_Alloc_mem(: : :, &base) should be paired with a call to MPI_Free_mem(base) (one
less level of indirection). Both arguments are declared to be of same type
void* so as to facilitate type casting. The Fortran binding is consistent with the C bindings: the Fortran MPI_ALLOC_MEM call returns in baseptr the
TYPE(C_PTR) pointer or the (integer valued) address of the allocated memory. The base argument of MPI_FREE_MEM is a choice argument, which passes (a reference
to) the variable
stored at that location. (End of rationale.)

Understood - this makes clear the MPI interface convention for the C side. However, the wording for the Fortran bindings is at best misleading.
If full consistency with the C interface were desired, the baseptr argument would need to be declared

TYPE(C_PTR), VALUE :: baseptr

and then users would be required to call the procedure via

CALL MPI_XXX( ..., c_loc(baseptr), ...)

With the signatures as currently defined, I stand by my opinion that the internal wrappers need to resolve the * vs ** issue.
See Fortran 2008, 15.3.7 para 2, item (5) (semantics of a dummy argument *without* the VALUE attribute).

at first glance, i also thought type should have been char ** instead of char *, but it is not a typo nor a mistake
Cheers,
Gilles

--
Dr. Reinhold Bader
Leibniz Supercomputing Centre (http://www.lrz.de) / HPC Systems and Services
Tel.: [phone] Fax: 8625

[Gilles Gouaillardet]

Tom,


regardless the (lack of) memory model in Fortran, there is an error in
testmpi3.f90

shar_mem is declared as an integer, and hence is not in the shared memory.

i attached my version of testmpi3.f90, which behaves just like the C
version,

at least when compiled with -g -O0 and with Open MPI master

/* i replaced shar_mem with fptr_mem */


Cheers,


Gilles

Post by Tom Rosmond
I am trying to understand the use of the shared memory features of
MPI-3 that allow direct sharing of the memory space of on-node
processes. Attached are 2 small test programs, one written in C
(testmpi3.c), the other F95 (testmpi3.f90) . They are solving the
identical 'halo' exchange problem. 'testmpi3.c' is a simplified
version of an example program from a presentation by Mark Lubin of
Intel. I wrote 'testmpi3.f90' to mimic the C version.
Also attached are 2 text files of the compile, execution, and output
CC_testmpi3.txt
F95_testmpi3.txt
Note: All 4 files are contained in the attached 'testmpi3.tar.gz'.
Comparing the outputs of each version, it is clear that the shared
memory copies in 'testmpi3.c' are working correctly, but not in
'testmpi3.f90'. As far as I can tell, the 2 programs are equivalent
up to line 134 of 'testmpi3.c' and lines 97-101 of 'testmpi3.f90'. I
thought the calls to 'c_f_pointer' would produce Fortran pointers that
would access the correct shared memory addresses as the C-pointers do
in 'testmpi3.c', but clearly that isn't happening. Can anyone explain
why not, and what is needed to make this happen. Any suggestions are
welcome.
Scientific Linux 6.8
INTEL FORTRAN and ICC version 15.0.2.164
OPEN-MPI 2.0.1
T. Rosmond
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[Tom Rosmond]

Gilles,

Thanks! With my very rudimentary understanding of C pointers and C
programming in general I missed that translation subtly. The revised
program runs fine with a variety of optimizations and debug options on
my test system.

Tom R.

Post by Gilles Gouaillardet
Tom,
regardless the (lack of) memory model in Fortran, there is an error in
testmpi3.f90
shar_mem is declared as an integer, and hence is not in the shared memory.
i attached my version of testmpi3.f90, which behaves just like the C
version,
at least when compiled with -g -O0 and with Open MPI master
/* i replaced shar_mem with fptr_mem */
Cheers,
Gilles

Post by Tom Rosmond
I am trying to understand the use of the shared memory features of
MPI-3 that allow direct sharing of the memory space of on-node
processes. Attached are 2 small test programs, one written in C
(testmpi3.c), the other F95 (testmpi3.f90) . They are solving the
identical 'halo' exchange problem. 'testmpi3.c' is a simplified
version of an example program from a presentation by Mark Lubin of
Intel. I wrote 'testmpi3.f90' to mimic the C version.
Also attached are 2 text files of the compile, execution, and output
CC_testmpi3.txt
F95_testmpi3.txt
Note: All 4 files are contained in the attached 'testmpi3.tar.gz'.
Comparing the outputs of each version, it is clear that the shared
memory copies in 'testmpi3.c' are working correctly, but not in
'testmpi3.f90'. As far as I can tell, the 2 programs are equivalent
up to line 134 of 'testmpi3.c' and lines 97-101 of 'testmpi3.f90'. I
thought the calls to 'c_f_pointer' would produce Fortran pointers
that would access the correct shared memory addresses as the
C-pointers do in 'testmpi3.c', but clearly that isn't happening. Can
anyone explain why not, and what is needed to make this happen. Any
suggestions are welcome.
Scientific Linux 6.8
INTEL FORTRAN and ICC version 15.0.2.164
OPEN-MPI 2.0.1
T. Rosmond
_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

_______________________________________________
users mailing list
https://rfd.newmexicoconsortium.org/mailman/listinfo/users