[OMPI users] Need help to run hybrid code.

[Pasha Pashaei]

Dear friends

I am going to run a hybrid MPI+OPENMP code.
As you can see in the below I played with the number of threads in various cases while running my main code with (Openmpi,Mpich,Mpich2).
As you can see in OpenMPi and Mpich it seems that openmp did not work at all as Total time did not change considerably. But in Mpich2 Total computational time increased with increasing the number of threads. It could be because of using virtual threads instead of physical threads(or you said that over-subscribing).


Hybrid code result (MPI + OpenMP) :
Your suggestions:
mvapich
mpirun -np 4 -genv OMP_NUM_THREADS 1 --bind-to hwthread:1 ./pjet.gfortran > output.txt
Total time = 7.290E+02
mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 ./pjet.gfortran > output.txt
Total time = 4.940E+02
mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 -map-by hwthread:8 ./pjet.gfortran > output.txt
Total time = 4.960E+02
mpirun -np 4 -genv OMP_NUM_THREADS 16 --bind-to hwthread:16 ./pjet.gfortran > output.txt
Total time = 4.502E+02
mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to core:16 -map-by core:16 ./pjet.gfortran > output.txt
Total time = 4.628E+02

Pervios commands
OpenMPI 1.8.1
mpirun -np 4 -x OMP_NUM_THREADS=1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.475E+02
mpirun -np 4 -x OMP_NUM_THREADS=8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.525E+02
mpirun -np 4 -x OMP_NUM_THREADS=16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.611E+02

mvapich
mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.441E+02
mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.535E+02
mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.552E+02
mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.591E+02

mvapich2
mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 4.935E+02
mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 5.562E+02
mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 6.392E+02
mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt
Total time = 8.170E+02

Then I used a simple "hybrid.f90" code and its result which I used to check whether the computer can recognize correct value of cores and threads or not. It showed that the correct values in all three in different cases.
here is its result:

Starting omp_dotprod_hybrid. Using 4 Cores...
Core 3 using 16 threads
Core 0 using 16 threads
Core 2 using 16 threads
Core 1 using 16 threads
Core 1 thread 0 partial sum = 0.0000000000000000
Core 3 thread 0 partial sum = 0.0000000000000000
Core 1 thread 4 partial sum = 0.0000000000000000
Core 3 thread 7 partial sum = 200.00000000000000
Core 1 thread 8 partial sum = 200.00000000000000
Core 3 thread 9 partial sum = 200.00000000000000
Core 1 thread 11 partial sum = 200.00000000000000
Core 3 thread 3 partial sum = 200.00000000000000
Core 1 thread 2 partial sum = 0.0000000000000000
Core 3 thread 5 partial sum = 0.0000000000000000
Core 1 thread 3 partial sum = 200.00000000000000
Core 3 thread 2 partial sum = 0.0000000000000000
Core 1 thread 13 partial sum = 200.00000000000000
Core 3 thread 12 partial sum = 200.00000000000000
Core 1 thread 1 partial sum = 200.00000000000000
Core 3 thread 1 partial sum = 200.00000000000000
Core 3 thread 8 partial sum = 0.0000000000000000
Core 1 thread 7 partial sum = 0.0000000000000000
Core 3 thread 11 partial sum = 200.00000000000000
Core 1 thread 15 partial sum = 0.0000000000000000
Core 3 thread 15 partial sum = 0.0000000000000000
Core 1 thread 10 partial sum = 200.00000000000000
Core 1 thread 9 partial sum = 200.00000000000000
Core 3 thread 13 partial sum = 0.0000000000000000
Core 1 thread 5 partial sum = 200.00000000000000
Core 3 thread 6 partial sum = 0.0000000000000000
Core 3 thread 4 partial sum = 0.0000000000000000
Core 1 thread 6 partial sum = 0.0000000000000000
Core 3 thread 14 partial sum = 200.00000000000000
Core 1 thread 12 partial sum = 0.0000000000000000
Core 3 thread 10 partial sum = 200.00000000000000
Core 0 thread 0 partial sum = 0.0000000000000000
Core 0 thread 14 partial sum = 200.00000000000000
Core 0 thread 8 partial sum = 200.00000000000000
Core 0 thread 7 partial sum = 0.0000000000000000
Core 0 thread 15 partial sum = 200.00000000000000
Core 0 thread 5 partial sum = 200.00000000000000
Core 0 thread 9 partial sum = 200.00000000000000
Core 0 thread 11 partial sum = 0.0000000000000000
Core 0 thread 10 partial sum = 200.00000000000000
Core 0 thread 6 partial sum = 200.00000000000000
Core 0 thread 3 partial sum = 200.00000000000000
Core 0 thread 4 partial sum = 0.0000000000000000
Core 0 thread 2 partial sum = 0.0000000000000000
Core 0 thread 13 partial sum = 0.0000000000000000
Core 0 thread 12 partial sum = 0.0000000000000000
Core 0 thread 1 partial sum = 0.0000000000000000
Core 0 partial sum = 1600.0000000000000
Core 2 thread 3 partial sum = 0.0000000000000000
Core 2 thread 15 partial sum = 0.0000000000000000
Core 2 thread 0 partial sum = 0.0000000000000000
Core 2 thread 2 partial sum = 200.00000000000000
Core 2 thread 4 partial sum = 0.0000000000000000
Core 2 thread 5 partial sum = 0.0000000000000000
Core 2 thread 9 partial sum = 200.00000000000000
Core 2 thread 7 partial sum = 0.0000000000000000
Core 2 thread 14 partial sum = 200.00000000000000
Core 2 thread 8 partial sum = 200.00000000000000
Core 2 thread 12 partial sum = 200.00000000000000
Core 2 thread 10 partial sum = 200.00000000000000
Core 2 thread 6 partial sum = 200.00000000000000
Core 2 thread 1 partial sum = 0.0000000000000000
Core 2 thread 13 partial sum = 0.0000000000000000
Core 2 thread 11 partial sum = 200.00000000000000
Core 2 partial sum = 1600.0000000000000
Core 3 partial sum = 1600.0000000000000
Core 1 thread 14 partial sum = 0.0000000000000000
Core 1 partial sum = 1600.0000000000000
Done. Hybrid version: global sum = 6400.0000000000000



Please tell me If I should check something. I still getting nowhere.
Best regards

Pasha Pashaei