Problem with MPI_Status conversion when C has 4-byte ints and Fortran has 8-byte INTEGERs

[ahaichen]

Open MPI version: 4.0.1

• installed from source
• build with: ./configure F77=gfortran FC=gfortran FFLAGS=-fdefault-integer-8 FCFLAGS=-fdefault-integer-8
• Operating system: Linux version 4.9.0-12-amd64 (debian-kernel@lists.debian.org)
• CPU: Intel(R) Xeon(R)

Hi all,

I get an issue when running in 64-bit MPI with the rank initialized 0, the code is

Program test
USE MPI_F08
IMPLICIT NONE

TYPE(MPI_Status)                            :: MPI_stat
Integer(kind=MPI_INTEGER_KIND) :: MPI_err
Integer(kind=MPI_INTEGER_KIND) :: MPI_id=0
Integer(kind=MPI_INTEGER_KIND) :: MPI_np
Integer                                              :: tempint,i_MPI

CALL MPI_Init(MPI_err)
CALL MPI_Comm_rank(MPI_COMM_WORLD,MPI_id,MPI_err)
CALL MPI_Comm_size(MPI_COMM_WORLD,MPI_np,MPI_err)

if(MPI_id==0) tempint=1
if(MPI_id/=0)  tempint=-1

write(*,*) 'MPI_np check1: MPI_id=',MPI_id,'MPI_np=',MPI_np,'tempint',tempint
if(MPI_id/=0) call MPI_Send(tempint,1,MPI_integer8,0,MPI_id,MPI_COMM_WORLD,MPI_err)
DO i_MPI=1,MPI_np-1
  if(MPI_id==0) call MPI_Recv(tempint,1,MPI_integer8,i_MPI,i_MPI,MPI_COMM_WORLD,MPI_stat,MPI_err)
ENDDO
write(*,*) 'MPI_np check2: MPI_id=',MPI_id,'MPI_np=',MPI_np,'tempint',tempint

CALL MPI_Finalize(MPI_err)
End program test

It is compiled with

mpifort -fdefault-integer-8 test.f90

run with

mpirun -np 3 ./a.out

the output for 3 threads will be

 MPI_np check1: MPI_id=                    0 MPI_np=                    3 tempint                    1
 MPI_np check1: MPI_id=                    2 MPI_np=                    3 tempint                   -1
 MPI_np check1: MPI_id=                    1 MPI_np=                    3 tempint                   -1
 MPI_np check2: MPI_id=                    1 MPI_np=                    3 tempint                   -1
 MPI_np check2: MPI_id=                    0 MPI_np=                    0 tempint                   -1
 MPI_np check2: MPI_id=                    2 MPI_np=                    3 tempint                   -1

It seems by giving "MPI_id=0", the "MPI_np" will turn to 0 (should be 3) on the rank 0 after a MPI_recv call.

MPI_np check2: MPI_id=                    0 MPI_np=                    0 tempint                   -1

I do not understand how does this happen. Does anyone have an idea on this issue? thanks a lot ~

[jsquyres]

Two questions:

1. What threads are you referring to? I do not see any threaded code in your example.
2. I see you compiled your app with -fdefault-integer-8. Did you build Open MPI with the same flag?

Additionally, can you supply the information requested by the github issue template?

• Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)
• If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.
• Please describe the system on which you are running
  • Operating system/version:
  • Computer hardware:
  • Network type:

Thanks!

[ahaichen]

Two questions:

1. What threads are you referring to? I do not see any threaded code in your example.
2. I see you compiled your app with -fdefault-integer-8. Did you build Open MPI with the same flag?

Additionally, can you supply the information requested by the github issue template?

• Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

• If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

• Please describe the system on which you are running

  • Operating system/version:
  • Computer hardware:
  • Network type:

Thanks!

Hi jsquyres,

I have updated my question with these information. thanks a lot ~

[rhc54]

I still don't see any sign that your code is threaded - are you using "thread" to refer to a distinct "process"? Would you please show the cmd line you use to generate that output?

[ahaichen]

I still don't see any sign that your code is threaded - are you using "thread" to refer to a distinct "process"? Would you please show the cmd line you use to generate that output?

sorry, I mean the process, I have updated the question. Thanks ~

[rhc54]

I guess I don't understand your question, then. You are receiving the sent values into the tempint location inside rank=0, and all the rank > 0 procs are sending you a value of -1. So after the loop, of course tempint in rank=0 is -1.

What am I missing?

[ahaichen]

I guess I don't understand your question, then. You are receiving the sent values into the tempint location inside rank=0, and all the rank > 0 procs are sending you a value of -1. So after the loop, of course tempint in rank=0 is -1.

What am I missing?

The result of tempint is correct, but if you check the value "MPI_np" on rank=0, it is 0, which should be 3.

MPI_np check2: MPI_id=                    0 MPI_np=                    0 tempint                   -1

Thanks ~

[jsquyres]

F77 and FFLAGS aren't used any more; you need to use FCFLAGS (and FC, which you're already using).

What might be happening here is that you're effectively building Open MPI without -fdefault-integer-8, but then building your app with -fdefault-integer-8. Things then go downhill from there because of integer size mismatches between your app and the Open MPI fortran library.

[ahaichen]

F77 and FFLAGS aren't used any more; you need to use FCFLAGS (and FC, which you're already using).

What might be happening here is that you're effectively building Open MPI without -fdefault-integer-8, but then building your app with -fdefault-integer-8. Things then go downhill from there because of integer size mismatches between your app and the Open MPI fortran library.

sorry, I just lost the FCFLAGS=-fdefault-integer-8 part when i copy the text.

when I remove the =0 in MPI_id=0 in the code, the result will be fine. That really confused me.

[jsquyres]

Your test program doesn't compile:

$ mpifort foo.f90 -o foo
foo.f90:19:42:

   19 |   if(MPI_id/=0) call MPI_Send(tempint,Int1,MPI_integer8,0,MPI_id,MPI_COMM_WORLD,MPI_err)
      |                                          1
Error: Symbol ‘int1’ at (1) has no IMPLICIT type

I'm assuming that's supposed to be 1. Even after fixing that, it does not compile for me:

$ mpifort foo.f90 -o foo
foo.f90:19:85:

   19 | ll MPI_Send(tempint,1,MPI_integer8,0,MPI_id,MPI_COMM_WORLD,MPI_err)
      |                                                                   1

Error: There is no specific subroutine for the generic ‘mpi_send’ at (1)
foo.f90:21:100:

   21 | tempint,1,MPI_integer8,i_MPI,i_MPI,MPI_COMM_WORLD,MPI_stat,MPI_err)
      |                                                                   1

Error: There is no specific subroutine for the generic ‘mpi_recv’ at (1)

So I'm not sure how you're getting results...?

Can you produce a consistent question that we can validate and test?

Thanks!

[ahaichen]

Your test program doesn't compile:

$ mpifort foo.f90 -o foo
foo.f90:19:42:

   19 |   if(MPI_id/=0) call MPI_Send(tempint,Int1,MPI_integer8,0,MPI_id,MPI_COMM_WORLD,MPI_err)
      |                                          1
Error: Symbol ‘int1’ at (1) has no IMPLICIT type

I'm assuming that's supposed to be 1. Even after fixing that, it does not compile for me:

$ mpifort foo.f90 -o foo
foo.f90:19:85:

   19 | ll MPI_Send(tempint,1,MPI_integer8,0,MPI_id,MPI_COMM_WORLD,MPI_err)
      |                                                                   1

Error: There is no specific subroutine for the generic ‘mpi_send’ at (1)
foo.f90:21:100:

   21 | tempint,1,MPI_integer8,i_MPI,i_MPI,MPI_COMM_WORLD,MPI_stat,MPI_err)
      |                                                                   1

Error: There is no specific subroutine for the generic ‘mpi_recv’ at (1)

So I'm not sure how you're getting results...?

Can you produce a consistent question that we can validate and test?

Thanks!

Yes, that should be 1, I do compile the code. I get similar problem before on another machine. If you try

Program test
USE MPI_F08
IMPLICIT NONE

TYPE(MPI_Status)                            :: MPI_stat
Integer(kind=MPI_INTEGER_KIND) :: MPI_err
Integer(kind=MPI_INTEGER_KIND) :: MPI_id=0
Integer(kind=MPI_INTEGER_KIND) :: MPI_np
Integer                                              :: tempint,i_MPI

CALL MPI_Init(MPI_err)
CALL MPI_Comm_rank(MPI_COMM_WORLD,MPI_id,MPI_err)
CALL MPI_Comm_size(MPI_COMM_WORLD,MPI_np,MPI_err)

if(MPI_id==0) tempint=1
if(MPI_id/=0)  tempint=-1

write(*,*) 'MPI_np check1: MPI_id=',MPI_id,'MPI_np=',MPI_np,'tempint',tempint
if(MPI_id/=0) call MPI_Send(tempint,INT(1,kind=MPI_INTEGER_KIND),MPI_integer8,&
INT(0,kind=MPI_INTEGER_KIND),MPI_id,MPI_COMM_WORLD,MPI_err)
DO i_MPI=1,MPI_np-1
   if(MPI_id==0) call MPI_Recv(tempint,INT(1,kind=MPI_INTEGER_KIND),MPI_integer8, &
                                i_MPI,i_MPI,MPI_COMM_WORLD,MPI_stat,MPI_err)
ENDDO
write(*,*) 'MPI_np check2: MPI_id=',MPI_id,'MPI_np=',MPI_np,'tempint',tempint

CALL MPI_Finalize(MPI_err)
End program test

I guess it should be fine.

mpifort -fdefault-integer-8 test.f90
mpirun -np 3 ./a.out

 MPI_np check1: MPI_id=                    0 MPI_np=                    3 tempint                    1
 MPI_np check1: MPI_id=                    1 MPI_np=                    3 tempint                   -1
 MPI_np check2: MPI_id=                    1 MPI_np=                    3 tempint                   -1
 MPI_np check1: MPI_id=                    2 MPI_np=                    3 tempint                   -1
 MPI_np check2: MPI_id=                    2 MPI_np=                    3 tempint                   -1
 MPI_np check2: MPI_id=                    0 MPI_np=                    0 tempint                   -1

Thanks ~

[ggouaillardet]

Thanks for the report, this is indeed a genuine bug.

The variable is overwritten at

1,1,test (foo.f90:23)
1,1,  mpi_recv_f08 (recv_f08.F90:27)
1,1,    ompi_recv_f (precv_f.c:91)
1,1,      PMPI_Status_c2f (pstatus_c2f.c:98)

One option is we should be using (after implementing!) PMPI_Status_c2f08() instead of PMPI_Status_c2f()
An other option is the defintion of type(MPI_Status) is incorrect when INTEGER size is 8

@jsquyres any ideas?

[ahaichen]

Thanks for the report, this is indeed a genuine bug.

The variable is overwritten at

1,1,test (foo.f90:23)
1,1,  mpi_recv_f08 (recv_f08.F90:27)
1,1,    ompi_recv_f (precv_f.c:91)
1,1,      PMPI_Status_c2f (pstatus_c2f.c:98)

One option is we should be using (after implementing!) PMPI_Status_c2f08() instead of PMPI_Status_c2f()
An other option is the defintion of type(MPI_Status) is incorrect when INTEGER size is 8

@jsquyres any ideas?

sure, thanks.

If it is indeed a bug, Do I have to go back to USE MPI at present to make it through? thanks ~

[jsquyres]

Ah, I see my mistake -- I did not compile the app with -fdefault-integer-8 -- All the extra KIND notations in the app confused/distracted me, and I neglected to use that flag. FWIW, my simpler form of the test program was:

Program test
  USE MPI_F08
  IMPLICIT NONE

  TYPE(MPI_Status) :: MPI_stat
  Integer :: rank
  Integer :: size
  Integer :: tempint,i

  CALL MPI_Init()
  CALL MPI_Comm_rank(MPI_COMM_WORLD,rank)
  CALL MPI_Comm_size(MPI_COMM_WORLD,size)

  tempint = -1
  if (rank == 0) tempint=1

  write(*,*) 'MPI_np check1: MPI_id=',rank,'MPI_np=',size,'tempint',tempint
  if (rank == 0) then
     do i = 1, size-1
        call MPI_Recv(tempint, 1, MPI_integer8, i, i, MPI_COMM_WORLD, MPI_stat)
    enddo
  else
     call MPI_Send(tempint, 1, MPI_integer8, 0, rank, MPI_COMM_WORLD)
  endif
  write(*,*) 'MPI_np check2: MPI_id=',rank,'MPI_np=',size,'tempint',tempint

  CALL MPI_Finalize()
End program test

@ahaichen No, the problem is not use mpi vs. use mpi_f08. The problem is in Open MPI's handling of the difference in default size of C integers vs. Fortran integers (i.e., 4 vs. 8). @ggouaillardet is pointing out that have an error in converting between C and Fortran status, because of this difference. I should note that this is not a common case (where size of C int != size of fortran INTEGER), and therefore something that probably has not been tested in years.

An alternative is to build Open MPI and your app without -fdefault-integer-8 and just increase the size of integers to 8 bytes in your Fortran code as necessary (i.e., just for specific values that need to be 8 byte integers).

@ggouaillardet and I are going to need to confer about this; it may actually be a large issue.