Compile fail with --enable-heterogeneous

[jsquyres]

On master, we're getting a compile fail when --enable-heterogeneous is specified:

make[1]: Entering directory `/home/jsquyres/git/ompi/ompi/mca/coll'
  CC       base/coll_base_alltoall.lo
In file included from ../../../opal/mca/threads/thread_usage.h:32,
                 from ../../../opal/class/opal_object.h:126,
                 from ../../../opal/datatype/opal_datatype.h:44,
                 from ../../../opal/datatype/opal_convertor.h:35,
                 from ../../../ompi/datatype/ompi_datatype.h:40,
                 from base/coll_base_alltoall.c:31:
base/coll_base_alltoall.c: In function ‘mca_coll_base_alltoall_intra_basic_inplace’:
base/coll_base_alltoall.c:81:85: error: invalid use of undefined type ‘struct opal_convertor_master_t’
   81 |         if( OPAL_UNLIKELY(opal_local_arch != ompi_proc->super.proc_convertor->master->remote_arch))  {
      |                                                                                     ^~
../../../opal/include/opal/prefetch.h:44:63: note: in definition of macro ‘OPAL_UNLIKELY’
   44 | #        define OPAL_UNLIKELY(expression) __builtin_expect(!!(expression), 0)
      |                                                               ^~~~~~~~~~
base/coll_base_alltoall.c:83:100: error: invalid use of undefined type ‘struct opal_convertor_master_t’
   83 |                                                             ompi_proc->super.proc_convertor->master->remote_sizes);
      |                                                                                                    ^~
make[1]: *** [base/coll_base_alltoall.lo] Error 1
make[1]: Leaving directory `/home/jsquyres/git/ompi/ompi/mca/coll'
make: *** [all-recursive] Error 1

I'm not sure what commit introduced the error.

[thesamesam]

Could the fix for this be backported to 4.1.x? We hit it in Gentoo with 4.1.2.

[jsquyres]

The heterogeneous functionality is broken; you should not be using it. Specifically: you should not include --enable-heterogeneous in v4.0.x or v4.1.x builds.

That being said, it looks like there was a mistake made in the v4.1.x README (perhaps an errant merge conflict resolution?).

• Notice how the master branch README.md has a "DO NOT USE" warning in the --enable-heterogeneous description: https://github.com/open-mpi/ompi/blob/master/README.md?plain=1#L1504-L1509
• But that same warning is accidentally in the --enable-spc option (which is right below the --enable-heterogeneous option) in the v4.1.x README: https://github.com/open-mpi/ompi/blob/v4.1.x/README#L1515-L1523
• Looks like the same thing happened on the v4.0.x branch, too: https://github.com/open-mpi/ompi/blob/v4.0.x/README#L1510-L1518

Yoinks. I will file PRs to fix the README's on v4.0.x and v4.1.x. Thanks for brining this to our attention!