v4.0.x: Backport: Ensure that --host / --hostfile nodes are always used in order provided

[rhc54]

Backport: Ensure that nodes are always used in order provided

Corresponds to following commits to OMPI master:

35a5971
2794ae4
aed06e6
5aa775c

Signed-off-by: Ralph Castain [email protected]

[rhc54]

@jsquyres @jjhursey Please check this out. Once we confirm it is okay, I'll use it as the basis for backports to the v3.x.y series.

[jsquyres]

This is a big change. I see:

• Deletion of the compress/bzip and compress/gzip components
• Addition of the compress/zlib component (which looks like it's at least kind of a rename of gzip->zlib)
• Deletion of the regx framework

I think that these are technically backwards-incompatible changes, right? Realistically, the end-user impact will likely be zero (i.e., I find it hard to believe that any real world user would be selecting / passing MCA params to the bzip/gzip components), but still -- we should acknowledge that we're intentionally breaking backwards compatibility here.

Also, I see changes to DSS packed/unpacked messages. Does this PR change the wire protocol? If so, does this have implications in container environments, e.g., where mpirun is outside the container and is a different version of OMPI than is inside the container?

[rhc54]

I think that these are technically backwards-incompatible changes, right? Realistically, the end-user impact will likely be zero (i.e., I find it hard to believe that any real world user would be selecting / passing MCA params to the bzip/gzip components), but still -- we should acknowledge that we're intentionally breaking backwards compatibility here.

Nobody could use those components because they were only used for CR, which was removed from the release branches long ago.

Also, I see changes to DSS packed/unpacked messages. Does this PR change the wire protocol? If so, does this have implications in container environments, e.g., where mpirun is outside the container and is a different version of OMPI than is inside the container?

We have always required that the daemons and mpirun all be at the same release level. Within that constraint, this doesn't break anything. However, if mpirun is outside the container, and the daemons are from some other version inside the container, then yes - launch will be broken.

[hppritcha]

@gpaulsen this is huge to merge in while we are in rc state. I'd say this waits till 4.0.2.

[gpaulsen]

@hppritcha I agree. 4.0.2 would be sufficient.

[jsquyres]

@rhc54 FWIW, I'm getting assertion failures with this PR:

$ mpirun --host mpi001,mpi002 hostname
mpirun: base/rml_base_msg_handlers.c:184: orte_rml_base_process_msg: Assertion `((void *)0) != orte_node_regex' failed.
[1]    14227 abort (core dumped)  mpirun --host mpi001,mpi002 hostname

[rhc54]

@jsquyres Should be okay now - I missed a spot in the backport.

[jsquyres]

@rhc54 Something still seems to be missing. I'm not getting assertion failures any more, but I'm getting the old/bad behavior:

$ mpirun --host mpi001,mpi002 ~/foo.sh
mpi001: 0 rank
mpi002: 1 rank
$ mpirun --host mpi002,mpi001 ~/foo.sh
mpi001: 0 rank
mpi002: 1 rank

[rhc54]

I'm beginning to wonder if this is worth all the pain...hitting my limits on how much time I can devote to it.

[jjhursey]

I'm going to take myself off as a review to prevent the Pull Request Reminder from bugging me. Add me back when this is ready for review.

[gpaulsen]

@ggouaillardet - Ralph said in today's web-ex that he realistically won't have time for this in the coming months. Is this something you could help with?

[gpaulsen]

Removing milestone as we're currently unsure of the scope, schedule of getting this PR working.