PML selection issues

[yburette]

We are observing two issues with the way the PML is selected:

1. on Truescale, when mpirun is used without any argument, the openib BTL (ob1 PML) gets selected over the PSM MTL (cm PML). This is a problem because users would like to get the best performance (with the PSM MTL) out-of-the-box. Instead, they have to add --mca pml cm.

2. currently, whichever of the ob1 or the cm PML gets initialized/loaded first wins! Unfortunately, the components are not always loaded in the same order...

[yburette]

ob1 PML has a default priority of 20.
cm PML has a default priority of 10.

if ob1 is initialized first, then cm_init() sees that its priority is lower and thus returns NULL.
---> openib BTL is selected

if cm is initialized first, then it looks at which MTL can be selected. If a MTL is usable, cm's priority is bumped to 30. ob1_init() then sees that its priority is lower and thus returns NULL.
---> PSM MTL is selected

[yburette]

https://github.com/open-mpi/ompi-release/blob/v1.10/ompi/mca/pml/ob1/pml_ob1_component.c#L251

if((*priority) > mca_pml_ob1.priority) { 
    *priority = mca_pml_ob1.priority;
    return NULL;
}

https://github.com/open-mpi/ompi-release/blob/v1.10/ompi/mca/pml/cm/pml_cm_component.c#L146

if((*priority) > ompi_pml_cm.default_priority) { 
    *priority = ompi_pml_cm.default_priority;
    return NULL;
}

[yburette]

A simple fix which works for Truescale is to increase the cm PML's default priority to 20 so that it can compete with ob1.
However, this might not work for the other components (e.g. MXM, portals).

@jsquyres @hppritcha @rhc54 What do you guys think?

[hjelmn]

I would think it would be safe to remove the extra check against ob1. I will take a look and see if I can determine why cm's author did that.

[rhc54]

Can we get this resolved soon?

[jsquyres]

@yburette I'm not sure I understand the proposed solution: if you increase the default cm priority to 20, what happens? Does CM then always increase its priority to 30 (assuming the PSM/PSM2 MTL is available)? If so, I think we need to check with @regrant and @tkordenbrock to see what this will do for the Portals BTL / MTL combo...?

(I agree we need a solution, though (ASAP) -- what you described with the openib BTL/PSM MTL being selected according to the order in which they were loaded is Bad)

[hjelmn]

@jsquyres I agree, which is why I propose #1039 to fix this issue.

[hjelmn]

I can make similar changes to ob1.

[jsquyres]

Hah -- I just commented on the same thing (do we need the same changes in ob1?) on #1039. Our emails crossed in the ether, I guess. 😄

[yburette]

@jsquyres @hjelmn
Good point and this is exactly what I lack knowledge of. For PSM/PSM2, Intel would like to have the MTL selected by default. What is the expected behavior for Portals and MXM?

I think we could increase the priority to 30 only for the desired MTLs and leave it at 20 (or even lower it back down to 10) for the others.

If we remove the conditional check from ob1, I believe we can increase cm's priority for the desired MTLs only.

All BTLs get ob1's priority (20), right?