Implement --enable-static-runtime configure option.

[gpaulsen]

From #729:

Jeff Squyres recommend this approach to solve this type of issue:

I do like the idea of a --enable-static-runtime kind of configure option. This would seem to solve all the problems, and not require the user to know/do anything at runtime.

The only problem is: I'm not entirely sure how to do that.

1. With the Intel compiler, there's the --static-intel option, but that doesn't make a wholly static orted, for example (it just statically links in the Intel libraries -- not all libraries).
2. Are there universal options for this for the other compilers?

...after thinking about this for a few minutes, it may be sufficient to do something like this:

if enable_static_runtime was passed
    RUNTIME_LDFLAGS=--static
    if compiler suite is intel
        RUNTIME_LDFLAGS="$RUNTIME_LDFLAGS --static-intel"
    fi
fi

Assuming $(RUNTIME_LDFLAGS) is used when linking the orted (and any other relevant executables?), Libtool might automatically translate --static into whatever is relevant for the underlying compiler, and we only have one special case for the Intel compiler suite.

[jsquyres]

@gpaulsen Your Github wiki-foo is weak. :-) I updated the description to be proper wiki notation.

[gpaulsen]

😳 I shall aspire to wiki-it-up ❗

[ggouaillardet]

fwiw, except having a statically linked orted (or orted that only links with system libs) i do not see how this can be achieved ...
libopen-rte.so, libopen-pal.so and all plugins (if any) need the intel runtime.
so far, i think the only way to achieve the desired result is to have orted binary link with libopen-rte.a, libopen-pal.a and no plugin at all.

[rhc54]

That is correct - Brian Barrett had done this a few years back. The key is that you build all of OMPI as static. Today, that still won't build the orted as fully static, so you have to add some magic dust to get that to include all the dependent libs. I don't remember what he had to do - something in the orted Makefile.am plus some configure stuff. You could send him a note and ask - he might remember.

[gpaulsen]

Why must all of OMPI be built static? Why not just link / archive (ar) each of the OMPI pieces against the static compiler runtime libraries? seems like you could still build a liborte.so, but just archive in the static C runtime.

[rhc54]

You could try that - I was just relaying what Brian had done. The problem he had was that he built OMPI static, but the orted was not fully static and was still trying to load dependent libs.

[ggouaillardet]

@gpaulsen is that even possible ? if i read you correctly, it means each .so includes the static compiler runtime lib. that would make all the .so very big, plus all symbols would be defined multiple time.
an other option could be to tweak the linker command so libopen-rte.so does not require the intel runtime. i am not sure the linker would allow this (undefined symbol) nor what will happen when linking orted (will the linker pull all the runtime symbols required by libopen-rte.so and not by orted itself, not to mention the plugins ...)

[gpaulsen]

I'd think it'd be possible if you took all of libX's .os and ar'ed them with the compiler runtime .a to create another .a (built PIC of course), and then created a .so from that. I'd think the ar would resolve any of the undefines, but not pickup any symbols that were not needed.
There can be weird other issues though if some "globals" (though possibly not globally scoped) from the compiler runtime exists multiple times in the final executable.

[jsquyres]

Remember: the goal here is not to make a 100% static orted, but rather just eliminate the need for setting LD_LIBRARY_PATH on remote nodes to find the compiler run-time libraries. The mechanism for doing this -- with the Intel compiler, at least -- just happens to be --static-intel, which statically links in the Intel compiler runtime libraries (vs. dynamically linking them in).

[rhc54]

I'm not sure that is completely accurate. We want to eliminate dependencies on any non-standard dynamic libraries. So if you build OMPI against an external hwloc, libevent, or libpmix, then we will also want to pick those up since they might not be installed in the remote LD_LIBRARY_PATH.

Given all the potential combinations, a static orted might be the easiest solution.

[jsquyres]

FWIW, I stated that the goal was to link in the compiler runtime because we know we can do that. It's easy, low-hanging fruit, and is >90% of the problems we've heard over the years.

It's difficult to make a 100% static orted (or, at least, an orted that is statically linked against libraries in non-standard locations) for the following reasons:

1. We don't know exactly what "non-standard locations" are for the target machine. The system administrator and/or user could well have set LD_LIBRARY_PATH or /etc/ld.so.conf to point to all the Right places for us.
2. We'd have to build both static and dynamic OMPI libraries (i.e., libopen-pal and libopen-rte), and then force the orted to link against the .a versions, not the dynamic versions. This is probably do-able, but it'll take some skill/creative workarounds and a bunch of work (i.e., this is not a normal model for Automake, and so you'll have to be creative to make it do that).
3. It also depends on having static libraries for all of orteds dependencies. E.g., if the user is building against external library X, and there's no libX.a, then we can't statically link it in. That being said, I don't think we have link libX as static -- there's also the option of rpath/runpath-ing libX's dynamic library location into the orted. But that must be done with care, too -- we don't want to rpath/runpath /usr or other well-known library locations (at a minimum). This solution is do-able, but also entails a bit of m4 work.

[ggouaillardet]

I made some tests today, and surprisingly, linking orted with --static is enough, assuming ompi was configure'd with --enable-static. it seems --static-intel is not even necessary since the generated c code does not seem to require the runtime (!)
I will run some more tests and make sure --enable-static-runtime does force --enable-static
(that does not look trivial at first glance, @jsquyres do you know an easy way to achieve this ?)

[jsquyres]

I'd guess that in the Intel compiler suite, if they see --static, it also enables --static-intel.

I agree that it might be problematic to ensure that if CFLAGS contains --static that OMPI's configure should automatically --enable-static. It might be sufficient for OMPI's configure to error out if CFLAGS contains --static and --enable-static was not specified...?