Cabal should probably add extra-lib-dirs to RPATH

[bgamari]

Describe the bug
In GHC #19350 it was noted that dynamic libraries findable via extra-lib-dirs are not found in GHCi on POSIX platforms. While in principle GHC has the information that it needs to find these libraries from the package database registration, due to limitations of the the POSIX dynamic linking interface, it is not possible for GHC to do better here.

I believe the solution here is to (on POSIX platforms) add Cabal's extra-lib-dirs paths to RPATH during the final link of libraries. This isn't a perfect solution (since it may result in spurious RPATH entries) but this seems to be the best we can do.

To Reproduce
See this testcase.

Expected behavior
GHCi is able to load libraries with foreign dependencies findable only via extra-lib-dirs.

System information

• Operating system
• cabal, ghc versions

Additional context
Add any other context about the problem here.

[phadej]

How this interacts with change proposed in #7094

[bgamari]

In #7094 we were essentially forced into placing more RPATH logic in GHC (specifically when targeting Darwin) since the patching we are forced to do in Darwin is rather involved. This was a pragmatic choice, but ultimately it was hack that meant conflating two distinct notions. Specifically, it means that GHC uses -L inconsistently with how other compilers treat that flag (since usually it merely affect the compile-time search path; in GHC it now also affects the run-time search behvaior).

To address GHC #19350 we could use a similar approach, teaching GHC to implicitly add -L directories to RPATH. However I think we should avoid this. -L and RPATH affect two distinct things; while on Darwin, due to the nature of the environment, we can conflate these, doing so on Linux would cause trouble for packagers (since you would almost certainly end up with redundant RPATH entries in some cases; e.g. when you are building a package in a temporary prefix for installation elsewhere). By keeping these two concepts distinct, GHC remains flexible enough to be usable in such cases.

In short, there is not appreciable interaction between #7094. #7094 is a pragmatic exception to the rule that -L and RPATH are orthogonal concerns. Here I am suggesting that the compiler continue considering them to be orthogonal but we add the policy logic for determining RPATH to Cabal.

[phadej]

I don't understand what this issue suggests?

Change extra-lib-dirs: directories to not generate -L flags, but -rpath ones? But on Darwin we won't as rpath support is not hasInstallNameTool, and then -rpath stuff is passed as -L anyway?

[bgamari]

Concretely, extra-lib-dirs should generate both -L flags and -rpath flags.

[phadej]

I was also going to suggest

Perhaps GHC should grow a -rpath flag of its own to abstract this away?

But you did it yourself.

[phadej]

For the reference, as to me rpath is mystical thing: https://en.wikipedia.org/wiki/Rpath

[phadej]

And to clarify. I'm just asking questions. I have no opinion on this, slightly worried that linking is becoming ad-hoc spagetti, but maybe it is just what it is.

[bgamari]

Unfortunately it does tend to be quite platform specific. There was sadly a reason why libtool was necessary for so many years.

[angerman]

Concretely, extra-lib-dirs should generate both -L flags and -rpath flags.

I'm not sure I agree, it should generate -L sure, but leave the rpath logic on darwin up to ghc. The whole idea that cabal takes care of rpath and ghc takes care of rpath is not very good. Ideally cabal would just delegate to hc, and hc would deal with the actual linking, let's not replicate linking logic in both.

[bgamari]

GHC #20026 is relevant here.

[Mikolaj]

Also, #7382 seems related.