GHC binaries from haskell.nix don't work on aarch64

[TravisWhitaker]

I believe that haskell.nix is patching GHC (or making some other change vs. upstream) in a way that impacts memory safety on aarch64.

With minor changes, haskell.nix can be used to build e.g. an 8.8.3 bootstrapped with 8.8.2 on aarch64: #620

Building this way with the branch from that linked PR:

nix-build -E "with import ../nixpkgs-channels/default.nix (import ./. {}).nixpkgsArgs; haskell-nix.compiler.ghc883"

Where ../nixpkgs-channels is pointed at current nixos-20.03, the build fails at the very end with this:

ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
ghc-stage2: internal error:     Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
make[1]: *** [utils/haddock/ghc.mk:20: utils/haddock/dist/build/Paths_haddock.dyn_o] Aborted (core dumped)

This occurs at the very end, where stage 2 is being used to build haddock. I initially suspected some memory barrier regressions in the 8.8 branch (and opened this https://gitlab.haskell.org/ghc/ghc/issues/18201), but I've failed to reproduce this issue with vanilla upstream 8.8.3.

I'm sifting through the build.mk and configure flags that haskell.nix chooses to see if I work out what might be causing this, next will be sifting through the patches that are applied.

[angerman]

@TravisWhitaker try disabling -fast-llvm

[TravisWhitaker]

Looks like this is due to “-fast-llvm.” If I remove it from the generated build.mk, I get a GHC 8.8.3 that works just fine.

[angerman]

@TravisWhitaker did you have -fast-llvm and -fno-plt everywhere?

[angerman]

@TravisWhitaker there is also -dno-llvm-mangler. So -fllvm -dno-llvm-mangler should do .ll -> .S -> .o. If that also doesn't crash this would be rather impressive.

[angerman]

Again, -fast-llvm can only work

• with -fno-plt
• without avx calls
• without -deadstrip on macOS.

[TravisWhitaker]

@angerman What I just tried on a hunch was my branch here with the bit here that adds -fast-llvm to GhcStage2HcOpts and GhcLibHcOpts commented out.

Looking at compiler/ghc/default.nix, I think we're breaking your first rule when building GHC itself; I don't see -fno-plt anywhere in the build.mk or configure flags we use.

[TravisWhitaker]

Should we build GHC itself with -fno-plt, or should we build it without -fast-llvm?

[angerman]

-fast-llvm + -fno-plt or no -fast-llvm and using the Mangler. I'd absolutely be for the first appraoch, as it allows LLVM to work off of the higher level LL instead of the lowered assembly.

[TravisWhitaker]

I’ll give that a go and see if we get a working aarch64 GHC.

[angerman]

to validate it's the function -> object rewrite, you could build without -fast-llvm but with -dno-llvm-mangler.

[angerman]

Actually, I think this might be more tricky than using -fno-plt. -fno-plt would only apply to $CC but not $HC. For $HC with -fllvm we need to make the emitted .ll contain nonlazybind attributes.

[angerman]

The relevant check is in https://github.com/microsoft/llvm/blob/master/test/CodeGen/AArch64/nonlazybind.ll

[angerman]

Here's a rough patch of what the llvm codegen probably needs to do: https://gist.github.com/angerman/0dbc0b6587ed4d3ca01299924d26c255

[TravisWhitaker]

Can we disable -fast-llvm until that's sorted? It definitely doesn't yield correct code on aarch64.

[bgamari]

Yes, I would agree with @TravisWhitaker; the solution here seems obvious: disable -fast-llvm. As I've previously noted, it really isn't safe in its current incarnation. Perhaps we can consider introducing something like it into a future release (c.f. GHC #18179) but using the variant in 8.8 and earlier is only going to cause more pain for everyone.

[angerman]

Can we disable -fast-llvm until that's sorted? It definitely doesn't yield correct code on aarch64.

@TravisWhitaker yes we should drop it. Even though I remain highly skeptical about using the Mangler, and using two different toolchains for code gen and assembly. However as this seems to be rather poorly supported in llvm, even nonlazybind is not guaranteed to be respected by every lowering backend.