GHC binaries from haskell.nix don't work on aarch64

[TravisWhitaker]

I believe that haskell.nix is patching GHC (or making some other change vs. upstream) in a way that impacts memory safety on aarch64.

With minor changes, haskell.nix can be used to build e.g. an 8.8.3 bootstrapped with 8.8.2 on aarch64: #620

Building this way with the branch from that linked PR:

nix-build -E "with import ../nixpkgs-channels/default.nix (import ./. {}).nixpkgsArgs; haskell-nix.compiler.ghc883"

Where ../nixpkgs-channels is pointed at current nixos-20.03, the build fails at the very end with this:

ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
ghc-stage2: internal error:     Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
ghc-stage2: internal error: evacuate: strange closure type -1860476400
    (GHC version 8.8.3 for aarch64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
make[1]: *** [utils/haddock/ghc.mk:20: utils/haddock/dist/build/Paths_haddock.dyn_o] Aborted (core dumped)

This occurs at the very end, where stage 2 is being used to build haddock. I initially suspected some memory barrier regressions in the 8.8 branch (and opened this https://gitlab.haskell.org/ghc/ghc/issues/18201), but I've failed to reproduce this issue with vanilla upstream 8.8.3.

I'm sifting through the build.mk and configure flags that haskell.nix chooses to see if I work out what might be causing this, next will be sifting through the patches that are applied.

[angerman]

@TravisWhitaker try disabling -fast-llvm

[TravisWhitaker]

Looks like this is due to “-fast-llvm.” If I remove it from the generated build.mk, I get a GHC 8.8.3 that works just fine.

[angerman]

@TravisWhitaker did you have -fast-llvm and -fno-plt everywhere?

[angerman]

@TravisWhitaker there is also -dno-llvm-mangler. So -fllvm -dno-llvm-mangler should do .ll -> .S -> .o. If that also doesn't crash this would be rather impressive.

[angerman]

Again, -fast-llvm can only work

• with -fno-plt
• without avx calls
• without -deadstrip on macOS.

[TravisWhitaker]

@angerman What I just tried on a hunch was my branch here with the bit here that adds -fast-llvm to GhcStage2HcOpts and GhcLibHcOpts commented out.

Looking at compiler/ghc/default.nix, I think we're breaking your first rule when building GHC itself; I don't see -fno-plt anywhere in the build.mk or configure flags we use.

[TravisWhitaker]

Should we build GHC itself with -fno-plt, or should we build it without -fast-llvm?

[angerman]

-fast-llvm + -fno-plt or no -fast-llvm and using the Mangler. I'd absolutely be for the first appraoch, as it allows LLVM to work off of the higher level LL instead of the lowered assembly.

[TravisWhitaker]

I’ll give that a go and see if we get a working aarch64 GHC.

[angerman]

to validate it's the function -> object rewrite, you could build without -fast-llvm but with -dno-llvm-mangler.

[angerman]

Actually, I think this might be more tricky than using -fno-plt. -fno-plt would only apply to $CC but not $HC. For $HC with -fllvm we need to make the emitted .ll contain nonlazybind attributes.

[angerman]

The relevant check is in https://github.com/microsoft/llvm/blob/master/test/CodeGen/AArch64/nonlazybind.ll

[angerman]

Here's a rough patch of what the llvm codegen probably needs to do: https://gist.github.com/angerman/0dbc0b6587ed4d3ca01299924d26c255

[TravisWhitaker]

Can we disable -fast-llvm until that's sorted? It definitely doesn't yield correct code on aarch64.