Use GHC CI infrastructure to produce release artifacts.

[phadej]

The current CI is good for daily development, but GHC CI has wider platform coverage, which would be great for providing binary releases.

Yet, the setup have to be made.

[bgamari]

Anyone interested should get in touch with me. I'm happy to advise.

[samuela]

Hey @bgamari I'm interested in taking a stab at this! What needs to be done?

[bgamari]

@samuela, great! Essentially, the task is to write a .gitlab-ci.yml similar to GHC's which uses the same build environments to build and package cabal-install. The GHC build environments already include cabal-install (since it's necessary to build GHC's Hadrian build system) so the task of building cabal-install should be fairly straightforward.

I have put up a quick experiment here. Feel free to use it as a starting point. The remaining work is the following:

• fix the logic (i.e. the find command in my experiment) for locating and copying the executable
• modify the Linux job in my experiment to rather use the CentOS 7 build instead of Debian 9 (this ensures that we build against an old glibc, allowing us to use the same executable across all glibc-based distributions)
• Possibly freeze the Hackage index date?
• Add jobs for (using the GHC .gitlab-ci.yml as a template):
  • AArch64 Linux
  • ARMv7 Linux
  • Darwin (macOS)
  • Windows
  • x86-64 FreeBSD
  • x86-64 Alpine Linux (musl)
  • any other platforms that I have forgotten
• add --enable-split-sections to the cabal command line
• build and package the documentation?
• possibly package the build products into a tarball?
• cache the Cabal store?

The platforms which don't use Docker are a bit tricky; on these platforms (Windows, Darwin, FreeBSD) GHC's .gitlab/ci.sh script is responsible for installing a GHC binary distribution. You'll need to replicate this functionality.

I would be happy to set up a chat to discuss any details if this would be helpful. Alternatively, find me in #ghc on irc.freenode.net.

Thanks for volunteering!

[hasufell]

fix the logic (i.e. the find command in my experiment) for locating and copying the executable

    - mkdir -p out
    - cabal update
    - cabal new-install --install-method=copy --overwrite-policy=always --installdir=$(pwd)/out cabal-install

If you want to use new-build and then find the exe, do:

    - mkdir -p out
    - cabal update
    - cabal new-build
    - cp "$(cabal new-exec --verbose=0 --offline sh -- -c 'command -v cabal')" out/cabal

[bgamari]

Thanks @hasufell! I adopted the v2-install approach in my branch.

[samuela]

@samuela, great! Essentially, the task is to write a .gitlab-ci.yml similar to GHC's which uses the same build environments to build and package cabal-install. The GHC build environments already include cabal-install (since it's necessary to build GHC's Hadrian build system) so the task of building cabal-install should be fairly straightforward.

Thanks for all the info! Since this project is hosted on GitHub whereas GHC is hosted on GitLab, we'll need to do something different than .gitlab-ci.yml, right? It looks as though cabal is currently using GitHub Actions and Travis. So is the ultimate goal to do these builds on cabal GH Actions, cabal Travis, or ghc gitlab CI?

[samuela]

@bgamari Oh, I think I may have misunderstood! So your fork lives on gitlab and has a .gitlab-ci.yml file. So which of these CI systems is/should be producing the bindists?

[bgamari]

@samuela the plan here is to use GHC's GitLab CI to prepare the cabal-install binary distributions since then we can easily piggy-back on the existing work put into GHC's infrastructure (not to mention, neither GitHub Actions nor Travis support the range of platforms supported by GHC).

[samuela]

Ok, so in order to start adapting https://gitlab.haskell.org/bgamari/cabal-build-test/-/commit/50b81bcf1208dbd7ea566d0a2e0fa428dbdd4d00 to work with centos, we need a new docker image in ci-images. Here's an attempt: https://gitlab.haskell.org/samuela/ci-images/-/commit/b50c3c8049321d8542e7bc29bb03c4e815765621 but it fails with

Step 10/25 : RUN curl -L https://downloads.haskell.org/~ghc/$GHC_VERSION/ghc-$GHC_VERSION-armv7-centos7-linux.tar.xz | tar -Jx
 ---> Running in 1ee41616a1d9
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   178  100   178    0     0    217      0 --:--:-- --:--:-- --:--:--   217
xz: (stdin): File format not recognized
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/bash -o pipefail -c curl -L https://downloads.haskell.org/~ghc/$GHC_VERSION/ghc-$GHC_VERSION-armv7-centos7-linux.tar.xz | tar -Jx' returned a non-zero code: 2

So we'll need ghc-8.8.3-armv7-centos7-linux.tar.xz first...

[hasufell]

There is armv7 for https://downloads.haskell.org/~ghc/8.2.2/ (debian 8). You could try to bootstrap from there.

[bgamari]

@samuela I would say let's stick with building on Debian 9 on ARMv7 and AArch64. We don't currently build CentOS bindists on these platforms; it's something that ideally we would ideally do but this will take time and most people tend to use Debian-based distributions on these platforms anyways.

[samuela]

Ok, I've started some work on this branch. Here are the pipeline results. So far armv7-linux-deb9 and aarch64-linux-deb9 seem to be working but x86_64-freebsd failed.

[hasufell]

@samuela nice, a few comments

1. I think the FreeBSD runner doesn't have a cabal-3.x yet. I also manually update it in the ghcup CI. You can download FreeBSD binaries here: https://hasufell.de/d/d3e215db133e4fcaa61e/ (I signed the hash file). @hvr hasn't responded to my request of adding those tarballs to downloads.haskell.org yet.

2. I believe the idea was to add this CI in the actual cabal repository and then build the current in-tree version of cabal-install: https://gitlab.haskell.org/bgamari/cabal-build-test I believe your version currently builds from hackage? @phadej there seems to be no cabal mirror on gitlab.haskell.org yet?