Alternatives to a self-hosted runner to build statically-linked Stack for Linux/AArch64?

[mpilgrem]

Stack currently uses a GitHub workflow and a self-hosted runner to build a statically-linked Stack for Linux/AArch64.

Historically, the runner has been provided by FP Complete. It provides Ubuntu 20.04.2 LTS on AArch64. The statically-linked Stack is then built in an Docker container providing Alpine Linux, with:

/usr/local/bin/stack etc/scripts/release.hs build --alpine --build-args --docker-stack-exe=image

The Docker image being specified (currently) by:

docker:
  enable: false
  repo: quay.io/benz0li/ghc-musl:9.6.4

@benz0li having managed to compile versions of GHC that work on Alpine Linux/AArch64.

The Haskell Foundation tried to provide a replacement runner but its machine provides NixOS on AArch64 and, as @chreekat has explained elsewhere, exposure to that environment is not good for building binaries.

I am thinking about possible alternatives. One thing I am going to explore - it may fail - is that GitHub runner macos-14 provides an AArch64 machine architecture and Docker is available for macOS/AArch64.

[hasufell]

I have runners for aarch64 linux that are not based on NixOS. They are currently hooked up to:

• haskell org
• stable-haskell org

I'm already building stack binaries there and it works.

The problem is you can't share runners across repos/orgs. So you have to start a separate agent on the same machine and they don't know about each other, potentially leading to resource exhaustion if they happen to run at the same time.

[mpilgrem]

@hasufell, thanks - your observation tallies with your issue #6252, which discussion was primarily about macOS/AArch64 but is now pertinent for Linux/AArch64.

[mpilgrem]

My experiment is not succeeding, so far, with:

Raw command: /opt/homebrew/bin/docker inspect quay.io/benz0li/ghc-musl:9.6.4
Standard output:
[]
Standard error:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

To investigate: (1) what is the source of that (2) can it be overcome?

EDIT: Perhaps it is that I need to start the Docker daemon manually: https://docker-docs.uclv.cu/config/daemon/#start-the-daemon-manually

EDIT2: That part of the Docker documentation appears to be misleading: moby/moby#27102

EDIT3: It seems possible that it is simply not possible to connect to the Docker daemon on macOS from the command line:

• https://stackoverflow.com/questions/44084846/cannot-connect-to-the-docker-daemon-on-macos
• create dev environment failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock docker/for-mac#6531

EDIT4: These blog posts may be of assistance:

• https://dhwaneetbhatt.com/blog/run-docker-without-docker-desktop-on-macos/
• https://dev.to/elliotalexander/how-to-use-docker-without-docker-desktop-on-macos-217m

[mpilgrem]

Using https://github.com/abiosoft/colima seemed promising, but colima fails to start, with (edited):

level=info msg="[hostagent] hostagent socket created at /Users/runner/.colima/_lima/colima/ha.sock"
level=info msg="[hostagent] Using system firmware (\"/opt/homebrew/share/qemu/edk2-aarch64-code.fd\")"
level=info msg="[hostagent] Starting QEMU (hint: to watch the boot progress, see \"/Users/runner/.colima/_lima/colima/serial*.log\")"
level=info msg="SSH Local Port: 49212"
level=info msg="[hostagent] Waiting for the essential requirement 1 of 4: \"ssh\""
level=info msg="[hostagent] Driver stopped due to error: \"signal: abort trap\""
level=info msg="[hostagent] Shutting down the host agent"
level=warning msg="[hostagent] failed to exit SSH master"
level=info msg="[hostagent] Shutting down QEMU with ACPI"
level=warning msg="[hostagent] Failed to remove SSH binding for port 49212"
level=warning msg="[hostagent] failed to open the QMP socket \"/Users/runner/.colima/_lima/colima/qmp.sock\", forcibly killing QEMU"
level=info msg="[hostagent] QEMU has already exited"
level=fatal msg="exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see \"/Users/runner/.colima/_lima/colima/ha.stderr.log\")"
level=fatal msg="error starting vm: error at 'creating and starting': exit status 1"
##[error]Process completed with exit code 1.

This is a known problem: the macos-14 runner uses M1 machine architecture, M1 does not support nested virtualisation, and colima needs the latter: abiosoft/colima#970.

[mpilgrem]

However, colima may offer a local solution because the virtualisation is not nested. On my Mac mini/M1:

brew install docker
brew install colima
stack upgrade --force-download # Ensure Stack version same as in Docker image
colima start
stack etc/scripts/release.hs build --alpine --build-args --docker-stack-exe=image

built executable files but, understandably, failed when the Haskell script came to test them in macOS:

_release/bin/stack-2.15.4.1-osx-aarch64/stack --version
_release/bin/stack-2.15.4.1-osx-aarch64/stack: _release/bin/stack-2.15.4.1-osx-aarch64/stack: cannot execute binary file

EDIT: If I create an Alpine Linux VM on my Mac mini with UTM and share the relevant directory with the VM, I can, however, test the executable:

/media/share/GitHub/commercialhaskell/stack/_release/bin/stack-2.15.4.1-osx-aarch64/stack --version
Version 2.15.4.1, Git revision c34001760ea06ca44537e3a0ead349b98006f324 (10548 commits) RELEASE-CANDIDATE aarch64 hack-0.436.0

[benz0li]

https://github.blog/changelog/2023-10-30-accelerate-your-ci-cd-with-arm-based-hosted-runners-in-github-actions/

[hasufell]

The private beta is enabled in the haskell org. But those runners are not free.

[mpilgrem]

@benz0li, may I ask - the (statically-linked?) GHC for Alpine Linux/AArch64 that you build - are the binary distributions separately available for use on the OS, or are they only published as part of a Docker image?