[Github Actions] Build Flake

[smklein]

Taking a look at The Github actions run on main, there appears to a build flake related to linking Nexus.

In the logs, this issues presents as the following:

Running `rustc --crate-name omicron_nexus ...
error: linking with `cc` failed: exit status: 1
...
  = note: collect2: error: ld returned 1 exit status
error: could not compile `omicron-nexus` due to previous error

Caused by:
  process didn't exit successfully: `rustc --crate-name omicron_nexus ...`

AFAICT, the first sign of this issue happening is roughly co-located with the following PRs:

• First appeared in https://github.com/oxidecomputer/omicron/actions/runs/2078057645 , in @jgallagher 's merge of MGS: Finish extracting SP communications to its own crate #786
• This was preceded by a handful of changes by @davepacheco for the "new internal lookup API"

However, given the flaky nature of the build, it is difficult to determine a root cause quite yet.

@jgallagher , @davepacheco - any chance this error looks familiar? Any known changes to linkage?

[bnaecker]

It doesn't look exactly the same, but I've run into this error most often when running out of memory during linking. This sort of fits with the error message in link to the "first failed run":

You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 63 MB

[jgallagher]

@bnaecker Where do you see that error? But certainly "link failed due to not enough disk space" sounds like a potential cause.

[bnaecker]

@jgallagher That's at the bottom of this page: https://github.com/oxidecomputer/omicron/actions/runs/2078057645, under the "Annotations" section for the build-and-test (ubuntu-18.04) run. I.e., not the captured output from the run.

[davepacheco]

@jgallagher , @davepacheco - any chance this error looks familiar? Any known changes to linkage?

I didn't see any such errors during my recent work on the lookup API. I used to see these more often until I switched to a system with more memory and added a swap device.

[jgallagher]

Any objecting to adding an "output disk space" step to CI a la 12d3296? Might narrow down if the flake looks space related.

[smklein]

Any objecting to adding an "output disk space" step to CI a la 12d3296? Might narrow down if the flake looks space related.

I think that's a great idea, more visibility in the build would help.

[jgallagher]

Maybe a bit more of a smoking gun that this is disk space related:

https://github.com/oxidecomputer/omicron/actions/runs/2097859388 is a failure on main. The pre-build step that reports disk usage claims there's 32GiB free. We made it through building (including linking), and the job appears to have died while running tests. The annotations on the run itself include

Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.289.2/_diag/Worker_20220405-174524-utc.log'

[jgallagher]

It looks like #880 fixed this:

1. We saw some evidence this was disk space related, and that PR increased the available space by > 50%.
2. Between the first failure and the PR being merged, there were 14 failures and only 3 successes of the Rust workflow on main.
3. Since the merge, there have been 7 successes and only 1 failure, which looks like an ephemeral external-service-related issue.

Based on that I'm going to close this, but please reopen if you disagree or want to keep an eye on it for longer.

[smklein]

Thanks @jgallagher for following through on this! Very much appreciated

[davepacheco]

That all sounds good -- and also it seems likely we're going to run into this again. We probably don't need to do anything now but it maybe should be on our radar to think about moving away from GitHub Actions.

[davepacheco]

Thanks for digging in and fixing this, @jgallagher!