CI: Fix random failures. Merge workflows

[olethanh]

This PR fix the failure that have been happening recently on the CI on the Digital Ocean Droplet VM, and that prevented the PR from passing and being merged.

This is accomplished in two way:

1. Simplify the way we use GitHub CI.
2. fix the interactions between the CI and the Digital Ocean API.

Some minor improvement and documentation are also included.

Related ClickUp, GitHub or Jira tickets : Jira ALEPH-499

Changes

United workflow

This PR introduce a big architectural change as it merge the following workflows into one:

• Build packages
• Test new runtime and examples
• Testing on DigitalOcean Droplets

This merge allow the reuse of packages build in the previous steps for the droplet test, reducing the number of package built from 13 to 3.

Advantages:

• fix package build failure because of rate limitation on other services [1].
• Reduce the chances of random errors.
• Reduce the total CI times requirement.
• Do not attempt to run the droplet test if the package building phase fail. Previously all the package build were launched in parallel which mean they all failed at the same point unnecessarily)
• Make it less costly and faster to run the failed jobs/steps.

Mainly it reduce the chances of CI failure due to external causes and make the failing step more apparent.

Digital Ocean Changes

It seem Digital Ocean changed the way the Droplet were provisioned and the command doctl compute get which we use to retrieve the newly created droplet ip.

The command sometime returned without the droplet info or without the network information set, presumably before the setup was finished.
Solution: A workaround were added against this, we now wait and run the command till it return the proper info.

Second issue is that we sometime got another IP, the one on the internal network, that could not be reached from the GitHub runner.
As the IP was recalculated between some steps, it make some steps fails while other worked.
Solution: 1. the IP is calculated only once and saved in a Github CI env var
Solution: 2. The code ensure the Public one is used.

Self proofreading checklist

• The new code clear, easy to read and well commented.
• New code does not duplicate the functions of builtin or popular libraries.
• An LLM was used to review the new code and look for simplifications.
• New classes and functions contain docstrings explaining what they provide.
• [n/a] All new code is covered by relevant tests.
• Documentation has been updated regarding these changes.
• [n/a] Dependencies update in the project.toml have been mirrored in the Debian package build script packaging/Makefile

How to test

it is not relevant to test manually locally.

Rebase your Pull Request on top of this branch or master once it is merged.

Print screen / video

Yay a pretty arrow

Notes

Do not squash this branch!

I reworked each commit to make them documented and self explanatory. The Github workflows file format is quiet unreadable and the commit history help.

[1] e.g.

curl -fsSL https://github.com/ipfs/kubo/releases/download/v0.33.2/kubo_v0.33.2_linux-amd64.tar.gz | tar -xz --directory ./target/kubo
curl: (22) The requested URL returned error: 403

That error happened randomly and was determined to be caused by rate limiting.

We also had previous failure with the Docker registry that prevented the use of the vm-connector docker image

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.33%. Comparing base (73013a2) to head (4396a91).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #787   +/-   ##
=======================================
  Coverage   63.33%   63.33%           
=======================================
  Files          77       77           
  Lines        6854     6854           
  Branches      576      576           
=======================================
  Hits         4341     4341           
  Misses       2325     2325           
  Partials      188      188           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.