Allow for Prebuilds on external build systems

[lucasvaltl]

Is your feature request related to a problem? Please describe

This requires a lot more research - this issue is aimed at capturing this idea.

Currently, Gitpod acts as a build system for our customers when creating prebuilds on certain events (e.g. when a new PR is merged). This further increases the complexity of understanding, installing and operating Gitpod for customers - especially if they already have a build system in place like Github actions or CircleCI.

Describe the behaviour you'd like

As an additional option, we could allow customers to execute prebuilds in Github Actions or Circle Orbs.

Additional context

• This came up in a meeting here.

[atduarte]

There is an important characteristic that makes us want to run prebuilds as we do: whatever runs (or fails) in the workspace, must run (or fail) in a workspace. Environments must be as similar as possible to ensure a great experience.
A prebuild is in fact just a workspace in headless mode. We backup it up, as we do backup a workspace that stops. This is very powerful because it will use the same environment variables, have the same network access, same underlying kernel, same type of filesystem, same permissions, …

Doing otherwise would make users have to spend their time figuring out why there are things running in one place and not the other; solve network issues; and would make us miserable because we would have to support whatever CI the client uses.

---

I understand that it makes explanations harder. Why does it increase the complexity of installing and operating Gitpod though? 🤔

[lucasvaltl]

Agree that explaining and understanding Gitpod is the biggest driver here. The other two are secondary and not that impactful:

• Installing = if we go down the rout of separate node pools for prebuilds, that is one more thing to have to create / manage
• Operating = another thing to be maintained and that needs to work. Granted, when using external build systems there is complexity with that as well.

@metcalfc had brought this up initially, maybe he can weigh in?

[mrzarquon]

I provided an example workflow that I'd want to see us have better support for in the docs section for the Dockerfile vs gitpod.yml open issue.

I don't see myself using Dockerfiles in any workspace long term, once I've got my dockerfile setup and working, I'd rather move that image build process to my CI system so that can happen at a different frequency than a prebuild process (ie, only build a new image when the repo has been tagged). That frees cycles for Gitpod to then do the highly context sensitive prebuild / creation of a workspace steps for us.

One downside is currently the image reference in a gitpod.yml is static, so I have to update the yaml whenever there's a new tagged image, or pin to latest and hope Gitpod refreshes its cache. I picture a feature where once could flag a project as "CI Assisted" that results in the following:

• When gitpod receives a prebuild webhook, if it is a CI Assisted project, it waits for a Gitpod Notifiy step to run in CI
• The notify step sends the CI job ID (to expose errors if the CI job fails) and predicted tag / final asset ID
• Gitpod waits for a second "done" step in CI to fire
• Gitpod pulls the image made by CI, and performs its Prebuild steps

Making it possible for Gitpod to incorporate a freshly built by CI image gets us part way there to external prebuild support - if you want to run something in existing CI, put it in your dockerfile/ci pipeline, if you want gitpod to use it, put it in the gitpod.yaml.

[metcalfc]

There is also some degree of duplication.

• CI solution builds on every commit
• Gitpod builds on every commit
  The user is paying for that compute twice. They could add the stage to create a workspace to their existing CI pipeline.

[atduarte]

I'd rather move that image build process to my CI system so that can happen at a different frequency than a prebuild process

@mrzarquon maybe I misunderstood what you are saying, but the image is only rebuilt if its definition (Dockerfile) changes or you force it,IIRC 🤔

[mrzarquon]

@atduarte That's part of the problem - the Dockerfile could load dynamic data so it could be run on a schedule to get the latest updates without having to modify the Dockerfile itself, my gitpod.yml setup to consume a 'gitpod' tag, but there's no method for me to notify Gitpod to refresh its besides loading that rebuild url in the browser (its not exposed as an curl-able accessible endpoint). To automate the image rebuild, I'd need to then do an auto-pr updating the .gitpod.yml and merging to main (and hope I don't create an infinite loop of CI jobs).

You might wonder why I would do that for a single project - and I wouldn't - but for an image consumed by 100 projects, I'd want a central way to bump versions of tools in the image and then know all downstream projects pick it up, without having to hound the maintainers of 100 repositories to bump their .gitpod.yml or their Dockerfile just to have Gitpod rebuild from it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.