Split the Gitpod app from the Workspace images in the KOTS installer

[mrzarquon]

Is your feature request related to a problem? Please describe

The latest airgap bundle is now 11GB, because of how the Replicated web installation works, users are required to upload the entire .airgap file to the UI even if they've already staged the images. This is because the application payload is in the same bundle as the images.

The massive airgap bundle size means that doing any deployment with airgap is slow and difficult, especially when using the web UI which is required if you're not familiar with the CLI only methods to perform an installation (or upgrade).

ie, intellij java image is 3gb, what use of this is to a vscode only installation?

Related to #10298 as we'd need to support having different registries for different purposes in Gitpod - one for workspaces vs one for hosting Gitpod's specific images.

Describe the behaviour you'd like

Only the very minimal base image should ship with the installer, so that upgrades and installations don't require pushing > 8gb of unnecessary images. The configuration of the default image and registry should be exposed as well, so users can change this as they go.

Documentation on how to pull/push other workspace images should be provided as an alternative. Workspace images could be populated with other registry mirroring systems since in essence it is requiring the users to grab latest of the workspace images (which is what we do when we build an airgap anyway).

Related Context

• Internal RFC related to this

[mrsimonemms]

When this ticket is accepted, the first stage should be to download the bundle, remove the workspace-full image from the tar file and recompress it. From memory, I did this when I first did the airgapped work and I don't recall this reducing the size by much - I think it was only about 1GB.

Personally, I don't think we should invest much time in this if the potential results are not significant (we can debate what "significant" means of course)

[mrzarquon]

Removing all IDEs except vscode and switching to workspace-base instead of full, I was able to re-compression the airgap bundle to 2.7gb. The VSCode web IDE image is 700mb while the VScode remote image is 14mb, so doesn't make sense to remove the remote image. That would get to a "hello $gitpod" level of use.

[corneliusludmann]

Experimented with a script that creates an air-gap bundle with images that have changed between two releases only: https://github.com/gitpod-io/gitpod/compare/clu/diff-airgap-bundle

Still in an experimental state. However, looks quite promising. We could add something like this to our build pipeline to create airgap diff bundles for each release.

[mrsimonemms]

I had a bit of brainwave about this the other day and suspect this might be a good way to go.

The first step would be to remove all the additionalImages in the Werft job. This would reduce the airgap bundle to it's smallest size possible and would contain just the release details, plus the images that are contained in the KOTS /manifests folder. (One thing of note - any additionalImages specified in the source code will need to be kept as it's only way of telling it about the Helm images).

The second step would be to do something with the mirror command in the Installer. Currently, we have two commands there:

• mirror kots - this generates all the possible permutations of images that can be generated and then updates the additionalImages in the kots-app.yaml file. This should probably be now thought of as deprecated, but there's some useful stuff in there we may use
• mirror list - this generate the images for a specific config. This is the original way of doing the airgapping (I wrote it on my balcony at the off-site) and expects the user to write a script to pull the images, rename them and then push to the new registry (there's an example for this is the docs). Whilst it's not recommended, it's the non-KOTS way of airgapping, so this is NOT deprecated

Realistically, what we need is something between the two - perhaps a mirror images command. Like the mirror kots command, it would produce all the permutations for the images in use. But, like the mirror list command, it would write the new tag names.

My suggestion is to have a mirror full command. This would receive the registry URL only and do the following stuff:

1. generate the full list of images, as per mirror kots
2. create the new list of image tags and output it as per mirror list

The example would then be as-per that in the mirror list command so that the user's script would download the image from the public registry, tag it and then push to their private registry.

This would have the effect of reducing the size of the airgap bundle to it's smallest possible size and, because Docker pull/push is correctly cached, it would reduce the time in downloading unnecessary images.

[corneliusludmann]

Thanks for your input, Simon. I would like to add my thoughts on some weak points of this approach that we should consider for the final concept.

The first step would be to remove all the additionalImages in the Werft job. This would reduce the airgap bundle to it's smallest size possible and would contain just the release details, plus the images that are contained in the KOTS /manifests folder.

That means even for the first install, you need to push the air-gap bundle and need the additional step of pushing the Gitpod images.

The example would then be as-per that in the mirror list command so that the user's script would download the image from the public registry, tag it and then push to their private registry.

Could we expect, that there is always a computer that has access to the public registries as well as the private registries (in order to pull and push images)? I mean, downloading an image bundle and copying it via scp to a machine in the air-gap network (and pushing it from there) or uploading it to the KOTS admin UI should be always possible. But pulling images from a machine in the air-gap network will not work and I don't know how hard it would be for the customer to expose the internal registry to a trusted computer that has access to the internet to push images from there.

Mitigation: You export the images on your local computer and copy them to the machine in the air-gap network to push them.

This would have the effect of reducing the size of the airgap bundle to it's smallest possible size and, because Docker pull/push is correctly cached, it would reduce the time in downloading unnecessary images.

The weak point here is that you rely on the local Docker cache. If you prune the Docker cache of your laptop or another colleague is pulling the images, then the cache will not help, right?

With my experiment above, I created a list of images that differ between 2 versions. That would solve the issue of relying on the Docker cache.

[corneliusludmann]

Experimented with a script that creates an air-gap bundle with images that have changed between two releases only: https://github.com/gitpod-io/gitpod/compare/clu/diff-airgap-bundle

The biggest issue with this approach is that we need to build such a bundle that is customer-specific. We could prepare it in a werft job but adding the user-specific app.tar.gz needs some kind of web service (or we need to build a dedicated bundle for each customer in advance and upload it to a bucket).

Alternative approach: We let build a minimal air-gap bundle with KOTS (without the additional images) as Simon proposed above. Additionally, we build a small CLI tool that pulls the images that have changed between 2 versions (the diff – similar to my experimental script above) and that adds these images to the downloaded air-gap bundle (with tar). With that approach, they would get a personalized, minimal air-gap bundle that they can copy / upload and use with KOTS. That would move the air-gap bundle creation from a web service to the local computer of the customer.

Advantage:

• Does not rely on Docker cache (since it calculates diff between releases).
• Does not need direct push access to the local registry from the same machine that pulls the images, keeps full KOTS air-gap bundle experience (since it adds the images to the KOTS air-gap bundle instead of pushing them directly).
• Does not need a web service on our side (since it's a CLI tool that runs on customers' computers).

Still a weak point: Needs an additional step for all installations (even the first one).

Disadvantage: We need to build and distribute a CLI tool (however, could also be part of the installer if we really want to avoid this).