Allow image-builder to consume and publish to different image registries with different docker credentials

[Pothulapati]

Problem

The image builder can only reference one secret object, and though that object can list 20 repositories in it and their corresponding credentials, we have no facility to manage or merge / combine multiple user provided registry credentials into a secret object.

This means that the image-builder cannot pull an image from one registry that requires credentials and then push to another different registry that requires credentials (It can if one of these is public). This is an issue for those cases where a customer is trying to get the workspace base image from one registry and them push the built image for a workspace (that adds customisations on top of the base image) to another image registry ready to be used to start workspaces. In practice, this means you cannot use an in-cluster registry to store workspace images and a private (i.e. one that needs credentials) registry to pull base images from. This is an issue especially in airgapped environments that use the in-cluster registry (we recommend AWS users to use the in-cluster registry backed by s3 for storage), which usually always use a private registry for base images.

❓ Why do we recommend using the in-cluster repository and not the other existing one?

We don’t support all registries for pushing to them (we really only test against GCR and Docker Registry, i’ve not got workspace-image push working with artifactory or harbor), they don’t want workspace-images (multi gigabyte images) stored in their registry space optimized for serving small containers very quickly, or where they have to pay per image push for scanning / auditing, etc

❓ Why don't we support all registries?

It wouldn't also solve the issue, but to your question for example, we lack ECR support because:

1. we need to do an AWS api call to create workspace-images manually before we can push an image to it
2. ECR credentials need to be refreshed every 12 hours, there is a beta kubelet feature to make this easier to do, but that won’t directly help registry facade. @mrzarquon did POC using a batch job to refresh the credential in a dockerconfigjson object

Technical details — Self-hosted team

With #10792, We started investing on ways to allow docker credentials to be passed by the user even when incluster registry is enabled, to allow for workspace base images to be pulled from different (& usually private) registries.

As the above issue describes, The fix initially seemed simple i.e to move credentials out of external so that a registry credentials secret can be passed even when an external registry is not being used.

Roadblock

Internally, The current way this works is in a either or approach i.e pass the secret if external registry is used, or pass builtin-registry-credentials if inclusterRegistry is enabled.

Now, We need a way to pass multiple secrets and thats where it gets tricky, as the current components (that need the registry communication) configs seem to take only a single dockerConfig file.

The installer's conventions doesn't let us merge these things for the following reasons:

• We can't take credentials directly in the config, as we have been using secrets as a way of any kind of credentials store (storage, certificates, etc)
• We can't also take a list of secrets, and merge them as the installer by design cannot read something from the cluster.

Potential solution

Components should instead have the ability to use multiple files, merging them before using them as a registry. This feels right as it confirms to the installer conventions, where the instalelr would just mount these secrets correctly.

Describe alternatives you've considered

As discussed above, the alternatives would break conventions but were considered.

Additional context

• [installer] Support adding image pull secrets even when the internal (in-cluster) registry is used #10792
• Slack Discussion 1: https://gitpod.slack.com/archives/C01KLC56NP7/p1660139003779169
• Slack Discussion 2: https://gitpod.slack.com/archives/C02F19UUW6S/p1660231293408299

[corneliusludmann]

Please correct me when I'm wrong, @Pothulapati. That is what I understood:

It seems the following components using the registry auth credentials:

• blobserve
  

  gitpod/install/installer/pkg/components/blobserve/deployment.go

  Lines 27 to 33 in eca8d8a

  	if pointer.BoolDeref(ctx.Config.ContainerRegistry.InCluster, false) {
  	secretName = dockerregistry.BuiltInRegistryAuth
  	} else if ctx.Config.ContainerRegistry.External != nil {
  	secretName = ctx.Config.ContainerRegistry.External.Certificate.Name
  	} else {
  	return nil, fmt.Errorf("%s: invalid container registry config", Component)
  	}

• image-builder-mk3
  

  gitpod/install/installer/pkg/components/image-builder-mk3/deployment.go

  Lines 26 to 32 in eca8d8a

  	if pointer.BoolDeref(ctx.Config.ContainerRegistry.InCluster, false) {
  	secretName = dockerregistry.BuiltInRegistryAuth
  	} else if ctx.Config.ContainerRegistry.External != nil {
  	secretName = ctx.Config.ContainerRegistry.External.Certificate.Name
  	} else {
  	return "", fmt.Errorf("%s: invalid container registry config", Component)
  	}

• registry-facade
  

  gitpod/install/installer/pkg/components/registry-facade/daemonset.go

  Lines 96 to 102 in eca8d8a

  	if pointer.BoolDeref(ctx.Config.ContainerRegistry.InCluster, false) {
  	secretName = dockerregistry.BuiltInRegistryAuth
  	} else if ctx.Config.ContainerRegistry.External != nil {
  	secretName = ctx.Config.ContainerRegistry.External.Certificate.Name
  	} else {
  	return nil, fmt.Errorf("%s: invalid container registry config", Component)
  	}

The problem is that the components allow using the built-in registry auth OR the external registry auth but not both. But customers that want to pull workspace images from a registry that needs auths (instead of Docker Hub) need to add an additional registry secret. That happens regularly when using air-gap because the workspace images will be mirrored in a local registry that is accessible in the air-gapped network.

We cannot merge the built-in registry auth and the user-given registry auth in the installer because the installer does not have access to the user-given registry auth (which is a K8s secret) during the rendering phase (render does not have access to the cluster).

Question (for Team Workspace?): Which of the components listed above need to pull the workspace images? Only image builder? Components that need to pull workspace images need to accept two secrets (for the built-in registry as well as the user-given secret).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.