[installer] Support adding image pull secrets even when the internal (in-cluster) registry is used

[corneliusludmann]

Currently, we can add image pull secrets only when an external registry is used like this:

containerRegistry:
  inCluster: false
  external:
    url: my-registry.example.com
    certificate:
      kind: secret
      name: my-registry-credentials

However, when we use an in-cluster registry, customers would probably still want to add pull secrets for workspace base images.

I would suggest to deprecate the certificate block under external and moving it directly under containerRegistry. Like this:

containerRegistry:
  inCluster: false
  external:
    url: my-registry.example.com
  certificate:
    kind: secret
    name: my-registry-credentials

(probably rename “certificate” to “credentials”, however, we probably want to keep this consistent to other occurences of “credentials”, e.g. for the object storage)

That would also allow adding workspace pull secrets when the in-cluster registry is used like this (see also #10791):

containerRegistry:
  inCluster: true
  certificate:
    kind: secret
    name: my-registry-credentials
  privateBaseImageRegistries:
    - my-registry.example.com
    - my-other-registry.example.com

In this case, we need to merge the in-cluster container secret with the secret given here to allow accessing both registries.

• code pointer
• see also this change where we do this in KOTS already. Once we have this natively supported in the installer, we can replace this hack in KOTS.
• See also - internal discussion about this

[lucasvaltl]

• We should do this soon - we created tech debt to solve a customer problem, but should solve this properly via this issue.

[lucasvaltl]

Internal mention of this here

[mrzarquon]

Allow certificate or a dockerconfig, since uploading a dockerconfig directly from a file means they can validate that they've set the correct credentials with a docker push/pull, before uploading the file.

Allow for multiple secrets and we merge them for one dockerconfig

[Pothulapati]

Started working on this. The planned resolution is to move credentials to be under containerRegistry so that it can specified irrespective of the type of container registry i.e internal or external.

Coming to the requirements that @mrzarquon specified,

Allow certificate or a dockerconfig, since uploading a dockerconfig directly from a file means they can validate that they've set the correct credentials with a docker push/pull, before uploading the file.

We can only specify setting a certificate (aka secret) as taking a dockerconfig directly means that the we might be storing auth credentials in the config (which has complexities, even if we mask them). But, As the certificate is created by the user using the .dockerconfig itself, They can always try doing the push/pull before creating the secret itself.

[Pothulapati]

Created #12060

[lucasvaltl]

@Pothulapati does #12060 replace this one in that case, let's close this one and prioritise the other one :)

[Pothulapati]

@lucasvaltl This is still needed, as #12060 only tackles the components side of things, but we still need to move this config in the installer (but only after that is done)