Add native support for AWS ECR

[aledbf]

Is your feature request related to a problem? Please describe

Provide native support to use ECR as a container registry and also allows the use of IAM Roles for Service Accounts (in pods)

Additional context

https://aws.amazon.com/es/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

[kylos101]

👋 @csweichel today you mentioned this would impact image-builder-mk3 and registry-facade. I'll leave this in a breakdown status, to show this requires investigation.

@jenting if you get time this week, after finishing #14219 and self-hosted release testing, could you help breakdown this issue? @sagor999 is out this week, and it would be good to get a head start, if we can.

@sagor999, when you return from vacation, may we ask you to investigate and socialize a proposed solution with @csweichel? Ideally we'd have a prototype working the week of Dec 2, and have it polished the week of Dec 9.

I spent some time investigating image-builder-mk3 👇.

image-builder-mk3

Possible idea

Have bob proxy (which runs in a separate network and mount namespace [1][2][3]) exchange a service account OIDC token for AWS role credentials via STS, and use the credentials to push to ECR.

1. In workspacekit, read the service account file's token into memory, hide the service account file from the workspace container somehow (see protecting the service account, below), and store it as an environment variable beginning with WORKSPACEKIT_ (so it's not added to the workspace envs)
2. bob proxy reads the new WORKSPACEKIT_<var> environment variable to get the service account token.
3. bob proxy exchanges the service account token for an AWS credential using awscli
4. bob proxy uses the credential to do auth for pushing images.

Protecting the service account

If we add the service account to the imagebuild pod, it'll be available to the workspace container's file system. We cannot delete the service account from the file system because it is read-only. But, we need to prevent bob build from being able to access&use the service account for image build workloads, while allowing bob proxy to exchange the service account for an AWS role credential. 🤔

Perhaps we can do something with ws-daemon, to change the mount for the service account on the workspace container? For example, after workspacekit prepares the WORKSPACEKIT_<var> environment variable, alter the mount such that it points at an empty file (similar in concept to this, but done at runtime)? 🤷

Limitations

We'll need to check for STS limitations. For example, say we have two or more image builds running, can they all negotiate credentials and push to ECR at the same time using the same service account? It's possible (1) subsequent exchanges invalidate prior credentials, causing image pushes to fail (2) subsequent exchanges fail until the first one is expired or enters a grace period, causing image pushes to fail (and perhaps forcing us to cache and centralize negotiated credentials).

[utam0k]

Come to think of it, we do have experience with EKS. Maybe we can give ourselves some hints?
https://github.com/gitpod-io/amazonlinux-testing

[utam0k]

@aledbf In terms of the registry-facade, the API basically follows the OCI distribution spec, so there should be no need to make any changes to that part of the API, right? Shouldn't there be an impact on IPFS?
Is it correct that the part that needs to be changed is the authentication and authorization part that uses the API?

[atduarte]

Duplicate of #12104