Investigate how to add support for AWS ECR

[atduarte]

Investigate how to achieve #14891 #12104

• build eks environment (We use the preview env with private AWS ECR)
• setup private ECR
• setup identity in IAM with proper roles
• setup service account in EKS to use identity from ☝
• run some experiments to determine a design that works. Options:
  • ...define an ECR secret that rotates every 12h (because it expires after that amount). See this for a summary and follow the link for background.
  • ...setup a service account, and use it to get a JWT with registry-facade and image-builder-mk3
  • ...setup a service account, and use it to get a JWT with blobserve

The AWS SDK will be helpful for both. To prototype sooner, it might be easier to start off by experimenting using the AWSCLI.

[atduarte]

@kylos101 suggestion for image-builder-mk3:

Possible idea

Have bob proxy (which runs in a separate network and mount namespace [1][2][3]) exchange a service account OIDC token for AWS role credentials via STS, and use the credentials to push to ECR.

In workspacekit, read the service account file's token into memory, hide the service account file from the workspace container somehow (see protecting the service account, below), and store it as an environment variable beginning with WORKSPACEKIT_ (so it's not added to the workspace envs)
bob proxy reads the new WORKSPACEKIT_ environment variable to get the service account token.
bob proxy exchanges the service account token for an AWS credential using awscli
bob proxy uses the credential to do auth for pushing images.

Protecting the service account

If we add the service account to the imagebuild pod, it'll be available to the workspace container's file system. We cannot delete the service account from the file system because it is read-only. But, we need to prevent bob build from being able to access&use the service account for image build workloads, while allowing bob proxy to exchange the service account for an AWS role credential. 🤔

Perhaps we can do something with ws-daemon, to change the mount for the service account on the workspace container? For example, after workspacekit prepares the WORKSPACEKIT_ environment variable, alter the mount such that it points at an empty file (similar in concept to this, but done at runtime)? 🤷

Limitations

We'll need to check for STS limitations. For example, say we have two or more image builds running, can they all negotiate credentials and push to ECR at the same time using the same service account? It's possible (1) subsequent exchanges invalidate prior credentials, causing image pushes to fail (2) subsequent exchanges fail until the first one is expired or enters a grace period, causing image pushes to fail (and perhaps forcing us to cache and centralize negotiated credentials).

[kylos101]

@sagor999 a second option, is option 2 here, where we support ECR, but rotate a secret on a schedule (say every 12h) rather than leverage a service account to retriever a JWT token at runtime.

[kylos101]

@sagor999 moving to in-progress as this is actively being worked.

Also, also a heads up, the werft jobs you used to produce AWS infra will "dispose" of the AWS infra after a time period (I think 24h).

So, if you need it to persist, we should follow-up with @adrienthebo. I believe he can help us "keep it around" until we're ready to delete it.

[csweichel]

image-builder, bob proxy and registry-facade all serve their auth using the dockerconfigjson format. Bob proxy requires this data to be in a secret so that we need not pass this data to image build workspaces.

All hiding from actual workloads is already implement in bob proxy, nothing needs to change here.

We already have automatic reloading of the relevant secrets in image-builder and registry-facade (#14679). All we need now is something that updates those secrets. @mrzarquon produced a first version of such a mechanism: https://github.com/mrzarquon/ecr-registry-refresh/blob/wip/k8s/main.py#L42-L46. The equivalent function in Go is GetAuthorizationToken.

Given the far reaching permissions required to update the secret (get and patch/update on secrets within the namespace), it seems prudent to run this updater as separate deployment rather than give the image-builder SA such wide ranging permissions.

[kylos101]

@jenting, may we ask for your help with this issue? Also, let me know if you have AWS experience or not. If you do not, we might want to see if we can get you some sync time with someone who does, to help serve as a primer/shortcut.

I requested AWS access for you. 🙏

This comment from Chris describes an ideal solution that we could use. Let's catch up when you return on Monday?

[jenting]

@jenting, may we ask for your help with this issue? Also, let me know if you have AWS experience or not.

Yes!

This comment from Chris describes an ideal solution that we could use. Let's catch up when you return on Monday?

I will

• bootstrap a AWS EKS cluster with terraform + kots to install the Gitpod.
• use the repo https://github.com/nuxeo-cloud/aws-iam-credential-rotate#rotate-ecr-credentials to proof and test the ECR credential rotation works as expected.

@csweichel will we garbage collect the old container image? I ask because I know we will remove the workspace content from object storage once the workspace is removed over X days. But I am not sure about the behavior of the container image garbage collection part.

[kylos101]

@jenting does ECR have any GC capabilities that are "native"? For example, if an image isn't pulled after X days, delete it.

Seems like there's no official support yet.

[kylos101]

@jenting internally, you mentioned:

, I will use the repo and run it as a sidecar container to periodically rotate the ECR credential.

A sidecar in which pod? Why a sidecar? I ask because of @csweichel 's comment:

it seems prudent to run this updater as separate deployment rather than give the image-builder SA such wide ranging permission

For example, why not run a separate deployment or cronjob?