[content-service] Add support to use existing S3 bucket #10073

aledbf · 2022-05-17T18:04:14Z

Description

This PR introduces a new "single bucket mode" to the S3 storage implementation.

In some scenarios, we could not have enough permissions to create buckets. In those cases the "one-bucket per user" implementation fails.

If there's a bucket name configured (see installer/content-service config change), that bucket name is used and the user ID is prefixed to the object name. If no such bucket name is configured, the old "one bucket per user" behaviour kicks in. This way, users can maintain backwards compatibility for now. There is no automatic migration or "hybrid mode" where we fall back to the old behaviour. We don't have enough installations in the wild which were able to actually use the old behaviour to justify introducing such a mode.

We maintain the old mode in GCP (and maybe even S3) for now because

it maintains backwards compatibility
it allows for region-local optimisations. That said, we could just as well introduce single regional buckets instead.
moving away from it would require migration for gitpod.io

How to test

Precursor: ensure the installation uses minio/S3, and that the bucket is configured

start/stop a workspace, make sure the backup happens and is restored
take a snapshot and open a workspace from it
run a prebuild and open a workspace from it
use the VS Code settings sync, make sure it still syncs
check out prebuild logs in the dashboard

Release Notes

[content-service] Add support to use a single S3 bucket

Fixes #10070

aledbf · 2022-05-17T22:54:17Z

/werft run

👍 started the job as gitpod-build-aledbf-bucket.7
(with .werft/ from main)

aledbf · 2022-05-17T23:47:42Z

Configuration example:

objectStorage:
  inCluster: false
  kind: minio
  s3:
    endpoint: minio.default.svc.cluster.local:9000
    accessKey: GlrlMt5e40dnW0GhveAX
    secretKey: 0L6XW0.GWcD8CBMseV31
    
    bucket: some-valid-bucket

corneliusludmann

Awesome to see this change! 🚀

Added a few comments regarding backward compatibility. 🤔

components/content-service/pkg/storage/minio.go

install/installer/pkg/config/v1/config.go

corneliusludmann · 2022-05-18T12:39:25Z

Follow-up issue: Add config for S3 bucket to KOTS (Replicated) installer.

corneliusludmann · 2022-05-18T12:41:16Z

Follow-up issue: Document S3 bucket config in self-hosted docs

sagor999 · 2022-05-20T00:32:08Z

components/content-service/pkg/storage/minio.go

 }

 func (rs *DirectMinIOStorage) objectName(name string) string {
-	return minioWorkspaceBackupObjectName(rs.WorkspaceName, name)
+	return minioWorkspaceBackupObjectName(rs.Username, rs.WorkspaceName, name)


Suggested change

return minioWorkspaceBackupObjectName(rs.Username, rs.WorkspaceName, name)

var username string

if rs.MinIOConfig.BucketName != "" {

username = rs.Username

}

return minioWorkspaceBackupObjectName(username, rs.WorkspaceName, name)

that way if bucket name is set, then we default to new way of doing things, otherwise we maintain backwards compatibility to old way of accessing buckets.

sagor999

can you please add unit tests that would ensure old code is working correctly, and new one also generates correct bucket and path.

sagor999 · 2022-05-23T21:28:06Z

@corneliusludmann @aledbf I added tests and fixed directMinio and presignedMinio access.
Tested in preview env, and made sure that now all content indeed is being written into dedicated bucket if it was set in installer config.

Please double check tests that I added to ensure they are correct.

This also ensures backward compatibility (rolling out this update should not break any existing self hosted installations).
But this feature should cannot be enabled on existing installations without losing all data or manually moving old buckets into new bucket.

sagor999

with tests it looks good now (feels weird approving my own changes to PR 😃 )

corneliusludmann

Looks good. Thanks, @aledbf and @sagor999! 🚀

corneliusludmann · 2022-05-24T12:36:32Z

components/content-service-api/go/config/config.go

@@ -105,6 +105,8 @@ type MinIOConfig struct {

 	Region         string `json:"region"`
 	ParallelUpload uint   `json:"parallelUpload,omitempty"`
+
+	BucketName string `json:"bucket"`


Suggested change

BucketName string `json:"bucket"`

BucketName string `json:"bucket,omitempty"`

Do we need omitempty here in case the value is not given or for backward compatibility reasons?

I don't think so but wanted to raise this question here just in case.

@corneliusludmann good catch! Since that param is optional, I added omitempty to it.

akosyakov

lgtm

mustard-mh

LGTM in supervisor mod 🙈

roboquat added do-not-merge/work-in-progress release-note size/S labels May 17, 2022

aledbf force-pushed the aledbf/bucket branch from 7fccf34 to 4730640 Compare May 17, 2022 18:05

roboquat added size/M and removed size/S labels May 17, 2022

aledbf force-pushed the aledbf/bucket branch 4 times, most recently from 8519e01 to 5401386 Compare May 17, 2022 22:38

roboquat added size/L and removed size/M labels May 17, 2022

aledbf force-pushed the aledbf/bucket branch from 5401386 to e0af1e5 Compare May 17, 2022 22:44

aledbf force-pushed the aledbf/bucket branch from e0af1e5 to 4b434ea Compare May 17, 2022 23:05

sagor999 mentioned this pull request May 17, 2022

Use a single object storage bucket per Gitpod installation #10070

Closed

aledbf marked this pull request as ready for review May 17, 2022 23:47

aledbf requested review from a team May 17, 2022 23:47

aledbf requested review from csweichel and geropl as code owners May 17, 2022 23:47

roboquat removed the do-not-merge/work-in-progress label May 17, 2022

github-actions bot added team: IDE team: delivery Issue belongs to the self-hosted team team: workspace Issue belongs to the Workspace team labels May 17, 2022

corneliusludmann requested changes May 18, 2022

View reviewed changes

components/content-service/pkg/storage/minio.go Show resolved Hide resolved

components/content-service/pkg/storage/minio.go Outdated Show resolved Hide resolved

install/installer/pkg/config/v1/config.go Show resolved Hide resolved

corneliusludmann mentioned this pull request May 18, 2022

Add config for S3 bucket to KOTS (Replicated) installer #10100

Closed

aledbf force-pushed the aledbf/bucket branch from 4b434ea to f1542a9 Compare May 19, 2022 06:59

sagor999 reviewed May 20, 2022

View reviewed changes

sagor999 suggested changes May 20, 2022

View reviewed changes

nandajavarma mentioned this pull request May 23, 2022

Add S3 bucket name to kots config #10188

Merged

roboquat added size/XL and removed size/L labels May 23, 2022

sagor999 force-pushed the aledbf/bucket branch from 450178e to 3d8de92 Compare May 23, 2022 20:27

sagor999 approved these changes May 23, 2022

View reviewed changes

corneliusludmann approved these changes May 24, 2022

View reviewed changes

akosyakov approved these changes May 24, 2022

View reviewed changes

mustard-mh approved these changes May 24, 2022

View reviewed changes

aledbf and others added 5 commits May 24, 2022 19:37

[content-service] Add support to use existing S3 bucket

689aa9b

Update go modules

53d61ee

[content-service] add tests to minio provider

c1bb7fd

[content-service] fix s3 access when using dedicated bucket

387d6bb

[content-service] add omitempty to BucketName for minio config

2bd5d6f

sagor999 force-pushed the aledbf/bucket branch from 72f3eab to 2bd5d6f Compare May 24, 2022 19:37

roboquat merged commit b053dae into main May 25, 2022

roboquat deleted the aledbf/bucket branch May 25, 2022 07:48

roboquat added the deployed: IDE IDE change is running in production label May 25, 2022

lucasvaltl mentioned this pull request Jul 6, 2022

Limit the scope access needed for S3/Blob storage #9484

Closed

-	return minioWorkspaceBackupObjectName(rs.Username, rs.WorkspaceName, name)
+	var username string
+	if rs.MinIOConfig.BucketName != "" {
+		username = rs.Username
+	}
+	return minioWorkspaceBackupObjectName(username, rs.WorkspaceName, name)

	BucketName string `json:"bucket"`
	BucketName string `json:"bucket,omitempty"`

[content-service] Add support to use existing S3 bucket #10073

[content-service] Add support to use existing S3 bucket #10073

Uh oh!

Conversation

aledbf commented May 17, 2022 • edited by corneliusludmann Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How to test

Release Notes

Uh oh!

aledbf commented May 17, 2022 • edited by werft-gitpod-dev-com bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aledbf commented May 17, 2022

Uh oh!

corneliusludmann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

corneliusludmann commented May 18, 2022

Uh oh!

corneliusludmann commented May 18, 2022

Uh oh!

sagor999 May 20, 2022

Choose a reason for hiding this comment

Uh oh!

sagor999 left a comment

Choose a reason for hiding this comment

Uh oh!

sagor999 commented May 23, 2022

Uh oh!

sagor999 left a comment

Choose a reason for hiding this comment

Uh oh!

corneliusludmann left a comment

Choose a reason for hiding this comment

Uh oh!

corneliusludmann May 24, 2022

Choose a reason for hiding this comment

Uh oh!

sagor999 May 24, 2022

Choose a reason for hiding this comment

Uh oh!

akosyakov left a comment

Choose a reason for hiding this comment

Uh oh!

mustard-mh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aledbf commented May 17, 2022 •

edited by corneliusludmann

Loading

aledbf commented May 17, 2022 •

edited by werft-gitpod-dev-com bot

Loading