Support using pre-allocated blob containers in azure. #660
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@tasdomas, would you mind setting the base branch of #660 (this) to #651 so I don't have to tweak GitHub filters too much to review only the changes it introduces? |
I was thinking that we'd land #651 first and then move on to this. |
tasdomas
force-pushed
the
d012-bucket-reuse-azure
branch
from
f149c7b to
aa5d79c
Compare
…re data source. Use common.RemoteStorage to initialize the data sources. Using rclone to verify storage during Read. Remove aws s3 client mocks and tests that rely on them.
tasdomas
force-pushed
the
d012-bucket-reuse-azure
branch
from
bfdf872 to
e93e8fb
Compare
tasdomas
commented
Sep 12, 2022
| ) | ||
|
|
||
| // NewExistingS3Bucket returns a new data source refering to a pre-allocated | ||
| // S3 bucket. | ||
| func NewExistingS3Bucket(client S3Client, credentials aws.Credentials, id string, region string, path string) *ExistingS3Bucket { | ||
| func NewExistingS3Bucket(credentials aws.Credentials, storageParams common.RemoteStorage) *ExistingS3Bucket { |
There was a problem hiding this comment.
Passing common.RemoteStorage to create a reference to an existing s3 bucket, instead of individual params.
tasdomas
commented
Sep 12, 2022
| return common.NotFoundError | ||
| } | ||
| return err | ||
| err := machine.CheckStorage(ctx, b.connection()) |
There was a problem hiding this comment.
Using rclone to verify bucket access (this will be done across all providers, sans k8s)
tasdomas
commented
Sep 12, 2022
| @@ -1,5 +0,0 @@ | |||
| package mocks | |||
There was a problem hiding this comment.
By moving to using rclone to validate bucket access, these mocks are no longer useful. I'll probably revisit the idea at some point.
0x2b3bfa0
reviewed
Sep 14, 2022
dacbd
approved these changes
Sep 14, 2022
casperdcl
added
cloud-az
tasdomas
pushed a commit
that referenced
this pull request
Oct 21, 2022
* Support for existing storage containers in aws and gcp. (#651) * Update config schema. * AWS support for existing S3 buckets. * Existing bucket support for gcp. * Add mocks and tests to existing bucket resource in aws. * Update docs with new pre-allocated container fields. * Support using pre-allocated blob containers in azure. (#660) * AWS support for existing S3 buckets. * Existing bucket support for gcp. * Fix subdirectory in rclone remote. * Blob container generates the rclone connection string. * Introduce a type for generating rclone connection strings. * Azure support for reusable blob containers. * Update docs. * Fix path prefix. * Initialize s3 existing bucket with RemoteStorage struct. * Update gcp and aws existing bucket data sources to align with the azure data source. Use common.RemoteStorage to initialize the data sources. Using rclone to verify storage during Read. Remove aws s3 client mocks and tests that rely on them. * Fix comment. * K8s support for specifying an existing persistent volume claim (#661) * K8s support for specifying an existing persistent volume claim. Co-authored-by: Helio Machado <[email protected]> * Update use of Identifier struct. * Combine container and container_path config keys. * Update docs. * Use split function from rclone. Co-authored-by: Helio Machado <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains a few other changes as well: