Skip to content

Conversation

@steveloughran
Copy link
Contributor

Stop using the versionId when building the list of files
to delete as the rename progresses

Testing: manual. For anyone wishing to replicate the problem on an unpatched release

  1. Have a versioned bucket.
  2. Latest version of cloudstore now allows you to create a set of session keys from an IAM role and json file, printing them as env vars and hadoop config options
  3. You need S3Guard enabled and the files to have been created individually by a client through S3Guard (i.e. the entries not discovered through listing). Then S3Guard caches the version ID in the DB, which is then used when deleting the object.
  4. mkdir src/; touchz src/file1 file2
  5. mv src dst

if that works, try renaming back.

Stop using the versionId when building the list of files
to delete as the rename progresses

Change-Id: I2dcb3d9c7b1a6a40d05800135ce087d42294589f
@steveloughran steveloughran added the fs/s3 changes related to hadoop-aws; submitter must declare test endpoint label Sep 15, 2020
Test verified to fail without the patch in the right env (version + S3guard
+ bulk).

Docs cover failures during rename

Change-Id: I0a2ce331df4b2728c6e0542cd62d5cb7fc6bc39b
Copy link
Contributor

@mukund-thakur mukund-thakur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Removed the version argument from the signatures of the methods,
which ensures it can never creep back in without major work.

Change-Id: Ie8f7f2677d82140246a41ac8dc96b69ea8dbb90e
@apache apache deleted a comment from hadoop-yetus Sep 16, 2020
@steveloughran
Copy link
Contributor Author

testing in progress: S3 london

@steveloughran
Copy link
Contributor Author

Test with -
mvit -Dparallel-tests -DtestsThreadCount=6 -Dmarkers=keep -Ds3guard -Ddynamo -Dfs.s3a.directory.marker.audit=true -Dscale

https://issues.apache.org/jira/browse/HADOOP-17263 : read() didn't get as much back as expected (unrelated)
https://issues.apache.org/jira/browse/HADOOP-17226 Failure of ITestAssumeRole.testRestrictedCommitActions

That -Dscale option kicks off some MR jobs which spawn extra processes beyond the maven JUnit runners, the system may be overloading. Surprised how it surfaces though. I can see it amplifies a race condition in a non-atomic ++ call (more chance of pre-emption, longer delay for rescheduling), but read() not receiving enough data

@steveloughran
Copy link
Contributor Author

@bgaborg - could you look @ this when you have a chance?

@bgaborg
Copy link

bgaborg commented Sep 22, 2020

LGTM, +1

Copy link

@bgaborg bgaborg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, +1

@steveloughran
Copy link
Contributor Author

thank you :)

@steveloughran steveloughran merged commit 7fae413 into apache:trunk Sep 22, 2020
bilaharith pushed a commit to bilaharith/hadoop that referenced this pull request Sep 27, 2020
@apache apache deleted a comment from hadoop-yetus Oct 14, 2020
asfgit pushed a commit that referenced this pull request Nov 18, 2020
…2303)

Contributed by Steve Loughran.

Change-Id: I8e89a402a24bd9fb958e0fa93d1a28191093851d
@steveloughran steveloughran deleted the s3/HADOOP-17261-deleteObjectVersionsRegression branch October 15, 2021 19:40
jojochuang pushed a commit to jojochuang/hadoop that referenced this pull request May 23, 2023
…ermission (apache#2303)

Contributed by Steve Loughran.

Change-Id: Ia29d9d87e592799ed68da4a5d9912660db2ac934
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fs/s3 changes related to hadoop-aws; submitter must declare test endpoint

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants