You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently EC Reads are less stable than replication reads because if 4 out of 9 datanodes in the block group are unable to connect, then the whole read fails. Erasure Coding reads need to be able to handle disconnects from DataNodes and retry after a backoff duration to avoid overloading the DataNodes while increasing the stability of the read.
Throttling on server side was another proposed solution, but we prefer this client side backoff for a few main reasons:
Throttling on the server would use up thread connections which have a maximum limit.
Throttling was originally added only for cohosting scenario to reduce impact on other services
Throttling would use up resources on the DataNode which could already be in a bad state.
What
The previous implementation followed a 4 phase algorithm to read.
Attempt to read chunks from the data blocks
Check for missing data chunks. Fail if there are more missing than the number of parity blocks, otherwise read parity blocks and null data blocks
Wait for data to be read into the buffers and handle any read errors by reading from more parity blocks
Check for missing blocks and either decode or fail.
The new implementation now merges phase 1-3 into a single loop:
Loop until we have enough blocks for read or decode, or we have too many missing blocks to succeed
Determine the number of chunks we need to fetch. ALLZERO chunks count towards this total. null data chunks also count towards this total unless there are MISSING or SLEEPING data chunks.
Read chunks until we have enough pending or fetched to be able to decode or normal read.
faster.
Get results from reads and handle exceptions by preparing more reads for decoding the missing data
Check if we should sleep before retrying any reads.
Check for missing blocks and either decode or fail.
Add two new states to StripingChunk:
"SLEEPING" to indicate that the node where the chunk is stored has failed and will be retried in the future
"READY" to indicate that the node where the chunk is stored is ready to be attempted
How was this patch tested?
Add unit test to TestWriteReadStripedFile
Covers RS(3,2) with 1 chunk on failed nodes, 2 chunks on failed nodes, and 3 chunks on failed nodes.
For code changes:
Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Currently EC Reads are less stable than replication reads because if 4 out of 9 datanodes in the block group are unable to connect, then the whole read fails. Erasure Coding reads need to be able to handle disconnects from DataNodes and retry after a backoff duration to avoid overloading the DataNodes while increasing the stability of the read.
Throttling on server side was another proposed solution, but we prefer this client side backoff for a few main reasons:
What
The previous implementation followed a 4 phase algorithm to read.
The new implementation now merges phase 1-3 into a single loop:
faster.
Add two new states to StripingChunk:
How was this patch tested?
Add unit test to
TestWriteReadStripedFile
For code changes:
LICENSE
,LICENSE-binary
,NOTICE-binary
files?