Limit max number of concurrent tenants shipping and synchronizing TSDB blocks #2026
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pstibrany
approved these changes
Jan 23, 2020
thorfour
approved these changes
Jan 23, 2020
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
pracucci
force-pushed
the
fix-blocks-shipper-concurrency
branch
from
a580258 to
50708c3
Compare
@pstibrany No real reason. I did it to simplify the merge with #1958 but shouldn't be too difficult if it's just 1 letter of difference, so I've unexported it. |
tomwilkie
approved these changes
Jan 24, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Currently there's no limit on the number of tenants concurrently synchronizing blocks in the querier and shipping blocks in ingester. This can lead to unpredictable high spikes in CPU and memory utilization when new blocks are compacted (remember that all blocks are compacted from the head nearly at the same time - see issue #2003).
In this PR I've introduced two flags to limit the concurrency:
-experimental.tsdb.ship-concurrencyto limit the number of tenants concurrently shipping blocks in the ingester-experimental.tsdb.bucket-store.tenant-sync-concurrencyto limit the number of tenants concurrently synchronizing blocks in the querierI've also introduced
UserTSDBand kept a similar design to the one proposed in #1958 in order to reduce the effort to merge the two PRs.Which issue(s) this PR fixes:
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]