Skip to content

Add rand to sparse requests for rejoin memberlist cluster #4747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 10, 2022

Conversation

danielblando
Copy link
Contributor

@danielblando danielblando commented Jun 6, 2022

Signed-off-by: Daniel Blando [email protected]

What this PR does:
When scaling pods simultaneously, we have a chance of then rejoining together. This can cause CPU instability with spikes around the time pods are rejoining. The PR try to mitigate the issue distributing the requests adding a jitter to reset the initial time for each pod.

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@danielblando danielblando marked this pull request as draft June 6, 2022 22:36
@danielblando danielblando changed the title Add rand to sparse requests Add rand to sparse requests for rejoin memberlist cluster Jun 6, 2022
@danielblando danielblando force-pushed the jitterRejoinInterval branch from a6f5d6b to 0caef42 Compare June 6, 2022 23:21
@danielblando danielblando marked this pull request as ready for review June 7, 2022 00:05
@danielblando danielblando force-pushed the jitterRejoinInterval branch from 0caef42 to 5c5aebb Compare June 8, 2022 17:39
@pull-request-size pull-request-size bot added size/S and removed size/XS labels Jun 8, 2022
@alanprot
Copy link
Member

alanprot commented Jun 9, 2022

LGTM

@danielblando danielblando force-pushed the jitterRejoinInterval branch from 5c5aebb to 9ebf137 Compare June 9, 2022 21:45
@pull-request-size pull-request-size bot added size/M and removed size/S labels Jun 9, 2022
Signed-off-by: Daniel Blando <[email protected]>
@danielblando danielblando force-pushed the jitterRejoinInterval branch from 9ebf137 to 071dd5f Compare June 9, 2022 22:12
@alvinlin123 alvinlin123 merged commit 4e69b84 into cortexproject:master Jun 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants