Skip to content

Added registered timestamp to instances in the ring #3248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

pracucci
Copy link
Contributor

What this PR does:
I'm working on adding shuffle sharding support for ingesters on the read path. A preliminary step is to have the timestamp at which an instance registered to the ring, which is what I'm proposing in this PR.

The changeset is large because I had to add the registered timestamp as parameter of the AddIngester() function, which is widely used in tests. A part from this, logic changes are pretty minimal.

A part from unit tests, I've also run some manual tests both on ingesters ring (to test Lifecycler) and store-gateways ring (to test BasicLifecycler):

  • Scale up / down an instance
  • Abruptly terminate (SIGKILL) an instance and then, after a while, restart it (the registered timestamp is preserved if the instance wasn't forgotten in the ring)
  • Click "Forget" in the ring HTTP page

Which issue(s) this PR fixes:
N/A

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Contributor

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Please take a look at comment at (*IngesterDesc).GetRegisteredAt.

return ringDesc, true, nil
}

// The instance already exists in the ring, so we can't change the registered timestamp (even if it's zero)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may not be clear why we don't set "registered at" if it's zero. I would suggest extending description of this field in ring.proto to describe its intended usage, and explaining why preserving zero is important.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the command. I haven't specifically referenced the shuffle sharding because it's not a thing yet and this is a generic property, but I tried to make it clear to never change it once its set (even if it's 0).

WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I meant is to explain purpose of the field. Something like: This field is used to find out subset of instances that could have possibly owned the specific token in the past.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I meant is to explain purpose of the field. Something like: This field is used to find out subset of instances that could have possibly owned the specific token in the past.

And from there, explain why updating to non-zero value is a bad idea.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, updated!

Signed-off-by: Marco Pracucci <[email protected]>
Copy link
Contributor

@pstibrany pstibrany left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, good job!

Signed-off-by: Marco Pracucci <[email protected]>
@pracucci pracucci merged commit e319a66 into cortexproject:master Sep 30, 2020
@pracucci pracucci deleted the add-joined-at-timestamp-to-ring branch September 30, 2020 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants