Skip to content

Add stats log when querying store gateway #5376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 5, 2023

Conversation

yeya24
Copy link
Contributor

@yeya24 yeya24 commented Jun 1, 2023

What this PR does:

Add series stats log in querier to log request stats going to each store gateway. This was added to Thanos recently thanos-io/thanos#6352.

Example:

ts=2023-06-01T16:29:19.133919Z caller=blocks_store_queryable.go:725 level=info msg="store gateway series request stats" instance=localhost:9008 queryable_chunk_bytes_fetched=47864 queryable_data_bytes_fetched=49149 blocks_queried=4 series_merged_count=11 chunks_merged_count=212 postings_touched=4 postings_touched_size_sum=192 postings_to_fetch=0 postings_fetched=4 postings_fetch_count=4 postings_fetched_size_sum=192 series_touched=44 series_touched_size_sum=3011 series_fetched=44 series_fetch_count=4 series_fetched_size_sum=0 chunks_touched=212 chunks_touched_size_sum=41419 chunks_fetched=212 chunks_fetch_count=4 chunks_fetched_size_sum=111767

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@yeya24 yeya24 force-pushed the store-gateway-stats branch from 2d1aff6 to b5c8812 Compare June 1, 2023 17:46
@yeya24 yeya24 force-pushed the store-gateway-stats branch from b5c8812 to a59bdc2 Compare June 1, 2023 17:49
@pull-request-size pull-request-size bot added size/M and removed size/L labels Jun 1, 2023
@yeya24
Copy link
Contributor Author

yeya24 commented Jun 1, 2023

I am thinking if we need to log stats per store gateway request or we only log the aggregated stats for all store gateway for a query. The previous option is more verbose but better to maybe identify query skews for different store gateways.

Signed-off-by: Ben Ye <[email protected]>
Copy link
Member

@friedrichg friedrichg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
I think is fine without aggregation, we lose important detail with aggregation

yeya24 added 2 commits June 2, 2023 12:10
@yeya24
Copy link
Contributor Author

yeya24 commented Jun 5, 2023

I will go ahead and merge the pr.

@yeya24 yeya24 closed this Jun 5, 2023
@yeya24 yeya24 reopened this Jun 5, 2023
@yeya24 yeya24 enabled auto-merge (squash) June 5, 2023 04:45
@yeya24 yeya24 merged commit 7bb2850 into cortexproject:master Jun 5, 2023
cathy-qiu pushed a commit to cathy-qiu/cortex that referenced this pull request Jun 7, 2023
* add stats log when querying store gateway

Signed-off-by: Ben Ye <[email protected]>

* update changelog

Signed-off-by: Ben Ye <[email protected]>

* add data_downloaded_size_sum

Signed-off-by: Ben Ye <[email protected]>

* update vendor

Signed-off-by: Ben Ye <[email protected]>

---------

Signed-off-by: Ben Ye <[email protected]>
Signed-off-by: Cathy Qiu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants