Skip to content

Conversation

kolesnikovae
Copy link
Collaborator

@kolesnikovae kolesnikovae commented Jun 12, 2025

Something I noticed while working on #4236: we unconditionally update the DiscardedBytes and DiscardedProfiles metrics after relabeling:

validation.DiscardedBytes.WithLabelValues(string(validation.DroppedByRelabelRules), req.TenantID).Add(float64(req.DiscardedBytesRelabeling))
validation.DiscardedProfiles.WithLabelValues(string(validation.DroppedByRelabelRules), req.TenantID).Add(float64(req.DiscardedProfilesRelabeling))

It's not a major issue, but in large-scale deployments with hundreds of distributors and tenants, we generate many time series unnecessarily (it's very unlikely that we ever need to report anything in this particular place). Additionally, having to filter with > 0 in each query is annoying

Comment on lines -1078 to +1083
if err := validation.ValidateLabels(v.limits, v.tenantID, labels); err != nil {
return connect.NewError(connect.CodeInvalidArgument, err)
}
return nil
return validation.ValidateLabels(v.limits, v.tenantID, labels)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels like it's the caller's responsibility to wrap the error

@kolesnikovae kolesnikovae marked this pull request as ready for review June 12, 2025 04:45
@kolesnikovae kolesnikovae requested a review from a team as a code owner June 12, 2025 04:45
Copy link
Contributor

@simonswine simonswine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good find! LGTM

@simonswine simonswine merged commit bb8d879 into main Jun 18, 2025
31 of 33 checks passed
@simonswine simonswine deleted the chore/distributor-discarded-optimization branch June 18, 2025 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants