Skip to content

Add AM aggregation group metrics #5592

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 7, 2023

Conversation

emanlodovice
Copy link
Contributor

@emanlodovice emanlodovice commented Oct 6, 2023

What this PR does:
This PR emits the alertmanager_dispatcher_aggregation_groups and alertmanager_dispatcher_alert_processing_duration_seconds metrics from alertmanager dispatcher.
image

Which issue(s) this PR fixes:
Fixes #5588

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Emmanuel Lodovice <[email protected]>
@emanlodovice
Copy link
Contributor Author

Not sure what is wrong with the tests, ci / test sometimes passes and sometimes fails

@yeya24
Copy link
Contributor

yeya24 commented Oct 7, 2023

==================
WARNING: DATA RACE
Read at 0x00c000cf0238 by goroutine 40067:
  github.com/thanos-io/thanos/pkg/block/indexheader.(*parallelBucketReader).GetRange.func1()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/parallel_bucket.go:73 +0x324
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /__w/cortex/cortex/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x82

Previous write at 0x00c000cf0238 by goroutine 40024:
  github.com/thanos-io/thanos/pkg/block/indexheader.(*parallelBucketReader).GetRange()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/parallel_bucket.go:80 +0x25c
  github.com/thanos-io/thanos/pkg/block/indexheader.(*chunkedIndexReader).CopySymbols()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/binary_reader.go:220 +0x1d4
  github.com/thanos-io/thanos/pkg/block/indexheader.WriteBinary()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/binary_reader.go:104 +0x404
  github.com/thanos-io/thanos/pkg/block/indexheader.NewBinaryReader()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/binary_reader.go:547 +0x506
  github.com/thanos-io/thanos/pkg/block/indexheader.(*ReaderPool).NewBinaryReader()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/reader_pool.go:89 +0x21a
  github.com/thanos-io/thanos/pkg/store.(*BucketStore).addBlock()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/store/bucket.go:744 +0x747
  github.com/thanos-io/thanos/pkg/store.(*BucketStore).SyncBlocks.func1()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/store/bucket.go:621 +0x7b

Goroutine 40067 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /__w/cortex/cortex/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0x12e
  github.com/thanos-io/thanos/pkg/block/indexheader.(*parallelBucketReader).GetRange()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/parallel_bucket.go:69 +0x236
  github.com/thanos-io/thanos/pkg/block/indexheader.(*chunkedIndexReader).CopySymbols()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/binary_reader.go:220 +0x1d4
  github.com/thanos-io/thanos/pkg/block/indexheader.WriteBinary()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/binary_reader.go:104 +0x404
  github.com/thanos-io/thanos/pkg/block/indexheader.NewBinaryReader()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/binary_reader.go:547 +0x506
  github.com/thanos-io/thanos/pkg/block/indexheader.(*ReaderPool).NewBinaryReader()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/reader_pool.go:89 +0x21a
  github.com/thanos-io/thanos/pkg/store.(*BucketStore).addBlock()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/store/bucket.go:744 +0x747
  github.com/thanos-io/thanos/pkg/store.(*BucketStore).SyncBlocks.func1()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/store/bucket.go:621 +0x7b

Goroutine 40024 (running) created at:
  github.com/thanos-io/thanos/pkg/store.(*BucketStore).SyncBlocks()
      /__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/store/bucket.go:619 +0x[126](https://github.com/cortexproject/cortex/actions/runs/6437075545/job/17481607608?pr=5592#step:6:127)
  github.com/cortexproject/cortex/pkg/storegateway.(*BucketStores).SyncBlocks.func1()
      /__w/cortex/cortex/pkg/storegateway/bucket_stores.go:171 +0x44
  github.com/cortexproject/cortex/pkg/storegateway.(*BucketStores).syncUsersBlocks.func2()
      /__w/cortex/cortex/pkg/storegateway/bucket_stores.go:242 +0x17c
==================
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xba8907]

goroutine 40067 [running]:
github.com/thanos-io/thanos/pkg/runutil.CloseWithErrCapture(0xc000295ee8, {0x0, 0x0}, {0x27be0e5, 0xc}, {0x0, 0x0, 0x0})
	/__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/runutil/runutil.go:[144](https://github.com/cortexproject/cortex/actions/runs/6437075545/job/17481607608?pr=5592#step:6:145) +0x307
github.com/thanos-io/thanos/pkg/block/indexheader.(*parallelBucketReader).GetRange.func1()
	/__w/cortex/cortex/vendor/github.com/thanos-io/thanos/pkg/block/indexheader/parallel_bucket.go:73 +0x3a6
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/__w/cortex/cortex/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x83
created by golang.org/x/sync/errgroup.(*Group).Go
	/__w/cortex/cortex/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0x12f
FAIL	github.com/cortexproject/cortex/pkg/storegateway	213.665s

Seems related to the block parallel download feature. I will take a look

Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yeya24 yeya24 merged commit c3e04bd into cortexproject:master Oct 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Emit alertmanager aggregation group metrics
2 participants