-
Notifications
You must be signed in to change notification settings - Fork 816
Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations #5213
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations #5213
Conversation
4bcb81d
to
7b54d0c
Compare
This is nice!!! I wanted to do this before as well!! :D Did u see any improvements on ingesters CPU? |
@alanprot In our test environment, without heavy queries, the CPU usage can be reduced by up to 25%, roughly 10% on average. |
…ory allocations Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>
7b54d0c
to
8ba5aa7
Compare
LGTM. Thanks |
Ok.. Seems that something weird here: I was writing unit tests for this and i noticed that: ex:
I think we cannot use decodeLen when using stream. From https://pkg.go.dev/github.com/klauspost/compress/snappy#section-readme
|
…ory allocations (cortexproject#5213) Signed-off-by: Xiaochao Dong (@damnever) <[email protected]> Signed-off-by: Alex Le <[email protected]>
…mize memory allocations (cortexproject#5213)" This reverts commit 4821ba3. Signed-off-by: Alex Le <[email protected]>
* Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations (#5213) Signed-off-by: Xiaochao Dong (@damnever) <[email protected]> Signed-off-by: Alex Le <[email protected]> * Fix S3 BucketWithRetries upload empty content issue Signed-off-by: Alex Le <[email protected]> * Update CHANGELOG Signed-off-by: Alex Le <[email protected]> * Revert "Implement grpc.Compressor.DecompressedSize for snappy to optimize memory allocations (#5213)" This reverts commit 4821ba3. Signed-off-by: Alex Le <[email protected]> * Only retry if input reader is seekable Signed-off-by: Alex Le <[email protected]> * Rename mock type Signed-off-by: Alex Le <[email protected]> * Add logging Signed-off-by: Alex Le <[email protected]> * nit fixing Signed-off-by: Alex Le <[email protected]> * add comment Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Xiaochao Dong (@damnever) <[email protected]> Signed-off-by: Alex Le <[email protected]> Co-authored-by: Xiaochao Dong <[email protected]>
…ory allocations (cortexproject#5213) Signed-off-by: Xiaochao Dong (@damnever) <[email protected]> Signed-off-by: Alex Le <[email protected]>
CPU usage can be reduced by up to 25% with this optimization.
Checklist
CHANGELOG.md
updated - the order of entries should be[CHANGE]
,[FEATURE]
,[ENHANCEMENT]
,[BUGFIX]