Skip to content

compress/flate, archive/zip: reduce memory allocations #59775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 6 commits into from
Closed

compress/flate, archive/zip: reduce memory allocations #59775

wants to merge 6 commits into from

Conversation

subtle-byte
Copy link
Contributor

The existing implementation allocates a new 4KB buffer each time it opens flate-encoded file in a zip archive. This commit allows the flate reader to reuse the buffer on call Reset instead of allocating a new one.

It is noticeable when a zip archive contains a huge amount of files, e.g. zip archive has 50_000 files, for each file 4KB buffer is allocated, so it is 200MB memory allocations. If files are read sequentially only one buffer is needed.

Fixes #59774

@gopherbot
Copy link
Contributor

This PR (HEAD: b4b877f) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/487675 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

@gopherbot
Copy link
Contributor

Message from [email protected]:

Patch Set 1:

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 1:

(3 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

This PR (HEAD: 36bf297) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/487675 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

@gopherbot
Copy link
Contributor

Message from [email protected]:

Patch Set 2:

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Maksim Meshkov:

Patch Set 2:

(3 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 2:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Maksim Meshkov:

Patch Set 2:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 2:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

This PR (HEAD: f7e416a) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/487675 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

@gopherbot
Copy link
Contributor

Message from [email protected]:

Patch Set 3:

(3 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Maksim Meshkov:

Patch Set 3:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 3:

(4 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Maksim Meshkov:

Patch Set 3:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 3:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

This PR (HEAD: e1f6aa8) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/487675 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

@gopherbot
Copy link
Contributor

Message from Maksim Meshkov:

Patch Set 3:

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from [email protected]:

Patch Set 4:

(3 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Maksim Meshkov:

Patch Set 4:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Maksim Meshkov:

Patch Set 4:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 4: Run-TryBot+1

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 4:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 4:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 5: Run-TryBot+1

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 5:

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 5:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 5: TryBot-Result-1

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

This PR (HEAD: 82223bf) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/487675 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

@gopherbot
Copy link
Contributor

Message from [email protected]:

Patch Set 6:

(3 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 6: Run-TryBot+1

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Maksim Meshkov:

Patch Set 6:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 6:

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 6:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 6: TryBot-Result-1

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 6:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

This PR (HEAD: f3f395b) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/go/+/487675 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

@gopherbot
Copy link
Contributor

Message from [email protected]:

Patch Set 7:

(3 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Maksim Meshkov:

Patch Set 7:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 7: Run-TryBot+1

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 7:

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Gopher Robot:

Patch Set 7: TryBot-Result+1

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

@gopherbot
Copy link
Contributor

Message from Ian Lance Taylor:

Patch Set 7: Auto-Submit+1 Code-Review+2 Run-TryBot+1

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/487675.
After addressing review feedback, remember to publish your drafts!

gopherbot pushed a commit that referenced this pull request May 24, 2023
@subtle-byte @gopherbot
compress/flate, archive/zip: reduce memory allocations
0845880 
The existing implementation allocates a new 4KB buffer each time it opens flate-encoded file in a zip archive. This commit allows the flate reader to reuse the buffer on call Reset instead of allocating a new one.

It is noticeable when a zip archive contains a huge amount of files, e.g. zip archive has 50_000 files, for each file 4KB buffer is allocated, so it is 200MB memory allocations. If files are read sequentially only one buffer is needed.

Fixes #59774

Change-Id: Ib16336b101ba58e8f0f30a45dc5fd4eeebc801a1
GitHub-Last-Rev: f3f395b
GitHub-Pull-Request: #59775
Reviewed-on: https://go-review.googlesource.com/c/go/+/487675
Run-TryBot: Ian Lance Taylor <[email protected]>
Auto-Submit: Ian Lance Taylor <[email protected]>
Run-TryBot: Ian Lance Taylor <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Ian Lance Taylor <[email protected]>
Reviewed-by: Matthew Dempsky <[email protected]>
@gopherbot
Copy link
Contributor

This PR is being closed because golang.org/cl/487675 has been merged.

@gopherbot gopherbot closed this May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

compress/flate, archive/zip: large memory allocations reading flate zip archive with many files
2 participants
@subtle-byte @gopherbot