Skip to content

Conversation

@tobyh-canva
Copy link
Contributor

@tobyh-canva tobyh-canva commented Oct 27, 2025

When py_binary/py_test were being built, they were flattening the runfiles
depsets at analysis time in order to create the zip file mapping manifest for
their implicit zipapp outputs. This flattening was necessary because they had
to filter out the original main executable from the runfiles that didn't belong
in the zipapp. This flattening is expensive for large builds, in some cases
adding over 400 seconds of time and significant memory overhead.

To fix, have the zip file manifest use the runfiles_with_exe object, which is
the runfiles, but pre-filtered for the files zip building doesn't want. This
then allows passing the depsets directly to Args.add_all and using map_each
to transform them.

Additionally, pass runfiles.empty_filenames using a lambda. Accessing that
attribute implicitly flattens the runfiles.

Finally, because the original profiles indicated str.format() was a non-trivial
amount of time (46 seconds / 15% of build time), switch to using + instead.

This is a more incremental alternative to #3380 which achieves most of the
same optimization with only Starlark changes, as opposed to introducing an
external script written in C++.

Profile of a large build, which shows a Starlark CPU profile. It shows an overall build
time of 305 seconds. 46 seconds (15%) are spent in map_zip_runfiles, half of which
is in str.startswith() and the other half in str.format().

@rickeylev
Copy link
Collaborator

As mentioned in the other PR comment: we can definitely accept PR until the questions about using an external tool are worked out.

Overall, LGTM. Because of the profile results in the other thread showing format() (as called by map_zip_runfiles) was a significant chunk of the time, changing to % or + will probably save some cycles. format_each might also help. I'm going to poke it a bit.

@tobyh-canva tobyh-canva changed the title zip file analysis optimisation: alternate starlark-only approach perf: improve analysis performance by for py_binary and py_test rules Nov 9, 2025
@rickeylev rickeylev changed the title perf: improve analysis performance by for py_binary and py_test rules refactor: defer zip manifest building to execution phase to improve analysis phase performance Nov 9, 2025
@rickeylev
Copy link
Collaborator

rickeylev commented Nov 10, 2025

For posterity: Before this change, about 462 seconds is spent in py_binary related to zip file manifest building.

With this change, the profile shows about 46 seconds spent in py_binary related to zip file manifest building.

profile

This profile is this change, but before changing to use + instead of format()

@rickeylev
Copy link
Collaborator

Ok, cleaned this up, switched it to use + instead of format().

@tobyh-canva If you have opportunity, could you run another profile? I'm interested to see how much of the 23 seconds of format() overhead in building the path strings is gone.

@rickeylev rickeylev enabled auto-merge November 10, 2025 00:27
@rickeylev rickeylev added this pull request to the merge queue Nov 10, 2025
@tobyh-canva
Copy link
Contributor Author

Running a profile right now, thanks heaps!

Merged via the queue into bazel-contrib:main with commit 4fb634e Nov 10, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants