Faster shared parts

[davidmorgan]

Using shared parts (edit: the the source_gen feature) currently slows large builds by about a factor of x2; see the benchmark added for #3800 for the data.

[bwilkerson]

By "shared parts" do you mean a part file that two libraries both include?

If so, is there a requirement for supporting this? The enhanced parts feature removes the ability to share parts in this way, so it's important to know if that isn't going to work. And if not, then we might want to let users know that moving off this pattern sooner will make builds faster, but not actually spend the time to improve performance.

[davidmorgan]

By "shared parts" do you mean a part file that two libraries both include?

Sorry, I should have been clearer, I meant the source_gen feature which combines the output of multiple generators. So nothing to do with language/tooling, very much a build_runner thing :)

[tylandercasper]

loosely linked to #language-4241
automatic "part " inclusions would make it a lot easier for every generator to have it's own part file

[bwilkerson]

Even without automatic part inclusions, enhanced parts would make it possible for each generator to generate a separate part file and then have build_runner generate a single part file that includes all of the generated parts. Users would only be required to add a single part directive for the top-level generated part.

[jakemac53]

Good point about enhanced parts, that would be a nice way to do this in the future

[davidmorgan]

Figured something out here:

The shared part builder works by globbing for outputs from all the shared generators

Glob lib/lib0001.*.g.part
Glob lib/lib0002.*.g.part
Glob lib/lib0003.*.g.part
...

each glob has to check against all inputs, so the number of matches checked is quadratic in the number of inputs.

It needs a very big project to notice: 1s for 1000 files, but up to 19s for 5000 files.

[jakemac53]

Makes sense, iirc the glob support does have some support for caching globs that are the same and ran in the same phase. So we might be able to do a broader glob like **/*.g.part, and then do our own filtering on that result, which should be a much smaller list?