Scalable builds

[davidmorgan]

The benchmark I'm adding in #3802 shows the problem pretty clearly,

json_serializable
libraries	clean/ms	no changes/ms	incremental/ms
1	21573	2840	4278
100	23040	2974	6327
250	28098	3285	12578
500	42940	4288	35061
750	69483	6561	67670
1000	115941	9178	119308

notice the incremental build time for 500 libraries and 1000 libraries: it increases from 35s to 119s, an increase of x3.4. For double the number of libraries the time increase should be x2 :)

The benchmark also runs for built_value, freezed and mockito, numbers on the PR, the story is pretty similar, except:

• built_value and json_serializable are about 2x as slow as freezed on the big build, it turns out this is purely because they use shared parts, filed Faster shared parts #3803 for that sub-problem
• mockito is about 2x as fast as freezed, I believe that is because mockito adds a part file to the test where it does not become a dep of all the other generators; conversely built_value, json_serializable and freezed all add part files in the app code, doubling the number of deps of all the generators

Apart from these multipliers the numbers are very similar, I think it does not matter much what the generators are actually doing.

[rrousselGit]

• mockito is about 2x as fast as freezed, I believe that is because mockito adds a part file to the test where it does not become a dep of all the other generators; conversely built_value, json_serializable and freezed all add part files in the app code, doubling the number of deps of all the generators

Going back to this:

This could be because freezed starts by asking for the AST of the generated file.
I need the AST, and build_runner doesn't give it to me. So I have to ask for it again. And I don't think it's cached, right?

If build_runner could expose us the AST, that'd be cool. Most of my generators use it.

[davidmorgan]

• mockito is about 2x as fast as freezed, I believe that is because mockito adds a part file to the test where it does not become a dep of all the other generators; conversely built_value, json_serializable and freezed all add part files in the app code, doubling the number of deps of all the generators

Going back to this:

This could be because freezed starts by asking for the AST of the generated file. I need the AST, and build_runner doesn't give it to me. So I have to ask for it again. And I don't think it's cached, right?

If build_runner could expose us the AST, that'd be cool. Most of my generators use it.

The performance issue I'm looking at is just due to the deps graph, it's nothing to do with what the generator does. When you look at the CPU flame charts of the large benchmarks, the generator barely shows up.

Once that's solved--yes, I think there's a possibility of handling ASTs better/faster, we'd have to compare :) thanks.

[t-beckmann]

+1 Just to let you know, this is an issue for me as I have a CI pipeline taking more than an hour to generate. Glad you are working on this!

[jonpittock]

definite issue for us. Big project. Mono repo. Almost unusable!

[esDotDev]

The other problem this shows is how slow the baseline performance levels are. 22s for first-run, and 6 seconds for incremental, with a single library, feel at least 10x bigger than they should be if we're going to get anything near the performance level of built-in data classes, or what was imagined w/ macros.

[davidmorgan]

@esDotDev yes, for sure.

You get quite a lot faster small incremental builds if you use watch mode; I'm not sure yet how much people are forced to use build or are not aware of watch, but ideally I'd like repeated use of build to be just as fast as watch. And for both to be faster than today :)

Filed #4019 for performance improvements for small builds.

Thanks :)

[rrousselGit]

IMO clean build speed is not nearly as important as the other two columns

It's important, don't mind me. But clean build can be multiple orders of magnitude slower than incremental builds.

Take flutter run.
The commend takes easily 10s of seconds to start the app. That's your "clean build".
But once started, hotreload takes a few milliseconds. That's your incremental build

---

For that reason, I feel like there's a missed opportunity by not investigating better cache invalidation strategies.

It's cool that we're trying to make writing 1000 files fast.
But more often than not, we should just be writing 1 file instead of 1000

[davidmorgan]

For that reason, I feel like there's a missed opportunity by not investigating better cache invalidation strategies.

There's an arc of work happening over in the analyzer to support fine-grained invalidation there

https://dart-review.googlesource.com/q/Fine.

which I expect will help significantly, possibly with some changes on the build_runner side too, we'll see.