Expand Perf Suite Codebases, Expand Perf Suite Operations

[DanielRosenwasser]

The perf suite code typically was added for specific scenarios, but if you're trying to get a sense of how sourcemap emit has changed, it's hard given that not all projects use that functionality.

I think we should

• Have perf suites projects enable emit
• Have perf suites projects enable source map emit
• Have perf suites projects enable declaration emit

[weswigham]

@rbuckton we can turn on declaration emit, source maps, and declaration maps for most (if not all) perf projects (and remove noEmitOnError), but doing so is going to invalidate historic relationships for the data, so if we want to, we'll have to regenerate some historical data - are we OK with doing that?

[MartinJohns]

@weswigham Perhaps you can do this additionally?

[DanielRosenwasser]

(CC @amcasey)

@MartinJohns can you elaborate?

[MartinJohns]

@DanielRosenwasser Continue to run the perf suite as is, but additionally run the perf suite with declaration emit, source maps and declaration maps enabled. Basically produce two data sets.

[rbuckton]

In our current setup, we run benchmarks against a snapshot of a specific project, using the settings specific to that project. This is intended to cover different scenarios, such as:

• Single-file output vs Multi-file output vs no output.
• No source maps, vs external source maps, vs internal source maps.
• Declaration emit vs. no declaration emit.
• Emitting with and without comments.
• Exercising different parts of the compiler (i.e., Compiler-Unions heavily uses union types to stress test those scenarios).

That way we aren't only focused on everything being a "kitchen sink" build, and we can catch performance regressions in non-"kitchen sink" areas (i.e., we don't want to slow down in code paths where we don't emit source maps). The current approach is somewhat randomized, but it covers most of the bases. I wouldn't want to have to run a benchmark against the same project for every permutation of every feature, as that doesn't scale.

@DanielRosenwasser Continue to run the perf suite as is, but additionally run the perf suite with declaration emit, source maps and declaration maps enabled. Basically produce two data sets.

What we can do is add new scenarios that point to the same sources but with different settings. That way old runs wouldn't be invalidated. So in that case we might have a "TFS+SourceMaps" scenario, etc.

[amcasey]

Related: #44251