Regression in Crossgen2 Throughput

[DrewScoggins]

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single - Microsoft.CodeAnalysis

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
CodeAnalysis	1854.95 ms	1921.10 ms	1.04

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single - Microsoft.CodeAnalysis*'

Histogram

Crossgen2 Throughput - Single - Microsoft.CodeAnalysis

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single - System.Net.Sockets

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Sockets	775.46 ms	786.31 ms	1.01

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single - System.Net.Sockets*'

Histogram

Crossgen2 Throughput - Single - System.Net.Sockets

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single - System.Private.CoreLib

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
CoreLib	2741.40 ms	3018.09 ms	1.10

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single - System.Private.CoreLib*'

Histogram

Crossgen2 Throughput - Single - System.Private.CoreLib

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single Threaded - Single - System.Data.Common

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Common	2563.62 ms	2740.17 ms	1.07

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single Threaded - Single - System.Data.Common*'

Histogram

Crossgen2 Throughput - Single Threaded - Single - System.Data.Common

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single - Microsoft.CodeAnalysis.VisualBasic

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
VisualBasic	3807.93 ms	3969.92 ms	1.04

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single - Microsoft.CodeAnalysis.VisualBasic*'

Histogram

Crossgen2 Throughput - Single - Microsoft.CodeAnalysis.VisualBasic

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single Threaded - Single - Microsoft.CodeAnalysis.CSharp

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
CSharp	7879.24 ms	8506.36 ms	1.08

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single Threaded - Single - Microsoft.CodeAnalysis.CSharp*'

Histogram

Crossgen2 Throughput - Single Threaded - Single - Microsoft.CodeAnalysis.CSharp

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single Threaded - Single - Microsoft.CodeAnalysis

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
CodeAnalysis	3577.61 ms	3794.41 ms	1.06

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single Threaded - Single - Microsoft.CodeAnalysis*'

Histogram

Crossgen2 Throughput - Single Threaded - Single - Microsoft.CodeAnalysis

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Composite - framework-r2r

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Crossgen2 Throughput - Composite - framework-r2r	32497.26 ms	35029.59 ms	1.08

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Composite - framework-r2r*'

Histogram

Crossgen2 Throughput - Composite - framework-r2r

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single Threaded - Single - System.Net.WebProxy

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
WebProxy	661.70 ms	667.57 ms	1.01

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single Threaded - Single - System.Net.WebProxy*'

Histogram

Crossgen2 Throughput - Single Threaded - Single - System.Net.WebProxy

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single - System.Data.Common

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Common	1268.30 ms	1327.32 ms	1.05

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single - System.Data.Common*'

Histogram

Crossgen2 Throughput - Single - System.Data.Common

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single - System.ServiceModel.Syndication

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Syndication	726.85 ms	732.04 ms	1.01

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single - System.ServiceModel.Syndication*'

Histogram

Crossgen2 Throughput - Single - System.ServiceModel.Syndication

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture	x64
OS	ubuntu 18.04
Diff	Diff

Regressions in Crossgen2 Throughput - Single Threaded - Single - System.Net.Sockets

Benchmark	Baseline	Test	Test/Base	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Sockets	1106.07 ms	1149.64 ms	1.04

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'Crossgen2 Throughput - Single Threaded - Single - System.Net.Sockets*'

Histogram

Crossgen2 Throughput - Single Threaded - Single - System.Net.Sockets

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[DrewScoggins]

This seems like the most likely change in the commit range, #50675

[EgorBo]

It'd be nice to somehow fix the regression but I'd love #50675 to stay anyway as I'm going to work on the inliner at some point and an ability to recognize some (foldable) calls is very useful (e.g. #50784)

[tannergooding]

CC. @AndyAyersMS, @jkotas

This looks to be the "true" cost (about 8% worst case) of #50675, since crossgen/prejit will be resolving the method tokens for basically every method.
I imagine this is also where caching the tokens would help the most. Since we are prejit, we will end up resolving all these tokens eventually anyways.

[AndyAyersMS]

Since we are prejit, we will end up resolving all these tokens eventually anyways.

If that was the case, and re-resolving an already resolved token is cheap, I would not have expected much of a regression. So either:

• re-resolving is not as cheap as we thought
• or the subsequent call to eeGetCallInfo is the part that is costly
• or we are doing a lot of speculative resolution we never did before.

My bet would be on eeGetCallInfo, I think it is doing a lot more work than we need here (we just need the method handle and a few flags)....

[tannergooding]

My bet would be on eeGetCallInfo, I think it is doing a lot more work than we need here (we just need the method handle and a few flags)....

We're only passing in CORINFO_CALLINFO_KINDONLY (and CORINFO_CALLINFO_CALLVIRT if its a callvirt) to try and limit the work done. If there is another API that does even less work, I'm happy to switch:

impResolveToken(codeAddr, &resolvedToken, CORINFO_TOKENKIND_Method);
eeGetCallInfo(&resolvedToken,
              (prefixFlags & PREFIX_CONSTRAINED) ? &constrainedResolvedToken : nullptr,
              combine(CORINFO_CALLINFO_KINDONLY,
                      (opcode == CEE_CALLVIRT) ? CORINFO_CALLINFO_CALLVIRT : CORINFO_CALLINFO_NONE),
              &callInfo);

[jkotas]

#51088 will make it a bit cheaper. There is a lot of other useless work that the speculative getCallInfo does:

• Compute class and method flags for the target
• Compute method signature
  Do you expect that these will be used? If not, we can add more flags to supress them.

[trylek]

Thanks for all the valuable clarifications in this thread. Do we believe it's still accurate to label this bug as "area-crossgen2-coreclr"? Can someone please either help me understand what's the expected Crossgen2 ask or consider changing the label?

[tannergooding]

Do you expect that these will be used? If not, we can add more flags to supress them.

@jkotas, right now we only care about "is this method an intrinsic?" (CORINFO_FLG_INTRINSIC or CORINFO_FLG_JIT_INTRINSIC) and the method handle so we can call getIntrinsicID and lookupNamedIntrinsic, respectively.

I doubt, at least initially, there will be anything interesting we want to do with arbitrary/non-intrinsic method tokens, so I think we can filter down to just those flags and the handle.

• lookupNamedIntrinsic already independently calls getMethodSig using the method handle for a couple paths, but that will be more "pay for play" than doing it for every token