Improve Benchmark Accuracy

[timcassell]

Fixes #1133
Fixes #2305
Fixes #1802
Fixes #2530

• Drastically simplified benchmark type code gen.
• Overhead return type is always void for a fair baseline for all benchmark return types.
• Benchmark method is called directly instead of wrapped in a delegate.
• Return value is popped off the stack instead of passed to Consumer or a NoInlining method.
• Added an assembly weaver that automatically applies MethodImplOptions.NoInlining to benchmark methods.
• Added additional tests.

[timcassell]

Unfortunately, I no longer have an Intel CPU to test this, but I think the new results from this PR should address the issues you had in #2334 @AndreyAkinshin (results look good on my AMD 9800X3D CPU in .Net 8).

[timcassell]

@AndyAyersMS Adam mentioned you are the expert I should tag to get your opinion on things like this. Do you have any thoughts or concerns about this approach?

[AndyAyersMS]

Do you have any thoughts or concerns about this approach?

I am not that familiar with how robust the weaving process is... maybe it would be good (if possible) to validate it on say the dotnet/performance suite of 5000ish benchmarks. I guess it also means there is no simple faithful source representation for the benchmark assembly, which might trip up some users that keep the sources around and use those for investigations?

What happens if you just compile and run those sources?

Do you have examples showing the impact on reported results?

• Overhead return type is always void for a fair baseline for all benchmark return types.

Can you explain this a bit more? The overhead is now some "fixed" workload measurement regardless of benchmark method signature?

[timcassell]

I am not that familiar with how robust the weaving process is... maybe it would be good (if possible) to validate it on say the dotnet/performance suite of 5000ish benchmarks.

I can give it a try.

I guess it also means there is no simple faithful source representation for the benchmark assembly, which might trip up some users that keep the sources around and use those for investigations?

Could you elaborate? The weaver just adds NoInlining to [Benchmark] annotated methods. Yeah it's not an exact 1:1 match of source code, but C# doesn't provide a way for us to do it ourselves, even with source generators. It's a small change that I'm not sure how it would impact users doing those types of deep investigations. I would expect anyone doing that to be a power user anyway.

What happens if you just compile and run those sources?

What do you mean? If the source project has a reference to BenchmarkDotNet.Weaver (transitive or otherwise) and it's built with msbuild, the assembly weaver will kick in whether they actually run the benchmarks or not. If they manually instantiate and use a benchmark class, it will just not inline the methods. All other behavior will be unaffected.

Do you have examples showing the impact on reported results?

With the benchmarks from #1133, virtually no impact on my Ryzen 9800X3D, slightly more accurate on Apple M3 (a CPU architecture which BDN seems to have overhead measurement issues for nano benchmarks). It seems to mostly impact Intel CPUs and older (pre-Zen) AMD CPUs (see results I obtained previously in #2334. Unfortunately I no longer have those old CPUs to test again).

Apple M3 results

master:

Method	Mean	Error	StdDev
Increment01	0.6632 ns	0.0378 ns	0.0711 ns
Increment02	0.0000 ns	0.0000 ns	0.0000 ns
Increment03	0.0000 ns	0.0000 ns	0.0000 ns
Increment04	0.0000 ns	0.0000 ns	0.0000 ns
Increment05	0.2523 ns	0.0002 ns	0.0002 ns
Increment06	0.5138 ns	0.0003 ns	0.0003 ns

PR:

Method	Mean	Error	StdDev
Increment01	0.0000 ns	0.0000 ns	0.0000 ns
Increment02	0.0000 ns	0.0000 ns	0.0000 ns
Increment03	0.0000 ns	0.0000 ns	0.0000 ns
Increment04	0.1300 ns	0.0009 ns	0.0007 ns
Increment05	0.3994 ns	0.0002 ns	0.0005 ns
Increment06	0.6680 ns	0.0010 ns	0.0009 ns

As for returning a value (like the issue #2305), I'm not sure if my 9800X3D is the best representative CPU compared to what dotnet/performance runs on, but I got these results:

[Benchmark] public Vector3 Vector3() => default;

master:

Method	Mean	Error	StdDev
Vector3	0.0146 ns	0.0026 ns	0.0022 ns

PR:

Method	Mean	Error	StdDev	Median
Vector3	0.0000 ns	0.0001 ns	0.0001 ns	0.0000 ns

• Overhead return type is always void for a fair baseline for all benchmark return types.

Can you explain this a bit more? The overhead is now some "fixed" workload measurement regardless of benchmark method signature?

It's a fix for #2305. Not quite, it still passes all the same arguments, only the return type is fixed to void. My previous attempt in #2309 was a failure. Essentially, the overhead disregards the cost of the return type, so the actual result will include the cost of the return type. Practically, you could reliably measure the cost of returning a large struct, which is currently impossible which has weird results currently.

[timcassell]

@AndyAyersMS Ignoring net462 (dotnet/performance#4869), I tried build with these changes, and I got this error.

Mono.Cecil.ResolutionException: Failed to resolve System.IO.FileOptions

I can't find anything about System.IO.FileOptions, except an enum, which I think is weird because it's not an assembly. Any ideas?

[AndyAyersMS]

@AndyAyersMS Ignoring net462 (dotnet/performance#4869), I tried build with these changes, and I got this error.

Mono.Cecil.ResolutionException: Failed to resolve System.IO.FileOptions

I can't find anything about System.IO.FileOptions, except an enum, which I think is weird because it's not an assembly. Any ideas?

@LoopedBard3 any ideas? Android support?

[Copilot]

Pull Request Overview

A concise summary:

• Simplifies benchmark type code generation by removing legacy helpers, unifying loop IL generation, and using direct stack operations.
• Streamlines declarations providers to just Sync/Async variants, dropping multiple specialized classes.
• Introduces a Weaver task (WeaveAssemblyTask) in BenchmarkDotNet.Weaver to automatically apply NoInlining to all [Benchmark] methods.

Reviewed Changes

Copilot reviewed 45 out of 45 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/BenchmarkDotNet/Helpers/Reflection.Emit/MethodBuilderExtensions.cs	Removed unused GetParameterTypes extension and System using.
src/BenchmarkDotNet/Helpers/Reflection.Emit/IlGeneratorStatementExtensions.cs	Refactored loop emission from index-based to decrement-based (EmitLoopBeginFromArgToZero / EmitLoopEndFromArgToZero).
src/BenchmarkDotNet/Helpers/Reflection.Emit/IlGeneratorEmitOpExtensions.cs	Replaced large indirect store switch with EmitStarg helper.
src/BenchmarkDotNet/Helpers/Reflection.Emit/IlGeneratorDefaultValueExtensions.cs	Deleted obsolete default-value IL emission helpers.
src/BenchmarkDotNet/Helpers/Reflection.Emit/IlGeneratorCallExtensions.cs	Removed unused DeclareOptionalLocalForInstanceCall.
src/BenchmarkDotNet/Engines/Consumer.cs	Stripped out unused SupportedTypes and helper methods.
src/BenchmarkDotNet/Code/DeclarationsProvider.cs	Collapsed multiple provider subclasses into SyncDeclarationsProvider and AsyncDeclarationsProvider.
src/BenchmarkDotNet/Code/CodeGenerator.cs	Removed conditional compilation and wiring to old providers; updated provider selection logic.
src/BenchmarkDotNet.Weaver/src/WeaveAssemblyTask.cs	Added WeaveAssemblyTask MSBuild task and custom resolver to apply NoInlining.

Comments suppressed due to low confidence (3)

src/BenchmarkDotNet/Helpers/Reflection.Emit/IlGeneratorStatementExtensions.cs:123

• [nitpick] The inline IL comment references IL_0002, which no longer matches the updated loopStartLabel. Update or remove this outdated offset comment to avoid confusion.

//                IL_0011: bge.s IL_0002

src/BenchmarkDotNet/Code/DeclarationsProvider.cs:51

• SyncDeclarationsProvider does not implement the abstract members ReturnsDefinition, OverheadMethodReturnType, and OverheadImplementation from DeclarationsProvider, which will cause compilation errors.

internal class SyncDeclarationsProvider : DeclarationsProvider

src/BenchmarkDotNet/Code/DeclarationsProvider.cs:56

• AsyncDeclarationsProvider is missing overrides for the abstract properties ReturnsDefinition and OverheadImplementation from DeclarationsProvider, leading to incomplete implementation.

internal class AsyncDeclarationsProvider : DeclarationsProvider