Unexpected allocation results on dry run

[timcassell]

I sometimes want to do a dry run just to see the allocations, not caring about the benchmark times. But the results were way off when I tried to do that.

Job.ShortRun/Default:

|                Method |        Job |            Runtime |          Mean |       Error | Allocated |
|---------------------- |----------- |------------------- |--------------:|------------:|----------:|
|     AsyncYieldTaskInt | Job-YPOTHS |           .NET 6.0 | 1,785.8192 ns |  80.9880 ns |     120 B |
| AsyncImmediateTaskInt | Job-YPOTHS |           .NET 6.0 |    20.3836 ns |   2.7206 ns |      72 B |
|               SyncInt | Job-YPOTHS |           .NET 6.0 |     0.4281 ns |   0.0409 ns |         - |
|     AsyncYieldTaskInt | Job-OPYGUE | .NET Framework 4.8 | 3,667.3738 ns | 182.9334 ns |     289 B |
| AsyncImmediateTaskInt | Job-OPYGUE | .NET Framework 4.8 |    25.2346 ns |   3.2654 ns |      80 B |
|               SyncInt | Job-OPYGUE | .NET Framework 4.8 |     0.4940 ns |   0.0421 ns |         - |

Job.Dry:

|                Method |        Job |            Runtime |       Mean | Error | Allocated |
|---------------------- |----------- |------------------- |-----------:|------:|----------:|
|     AsyncYieldTaskInt | Job-NZRJKJ |           .NET 6.0 | 6,627.7 us |    NA |    1184 B |
| AsyncImmediateTaskInt | Job-NZRJKJ |           .NET 6.0 |   778.8 us |    NA |     456 B |
|               SyncInt | Job-NZRJKJ |           .NET 6.0 |   387.0 us |    NA |     384 B |
|     AsyncYieldTaskInt | Job-DOGRVK | .NET Framework 4.8 | 3,823.9 us |    NA |         - |
| AsyncImmediateTaskInt | Job-DOGRVK | .NET Framework 4.8 | 1,762.9 us |    NA |         - |
|               SyncInt | Job-DOGRVK | .NET Framework 4.8 |   519.2 us |    NA |         - |

[Benchmark]
public async Task<int> AsyncYieldTaskInt()
{
    await Task.Yield();
    unchecked { ++counter; }
    return 42;
}

[Benchmark]
public Task<int> AsyncImmediateTaskInt()
{
    unchecked { ++counter; }
    return Task.FromResult(42);
}

[Benchmark]
public int SyncInt()
{
    unchecked { ++counter; }
    return 42;
}

cc @adamsitnik

[timcassell]

I'm seeing similar weirdness for long-running benchmarks even with the default job:

|                Method |        Job |            Runtime | Allocated |
|---------------------- |----------- |------------------- |----------:|
|     AsyncYieldTaskInt | Job-WLKFHY |           .NET 6.0 |    3176 B |
|     AsyncYieldTaskInt | Job-HTOUPK | .NET Framework 4.8 |         - |

[Benchmark]
public async Task<int> AsyncYieldTaskInt()
{
    await Task.Delay(TimeSpan.FromSeconds(1));
    unchecked { ++counter; }
    return 42;
}

[adamsitnik]

You need to use memory profiler to figure out who is allocating the memory. It's most likely some other thread (like Tiered JIT background thread)

[timcassell]

It seems that is expected when methods have not been promoted to a higher JIT tier. For .Net Framework, I'm not really sure, I'm guessing it's just not precise enough to measure a single run.