Discuss and track points raised by @nietras

[mattwarren]

Original discussion aspnet/KestrelHttpServer#647 (comment) and aspnet/KestrelHttpServer#647 (comment)

My biggest issue I guess is that BenchmarkDotNet does not come with memory profiling as default and with it enabled by default. This could be as simple as a minimum to add GC collection counts for the different generations (this is pretty easy to measure and does not require external things).

Additionally, diagnosers are as far as I can tell still not available as a nuget package.

There is very little info on using and setting up diagnosers, again specifically for memory benchmarking but also something I think has great value actual machine assembly code output.

• Consider enabling Memory Profiling by default?
• NuGet package for Diagnosers
• Better docs for Diagnosers
• Improved (or fix) the Assembly and/or IL Diagnoser
• Fix Referenced assembly dll-file (directly via file) not referenced in generated Program.csproj #72

[nietras]

Awesome! If you need any feedback let me know.

[mattwarren]

Does the lists of tasks I added cover everything, or is there something I missed out?

[nietras]

No that is everything. Thanks.

[adamsitnik]

@mattwarren I know that you have moved Diagnosers from main part mostly because of DNX/CORE support. I have managed to get it working for DNX, and I believe that with some #ifdefs for CORE I could merge it back again to main project. Would that be useful or you would rather stay with separate project?

[mattwarren]

@adamsitnik sorry, I missed this comment, the BenchmarkDotNet issues have been busy the last week or so!!

I know that you have moved Diagnosers from main part mostly because of DNX/CORE support.

The other reason is that @AndreyAkinshin doesn't want any third party dependencies in the main/core project (which I agree with BTW). So there's also this to consider.

At the moment the Diagnosers are too hard to use, because they're not included OOTB, you have to use the GitHub source. Maybe once we get NuGet package for them and better docs that'll be enough.

I'm not sure what the best answer is @AndreyAkinshin what do you think?

[AndreyAkinshin]

Diagnosers should be a separated project with a separated NuGet package.
In the future, I really want to run benchmarks on Linux (plus Mono or CoreCLR). So, it will be another separated project and another NuGet package for Linux diagnosers (via xperf). BTW, maybe we should rename current package to BenchmarkDotNet.Diagnostics.Windows.

[nietras]

I agree that platform specific diagnosers should be separated in separate nuget packages. The main point is that there should be nuget packages for easy access.

If these could be made platform agnostic in some way at runtime it would of course be best, that is you choose a ClrMemoryDiagnoser in your benchmark and the proper platform specific diagnoser implementation is chosen based on current platform or something.

[AndreyAkinshin]

We can do the following system of NuGet packages:

• BenchmarkDotNet: core library without any third party dependencies
• BenchmarkDotNet.Diagnostics.Windows (or BenchmarkDotNet.Diagnostics.Etw)
• BenchmarkDotNet.Diagnostics.Unix (or BenchmarkDotNet.Diagnostics.Xperf)
• And BenchmarkDotNet.Ultimate to rule them all.
  So, If I need only core without dependencies, I install BenchmarkDotNet. If I need some specific dependencies (I know what I'm doing: e.g. I want to have some Windows-specific diagnostics, some plotters based on 3rdParty libraries, etc.; then I use only specific set of packages). If I don't care about dependencies and just want to have all features from the box, I install BenchmarkDotNet.Ultimate.

[adamsitnik]

One of the side effects of my recent changes is that VS produces a nuget package for Diagnostics. You can just build the solution and take it from artifacts\bin\BenchmarkDotNet.Diagnostics\Release.

So if you would like to release it now in the actual form it should be relatively easy.

[AndreyAkinshin]

@mattwarren, do we need to do anything else related to this issue?

[mattwarren]

No it's fine, it can be closed