Add possibility to run coverlet in standalone mode

[jakubch1]

It would be great to run coverlet without any tests and collect code coverage for application. What I mean here is to for example having console application, instrument it and then start it (without any vstest or anything). When application finishes collect code coverage.

For example:
Let's assume we have app.exe and references of it ref1.dll and ref2.dll
It would be good to be able to

1. instrument all above through command line for example:
   coverlet instrument app.exe ref1.dll ref2.dll
2. run application
   .\app.exe
3. get code coverage from application run.

It could also work in such way that during application execution it will be "logging" all hits to file:line (it could send it to some "monitor")
And after the process finishes it could generate code coverage report based on such data (by another coverlet produceReport data.txt). Such approach would also give possibility to get coverage report for particular time frame without need of process termination.

Such feature will give possibility to collect code coverage for:

1. backend services when tests are just communicating with it through http etc.
2. any other than vstest test runner

[MarcoRossignoli]

Some notes:

It could also work in such way that during application execution it will be "logging" all hits to file:line (it could send it to some "monitor")

Maybe this is not doable, in past coverlet registered hits on file directly for every call, but it was very slow so we moved to in ram booking in a int array and at the end we flush to disk, so I think that send coverage for every hits realtime won't work well, think for example a loop that would send n hits for every cycle, it was slow on file, I think will be worst with network, BTW I don't have numbers and I'm speculating, maybe tcp buffering works well also in this case.
Maybe some bulk send every some time, when I talk about hits file imagine a simple array of int https://github.com/tonerdo/coverlet/blob/master/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L22

This is the place where instrumentation start, for every module(found or filtered) we generate an InstrumenterResult with all needed data(branches/lines) to calculate coverage from hits. This array is the results of instrumentation in you sample app.exe ref1.dll ref2.dll list of 3 InstrumenterResult object that we need to "save" somewhere at the moment we keep in memory for collectors and .net tool and to file for msbuild.

instrument all above through command line for example:

Some ideas

1. Does it make sense build a coverlet command or maybe it's more useful expose core feature as package and let developers to interact with engine(similar to this experiment, here the exposed interface https://github.com/tonerdo/coverlet/pull/628/files#diff-743457133023fd26c409505936137f4f) and build what they want to it, I mean your scenario address your specific issue.

Take a look at live sample(in PR) https://twitter.com/MarcoRossignoli/status/1201546945589841922 in this case first command instrument dll and generates the instrumentation result and the second command take the instrumentation result and use it to get realtime coverage, because they need to know the mappings.

2. Another idea could be add a new tcp mode tracker, where during instrumentation an address is passed ip:port and bulk parameter(how many time send data) and tracker simply send data to that address realtime and on process exit.

coverlet instrumentation --filter globbingfilter --type tcp --tcpAddress 127.0.0.1 --tcpPort 1000 --frequency 100 --instrumentationResultFilePath file

frequency: send every new records https://github.com/tonerdo/coverlet/blob/528a769c9ab98762935490372e40720ed7219618/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L52

And the protocol cloud be

• Connect() on first tracker access
• RecordedHits(json) <- collection libName/hitCandidate+hitcount
• Disconnect() process exit

Also we need to expose an object model to allow the accounting on server side, pseudo code

Instrumentation inst = new Instrumentation(instrumentationResultFilePath);
inst.ReadCoverage(received json);
inst.ShowSomethingSomeWhere();

Or simply send a binary format that server only need to append to some stream(file, memory) something similar to a database transaction log and another command coverlet reports hitfilesavedbyserver can handle and after pass to generate report part of tool as normal hit file.
Or again...simply send all hits file every some frequency.
Also in this case we should expose some "server" classes to take the stream and write somewhere.

cc: @tonerdo @petli

[MarcoRossignoli]

@jakubch1 one question, it's not clear to me how do you see "the server side part" that receive the hits:

Is implemented in coverlet like a command that stay up on a port until some signal?
Is a model exposed by coverlet and used by server?

[jakubch1]

@MarcoRossignoli I will try to create something based on this what you said and I will see if this works. Maybe you are right that we don't need to have this inside coverlet. Basically people can develop what they need. Any other user asked for such thing?

During the call you said that sometimes there is issue that report is not generated because testhost process is terminated too fast. Do you have any reports for it? In case of vstest it could work in such way that array with hits could be inside shared memory (accessible by both data collector and testhost) and when data collector gets event that all tests are done it (data collector) could generate report.

[MarcoRossignoli]

@MarcoRossignoli I will try to create something based on this what you said and I will see if this works.

What do you mean?Do you use coverlet as coverage engine or not?

@MarcoRossignoli I will try to create something based on this what you said and I will see if this works. Maybe you are right that we don't need to have this inside coverlet. Basically people can develop what they need. Any other user asked for such thing?

You(vstest old team) and some other users asked for standalone release of coverlet core features #212 (check thumbs up), this is an explorative PR #628

During the call you said that sometimes there is issue that report is not generated because testhost process is terminated too fast.

Main issue #210
VSTest issue microsoft/vstest#1900 (comment)
Some issues https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+rabbitmq+is%3Aclosed
Some fix on other repo SixLabors/ImageSharp#1094

In case of vstest it could work in such way that array with hits could be inside shared memory (accessible by both data collector and testhost) and when data collector gets event that all tests are done it (data collector) could generate report.

We don't have issue with collectors because in-process one is synched with test end events, we have that issue with msbuild integration because it's based on process exit event.
I asked for an increment of timeout microsoft/vstest#2379 to mitigate the issue on msbuild(the most used driver)

This is our in-proc implementation https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/InProcDataCollection/CoverletInProcDataCollector.cs#L46

Collectors integration is the more resilient way to use coverlet but there are some unsupported features due to vstest platform behaviour:

• netfx support 0% coverage reported on netfx target #705
• show results on terminal Show coverage summary on terminal #681
• automatic merge at the end of the tests MergeWith in dotnet test is not working con vstest integration(collectors) #662 (solution wide)
• Specify deterministic output directory Unable to specify output directory when using VSTest integration? #500 (I opened an issue some days ago Avoid guid at the end of outputDirectory microsoft/vstest#2378)

Aggregate collectors issues https://github.com/tonerdo/coverlet/issues?q=is%3Aissue+is%3Aopen+label%3Adriver-collectors

[jakubch1]

What do you mean?Do you use coverlet as coverage engine or not?

I want to try to build something on top of coverlet to check that it allows to replace stuff correctly with different implementation.

We don't have issue with collectors because in-process one is synched with test end events, we have that issue with msbuild integration because it's based on process exit event.
I asked for an increment of timeout microsoft/vstest#2379 to mitigate the issue on msbuild(the most used driver)

I don't understand this. Msbuild integration is internally running vstest so what is the issue?

This is our in-proc implementation https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/InProcDataCollection/CoverletInProcDataCollector.cs#L46

If I understand correctly this works only for .net Core. So suggestion to use shared memory between OutProcCollector and testhost is still valid. I was checking other threads regarding this issue and Mayank was also suggesting this. By using shared memory and generating report inside data collector you could maybe even not need Coverlet nuget package reference inside test projects. We could make call and discuss this.

* automatic merge at the end of the tests #662 (solution wide)

I'm planning to work on this feature in near future :) The initial idea is to create something as session inside vstest.console.exe (process will run in client mode like in VS) and vstest.console will gather all coverage reports from test projects executions. On session finish it will group by coverage reports based on data collector uri. Then data collectors will again be invoked to merge reports. We will let you know when we agree on design.

[MarcoRossignoli]

I don't understand this. Msbuild integration is internally running vstest so what is the issue?

The issue is that with msbuild version the flush of hits to file is done on process exists and with collectors is done by in-proc collector, so if during process exits vstest plat kill host process msbuild version doesn't flush file correctly.

If I understand correctly this works only for .net Core

At the moment yes...because I don't know why collectors wasn't compiled as netstandard2.0 by @vagisha-nidhi , but will support also netfx if we'll compile for netstandard.
I did same question on vstest repo, can we compile collectors for netstandard?Are netstandard2.0 collectors supported by vstest plat? https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/coverlet.collector.csproj#L3
In that case collectors works also on netfx.

We could make call and discuss this.

Yep a good idea.

I'm planning to work on this feature in near future :)

This is great!

[jakubch1]

At the moment yes...because I don't know why collectors wasn't compiled as netstandard2.0 by @vagisha-nidhi , but will support also netfx if we'll compile for netstandard.
I did same question on vstest repo, can we compile collectors for netstandard?Are netstandard2.0 collectors supported by vstest plat? https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/coverlet.collector.csproj#L3
In that case collectors works also on netfx.

We had internal discussion with code coverage experts regarding this. We believe coverlet should use shared memory between testhost and out proc data collector and generate report inside out proc data collector. This is approach which is used in several code coverage solutions that we know. It fixes all issues with crush/fast kill of test host process.

[MarcoRossignoli]

We had internal discussion with code coverage experts regarding this. We believe coverlet should use shared memory between testhost and out proc data collector and generate report inside out proc data collector. This is approach which is used in several code coverage solutions that we know. It fixes all issues with crush/fast kill of test host process.

Maybe I wasn't clear, we don't have any problem with collectors version of coverlet, it works as expected.

The problem is with msbuild version, where there is no control on process shutdown. I try to explain better the difference between msbuild and collectors(I call "driver" the way to use coverlet runtime, for instance msbuild is a driver, collectors is a driver) run:

1. msbuild

if you want to use msbuild command you have to add coverlet.msbuild package to your test project and you can invoke coverage with command:

dotnet test /p:CollectCoverage=true

This is the most used way https://nugettrends.com/packages?months=12&ids=coverlet.msbuild today.

How it works:

1. msbuild task is injected on build pipeline https://github.com/tonerdo/coverlet/blob/master/src/coverlet.msbuild.tasks/coverlet.msbuild.targets#L49, here we select the dll to instrument and do instrumentation.
2. Test plat span testhost.exe process and start tests. Thanks to injected tracker class we collect in data memory(array of int).
3. Tests end and process testhost.exe will closes and on process exits event we flush in memory hit file to disk https://github.com/tonerdo/coverlet/blob/master/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L76
4. When tests end another msbuild task will run after target VSTest https://github.com/tonerdo/coverlet/blob/master/src/coverlet.msbuild.tasks/coverlet.msbuild.targets#L68 and in this task we load hits file and it generates file report.

Here the "fast kill" issue on point 3 does not allow to process exit to flush file correctly and we don't have any control on where msbuild task will be hosted so we cannot simply open a shared memory, but to fix this issue with your idea we should span our new process that communicates with testhost and in some way read that data on msbuild task after VSTest target. This is a huge change.

2. Collectors

if you want to use collectors integration you have to add coverlet.collector package to your test project and you can invoke coverage with command:

dotnet test --collect:"XPlat Code Coverage"

How it works:

1. When test platform see a collector named XPlat Code Coverage injects out-of-process and in-process coverlet collectors:

https://github.com/microsoft/vstest/blob/master/src/vstest.console/Processors/CollectArgumentProcessor.cs#L176 here in-process injection
out-of-process collector is injected throught friendly name https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs#L20

2. VStest plat span throught vstest.console process span 2 process datacollector and testhost inside datacollector hosts out-of-process data collector and inside testhost hosts in-process data collector.
3. OnSessionStart out-of-process data collector instruments dlls https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs#L140
4. Tests run, thanks to injected tracker class we collect in data memory(array of int).
5. On tests end in-process data collector flush hit file, here is the key https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/InProcDataCollection/CoverletInProcDataCollector.cs#L57, traker code https://github.com/tonerdo/coverlet/blob/master/src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs#L76
6. Test host process closes.
7. OnSessionEnd out of process collector read hit file and generates report file https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/DataCollection/CoverletCoverageCollector.cs#L154

So in case of collectors integration we have the guarantee of hit file flush because in-process collector listen test end event and force flush and not rely on process exits https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/InProcDataCollection/CoverletInProcDataCollector.cs#L57

So there is NO issue with collectors but there is issue with msbuild driver.

If it's not clear we should talk via chat or call, it's hard to explain in a issue post.

[jakubch1]

When we can have a meeting? I can do it today after 7PM or next week Wednesday or Friday after 6PM.

[MarcoRossignoli]

@jakubch1 sorry I read now, if you want I'm here...or ok for Wednesday after 6PM.

[bignis]

I'll chime in with my use case that I think this Issue fits well with.

I maintain a web-based business application that doesn't have very many unit tests, but has plenty of automated tests through Selenium in a web browser. I want to assess how well these Selenium web tests cover my business logic.

If I could instrument my binaries as @jakubch1 described, I could run the Selenium tests in a browser, look at the coverage report afterward, and inform my QA team of where additional Selenium tests are needed to achieve better coverage. I wouldn't need real-time reporting for my use case.

[petli]

Before going down the shared memory road have a look at this comment from the last time it was tried to see if it can be done in a way that doesn't break System.Private.CoreLib:
#276 (comment)

Edit: I'm now caught up and see that this was already noticed in #808, great!