support code coverage when testing

[andrewrk]

Add a --coverage argument to zig test and have it generate a nice report.

Related:

• debug info audit - many virtual memory addresses have strange source locations #20989
• unreachable code paths need to be excluded from having coverage instrumentation #20992

[PavelVozenilek]

Based on my unpleasant experience with gcov (GCC coverage tool) in foolish drive for 100% coverage: there should be some easy way to label certain lines or whole functions as "doesn't need coverage". All kinds of unlikely situations, impossible to emulate.

gcov doesn't support anything like this, so I write a tool which scanned all C source files, searched for "no coverage" tags and removed these lines from gcov output.

It looked like:

if (...)  {
    exit(1); //--   (comment "//--" means no coverage for this line)
}

// The whole function is taken out of coverage via the //-->> and //<<-- brackets
void foo(void) { //-->>
  ...
} //<<--

There should be also some way to skip coverage checking for 3pp libraries, w/o touching their source code.

[tiehuis]

Just though I'd mention that kcov works perfectly as is with zig at the moment. I think coverage in general is slightly more useful than it normally would be for zig as well, since it clearly shows what segments of code were never compiled. This helps with compile-time evaluation heavy bits.

Only steps required are:

zig test bigint.zig
kcov kcov-out ./zig-cache/test
firefox kcov-out/index.html

[andrewrk]

Oh that's really interesting with the defers in yellow. The number tells you how many different ways there were to exit the block, and it only goes green if you manage to hit them all (in this case by having each try activate the error handling). Same thing with try, it'll be yellow unless you managed to test the error return.

The defer thing might change with #283

[andrewrk]

I tried this using zig test on the std lib. It's really quite useful, but I see some confusing output. In my test results I see:

Test 341/466 os.path.join...OK
Test 342/466 os.path.isAbsoluteWindows...OK
Test 343/466 os.path.isAbsolutePosix...OK
Test 344/466 os.path.windowsParsePath...OK
Test 345/466 os.path.resolve...OK
Test 346/466 os.path.resolveWindows...OK
Test 347/466 os.path.resolvePosix...OK
Test 348/466 os.path.dirnamePosix...OK
Test 349/466 os.path.dirnameWindows...OK
Test 350/466 os.path.basename...OK
Test 351/466 os.path.relative...OK
Test 352/466 os.path.real...OK

but in the coverage report I see no coverage:

I wonder what's going on here.