Coverage Data

[JoranHonig]

For many mutation testing optimisations it is required / helpful if we have code coverage information.
Specifically we need to be able to determine which tests cover specific lines of code.

[yxliang01]

https://github.com/sc-forks/solidity-coverage might be a good one to use. More importantly, many major smart contract projects have already implemented the coverage generation script for running it automatically end-to-end.

[JoranHonig]

@yxliang01 would it be possible to use this tool to get coverage information for each specifc unit test?
Or would we basically need to generate a coverage report for each diferrent unit test?

[yxliang01]

@JoranHonig As far as I know, no. But, do we really need to get the coverage for each unit test?

[area]

I believe that the intention is to only run tests during a mutation that touch the line you're mutating. There may be others, but I'm not familiar with mutation testing in general to be able to comment if there are cleverer improvements beyond that.

I agree this is not something that can currently be done with solidity-coverage, but pinging @cgewecke just so he knows that this something that people are talking about. The beta 0.7.0 branch he has been working on (which is radically reworked for the better) is certainly much closer to being able to do this than the 0.6.x branches had been, due to a closer integration with truffle.

[JoranHonig]

I believe that the intention is to only run tests during a mutation that touch the line you're mutating.

@area This is exactly the optimisation that I'm aiming for.

If such a feature could be provided by solidity-coverage would be awesome!

[yxliang01]

I see. But, I feel one thing could possibly already be done is to get the coverage data for one test file by isolating it from others.

[JoranHonig]

@yxliang01 True. We might also be able to leverage Mocha's grep functionality to execute singular tests.

[cgewecke]

I believe that the intention is to only run tests during a mutation that touch the line you're mutating.

The new coverage client continuously updates an in-memory map of file:line hits. And mocha's third-party-reporters might also be helpful - they let you run hooks for each stage in the test suite execution. So you could do something like...

• instrument all the Solidity files
• compile
• run full test suite with the coverage client
  • within the mocha reporter, during each it block
    • inspect the coverage file:line map to detect new hits
    • ++ or create an entry in a map of file:line --> [test descriptions]
    • reset the coverage file:line map to zero hits.

I feel one thing could possibly already be done is to get the coverage data for one test file by isolating it from others.

This actually might be the simplest thing to do. The only drawback could be that in practice each file can have many tests. On the other hand, they're often related to the same code.

Incidentally, this topic once came up at solidity-coverage in relation to other research in issue 209. In that case, the engineer was trying to do risk assessment based on an academic paper which IIRC suggests you can narrow the field of audit inquiry by looking at how many tests touch a given line of code. It's called the Tarantula Automatic Fault-Localization Technique.

(So you might get a 2-for-1 if you build this plus This Week In Ethereum lists/highlights all security related projects with the word 'tarantula' in their title.)

[JoranHonig]

Hi @cgewecke,

Thanks for the extensive response!

This actually might be the simplest thing to do. The only drawback could be that in practice each file can have many tests. On the other hand, they're often related to the same code.

Assuming that I can find a cheap way to list all the tests in the project, I could execute a single one of those tests by adding something like mocha: { grep: "test_name" } to module.exports truffle-config.js right?
In which case it is possible to test coverage at a higher granularity.

Incidentally, this topic once came up at solidity-coverage in relation to other research in issue 209. In that case, the engineer was trying to do risk assessment based on an academic paper which IIRC suggests you can narrow the field of audit inquiry by looking at how many tests touch a given line of code. It's called the Tarantula Automatic Fault-Localization Technique.

That's very interesting! I'm not opposed to extending Vertigo with a command that allows you to do fault localisation.

Ps. Were you planning to extend solidity-coverage to report the coverage per unit test?

[cgewecke]

I'm not opposed to extending Vertigo with a command that allows you to do fault localisation.

Oh ok, that's great! I will tentatively add tests/line to the work scheduled for solidity-coverage - suspect there might be a number of uses for that data set.

May be a bit before it gets done, I'm slightly over-subscribed at the moment.

[yxliang01]

@cgewecke You meant you plan to implement reporting coverage per unit test?

[cgewecke]

@yxliang01

You meant you plan to implement reporting coverage per unit test?

I was thinking something more like:

$ solidity-coverage --outputTestsPerLine

which would produce an object like:

[
  FileName.sol: {
    8: ['it errors when called by alice', 'it can set an owner'],
    13: ['it etc...' ]
  },
  AnotherFile.sol: {...},
] 

This data would just be an input that mutation testing and fault localization strategies could use.

What are you thinking of?

[yxliang01]

@cgewecke Hmm, if we simply get the test description, it's not enough to do fault localization though. I would expect there are repeated names. But, if we can ensure the fact that one test case can only maximally add 1 element onto the array, then this is good enough.e

[cgewecke]

if we can ensure the fact that one test case can only maximally add 1 element onto the array

@yxliang01 Ah! Some questions...

• Could you write the object as you imagine it?
• Is 1 element a filename:line element?
• Is the object I proposed inside out from what it should be?
• Do fault-localization and mutation consume the same data set, or slightly different?

We can definitely deduplicate the names in the array, that's a good point.