Produce tests per line data

[cgewecke]

It has been proposed in JoranHonig/vertigo 20 that being able to identify exactly which tests exercise which line of Solidity would be useful data for mutation testing and fault-localization techniques.

The issue linked to above includes lots of discussion about how to achieve this. Would probably be a command separate from coverage & just consume some services available through the beta's new API.

Maybe even in it's own repo at sc-forks org? With SC as a dependency?

See also: #209

[yxliang01]

Cool! I think this issue moves solidity-coverage from a pure utility to become a more framework or library-like project :)

[JoranHonig]

It's been a while 😅. If you could point me in the right direction then I might be able to contribute this feature.

[cgewecke]

Hi @JoranHonig!

That sounds great. Am about to make a set of +/- large changes here as part of #541.

Could I get back to you next week on this? (Will have a better sense of how this might be possible then.)

---

Am copying over some of the design details from the Vertigo discussion.

The generated object might look something like this?

{
 File.sol: {
   8: [
     {
       test: 'it errors when called by alice',
       file: '/Users/area/code/tests/alice.js'
     }, ...
   ], ...
 },
 AnotherFile.sol: {...},
}

The test names would come from a custom mocha reporter attached before the test command is run.

Mocha details are:

reporter's 'test' hook (for it blocks)

runner.on("test", info => {
}

info (much abbreviated)

info: {
  suite: {
    title: 'Contract: EtherRouter Proxy',
    file: '/Users/cgewecke/code/eth-gas-reporter/mock/test/etherrouter.js' 
  },
  test: {
    "title": "Resolves methods routed through an EtherRouter proxy",
    "file": "/Users/cgewecke/code/eth-gas-reporter/mock/test/etherrouter.js",
  } 
},

[JoranHonig]

Thanks! Take your time with #541 👍.

The dataformat looks good to me! It'd be perfect for mutation testing.

[cgewecke]

Update: have a working version of this in #593 which will published as part of the new v.0.8 beta in the next couple days.

Am calling this data a "test matrix". It can be generated using the --matrix flag and get's written to testMatrix.json in the project's root repository. (The file name is configurable).

Output for Truffle's metacoin repository looks like:

{
 "contracts/MetaCoin.sol": {
  "17": [
   {
    "title": "should put 10000 MetaCoin in the first account",
    "file": "test/metacoin.js"
   }
  ],
  "21": [
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ],
  "22": [
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ],
  "23": [
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ],
  "24": [
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ],
  "25": [
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ],
  "29": [
   {
    "title": "should call a function that depends on a linked library",
    "file": "test/metacoin.js"
   }
  ],
  "33": [
   {
    "title": "should put 10000 MetaCoin in the first account",
    "file": "test/metacoin.js"
   },
   {
    "title": "should call a function that depends on a linked library",
    "file": "test/metacoin.js"
   },
   {
    "title": "should send coin correctly",
    "file": "test/metacoin.js"
   }
  ]
 },
 "contracts/ConvertLib.sol": {
  "7": [
   {
    "title": "should call a function that depends on a linked library",
    "file": "test/metacoin.js"
   }
  ]
 }
}

[JoranHonig]

@cgewecke 🎉 This is awesome!

Thanks for this feature!

[JoranHonig]

@cgewecke I'm playing around with this right now (doing an implementation of the fault localisation algorithm you mentioned 😄).

Here is the implementation that I'm working on https://github.com/JoranHonig/tarantula

I think it might be necessary to use the fullTitle for each test, otherwise you might be able to get collisions (multiple tests with the same title in a file).

[JoranHonig]

Another question: I think mocha doesn't give a "file" for the tests that are actually written in solidity (if I'm using truffle). Is the "file" param also optional in solidity-coverage's results?

[cgewecke]

@JoranHonig Wow! Super cool.

multiple tests with the same title in a file

These are filtered out while generating the matrix. If a title occurs twice for the same line & test file it's skipped. Please let me know if you see these in the output though since they shouldn't be there.

tests that are actually written in solidity

Unfortunately solidity-coverage doesn't support Truffle's native solidity testing framework atm. (Issue #578) It's disabled because of a bug in the Truffle compilation pipeline. Will need to revisit this

[JoranHonig]

@cgewecke do you know a clean way of getting out the test results (passed/failed) when checking test coverage?

--matrix would output the coverage results to a file (super nice), but not this information right?

[cgewecke]

@JoranHonig It's possible to start collecting it - the mocha reporter has 'pass' and 'fail' hooks that could be repurposed for this.

Out of curiosity - is this data useful for the mutation tester or fault localization or something else?

One (possibly incorrect) premise I've had is that matrix reporter would be run once just to generate a map. And that the execution context is a test suite with probably all passing tests.

[cgewecke]

@JoranHonig How did you produce the test data in your tarantula unit test?

Perhaps the --matrix should also generate that?