Add sourcelist to compilation artifacts

[JoranHonig]

Component

Forge

Describe the feature you would like

The new compilation artifacts have almost all the necessary information needed for downstream tools to work with forges' compilation artifacts.

The one thing that's missing is the solc sourcelist, this provides a mapping from file indices to file names. This is necessary when you want to map a bytecode location to a source location in a solidity file.

Additional context

No response

[mattsse]

checked https://docs.soliditylang.org/en/v0.8.13/using-the-compiler.html and solc arguments for sourcelist but came up short, but perhaps you're referring to the sourceMap object of the bytecode object?

[onbjerg]

Maybe sourcelist -> sources?

[JoranHonig]

The terminology changes from tool to tool, apologies I should've used the solc wording.

@onbjerg yes! However, not everything about the sources is strictly necessary. I'd particularly like to have the file id's.

They will be in output['sources'][sourceName]['id']. Some other tools create / output a list or mapping from id to sources.

[mattsse]

I see, this is part of the evm.bytecode.generatedSources - Sources generated by the compiler setting, setting evm.bytecode.generatedSources should result in an object

"generatedSources": [{
              // Yul AST
              "ast": {/* ... */},
              // Source file in its text form (may contain comments)
              "contents":"{ function abi_decode(start, end) -> data { data := calldataload(start) } }",
              // Source file ID, used for source references, same "namespace" as the Solidity source files
              "id": 2,
              "language": "Yul",
              "name": "#utility.yul"
            }]

if this is set as extra-file, should we emit a list of all ids to sources?

[JoranHonig]

Not 100% I understand what setting as extra-file means, but yeah that would be perfect!

A format that maps id's to files would be best for the purposes of resolving source map elements. I'm not sure however of the other requirements in foundry, so I'll leave this up to you!

It would be perfectly fine to retain the existing solidity output structure for my purposes.

[mattsse]

there are currently two arguments that allow you to configure what's included in the artifact and what also should be emitted as separate file

        --extra-output <EXTRA_OUTPUT>...
            Extra output to include in the contract's artifact.
            
            Example keys: evm.assembly, ewasm, ir, irOptimized, metadata
            
            For a full description, see https://docs.soliditylang.org/en/v0.8.13/using-the-compiler.html#input-description

        --extra-output-files <EXTRA_OUTPUT_FILES>...
            Extra output to write to separate files.

which can also be set in the config file

But these are all on a per artifact (contract) basis, so perhaps we'd need some kind of top level lookup table as json file that maps ids to artifact files?

[JoranHonig]

But these are all on a per artifact (contract) basis, so perhaps we'd need some kind of top level lookup table as json file that maps ids to artifact files?

That would be perfect. Heads up though, incremental compilation (only re-compiling files with changes) and maintaining the integrity of this lookup table is going to be important.

[gakonst]

@robertabbott maybe this

[clouds56]

Is any progress?
Shall we have manually insert id in metadata.sources.*.id in separate artifacts? Maybe an option that could be enabled?

  "metadata": {
    "compiler": {
      "version": "0.8.19+commit.7dd6d404"
    },
    ...
    "sources": {
      "node_modules/@openzeppelin/contracts/token/ERC20/ERC20.sol": {
        "keccak256": "0x4ffc0547c02ad22925310c585c0f166f8759e2648a09e9b489100c42f15dd98d",
        "urls": [
          "bzz-raw://15f52f51413a9de1ff191e2f6367c62178e1df7806d7880fe857a98b0b66253d",
          "dweb:/ipfs/QmaQG1fwfgUt5E9nu2cccFiV47B2V78MM1tCy1qB7n4MsH"
        ],
        "license": "MIT",
        "id": 1 // add this
      },
    }
  }

If this is accept, I could help here.

[klkvr]

After #140 cache now keeps a mapping source -> build where each build keeps a list of source ids, which match those appearing in sourcemaps