Store more metadata in JLL packages/tarballs

[staticfloat]

I think we need to start storing more metadata within tarballs/JLL packages. In particular, I think the following list of things would be nice to store:

• BB version+gitsha used for build: useful for build caching
• hash of source build recipe: useful for build caching
• Source URLs: useful for GPL compliance
• Source licenses (the contents of share/licenses/${src_name}/): useful for GPL compliance

We should store these in the tarballs, but also in the JLL package itself, for easy inspection.

[giordano]

I think this has been partially addressed by #794, at least as long as the README of the JLL packages is concerned

[fingolfin]

More things:

• homepage of the project being wrapped
• a short description of what it is / does
• possibly information about maintainers (of the JLL)?!? I.e. "whom should I contact if I have trouble with this JLL" (not sure how useful this really is, though, perhaps people should just submit Yggdrasil issues and we'll sort it out based on the git log of a given recipe?)

[giordano]

possibly information about maintainers

Yeah, Spack has a nice bot which pings the designated maintainer of a package when it's touched.

[vchuravy]

What would also be somewhat cool is to record the the flags to cmake/configure.

[jeremiahpslewis]

• possibly information about maintainers (of the JLL)?!? I.e. "whom should I contact if I have trouble with this JLL" (not sure how useful this really is, though, perhaps people should just submit Yggdrasil issues and we'll sort it out based on the git log of a given recipe?)

Think this is relevant for two reasons: 1) would make it possible to spin up a github action for ‘air traffic control’; 2) not just a question of who to call, but who has a stake in the game; a maintainer role established in the project culture should make ppl feel engaged, responsible. You only get this at scale if it’s formalized.

[giordano]

Other things I believe might be good to add to a metadata file:

• list of all products, for all platforms, with the name with which we dlopen it. I think that having it could help us address Problems with musl's ldso on 1.6 JuliaLang/julia#40556 if we use in DT_NEEDED the dlopen name of the needed lib
• list of BuildDependencys (per platform, even if the same for all platforms). In this way we can propagate build dependencies, which are lost in the registry
• maybe also version number of toolchains used for each platform (see expose compiler version used to build JLL somewhere #1110 for the background)

[fingolfin]

So, to get a bit more specific: how would we store those? Perhaps as a meta.json file, which is added into the JLL repository, and then we can just (de-)serialize a suitable Dict with whatever keys we want. That should make it easy to add/remove keys.

Does that seem OK? Is JSON fine? If not what else? Should the file be called something else?

Next, we need to get the data: Some of it we can fill in automatically:

• BB version+gitsha used for build
• hash of source build recipe
• Source URLs
• list of all products, for all platforms, with the name with which we dlopen it
• list of BuildDependencys (per platform, even if the same for all platforms), presumably including version/compat restrictions
• version number of toolchains

Some must be provided by the user / passed to build_tarballs

• license, e.g. an SPDX identifier
• upstream homepage
• a description
• maintainer(s)

So we need a mechanism for passing this information to build_tarballs. Some possibilities:

1. Add an optional keyword argument metadata which takes a Dict{Symbol,Any} or so -- easiest to implement, flexible, but also harder to sanitize
2. Add a fixed list of KW args, e.g. maintainers::Vector{String} = String[], desc::String = "", homepage::String = "", license::String = "" -- also simple, less flexible, but also easier to use (arguable)
3. Add a struct PackageMetadata with all this info, but perhaps also space for other entries -- more complex, easy to use for common cases, could still allow for arbitrary package specific keys

What do people prefer? Other suggestions?

Once these two pieces are in place, we can build tooling atop them, for example:

• add some of the metadata into the generated README
• add a JLL wrapper function for accessing the metadata of a JLL
• add code which queries existing JLLs for e.g. build dependencies and uses these automatically in a builder recipe (e.g. a function add_recursive_build_deps(deps::Vector))

[jeremiahpslewis]

@fingolfin

Does that seem OK? Is JSON fine? If not what else? Should the file be called something else?

What do you think about using TOML? Looking at the *_jll repos it seems like that is the dominant metadata file format, but I may be missing something.