feat: support dynamic metadata (#197)

This is a first pass at a scheme for supporting dynamic metadata in a
generic way. It keeps everything as simple as possible by using
functions in modules with a very simple signature to provide each value.
I am happy to discuss the proposed interface and make changes, but
thought it would be best to get an implementation out there to have
something concrete to discuss.

Scikit-build-core's settings have been extended to have an additional
metadata field which is just an optional list mapping string keys to
string values. For each entry in this list, if the key is also included
in the project.dynamic list, scikit-build-core will look up an a module
with a function. This should be a function that accepts the parsed
pyproject.toml (this seemed like a minimum but the final chosen
signature is very much up for debate, for example, maybe the config
settings dict should also be passed) and returns the dict that should be
merged into the pyproject["project"] entry in the loaded toml config.

Also included are a couple of core "plugins" which provide entry-points
for setuptools_scm and hatch-fancy-pypi-readme, essentially just
adapters as neither tool provides an explicit function matching the
proposed entry-point signature.

For tests, I have created a fixture which mocks the entrypoints list
with fake functions to give more direct control over what values we want
to test. I have duplicated most of the "simplest_c" test package to
allow the complete build process to be applied, but it is also possible
to test with only a pyproject.toml file, and in fact most of the tests I
have currently implemented do only use the pyproject.toml file and stop
at testing the calculated metadata, although this can of course be
changed. There is also a simple test for the setuptools_scm and
hatch-fancy-pypi-readme plugins to make sure they are working correctly.

Adding setuptools_scm as a test dependency to test the plugin had the
unexpected effect of making setuptools backend tests fail because of the
file finder hook setuptools_scm provides which includes unexpected files
(because they are under source control) in the sdist. To solve this, I
have modified those tests to copy the package files to a temporary
location before building so that the build source is not under git.

---------

Signed-off-by: Henry Schreiner <[email protected]>
Co-authored-by: Ben Rowland <[email protected]>
Co-authored-by: Henry Schreiner <[email protected]>

Author: 3 people

.github/workflows/ci.yml

@@ -81,7 +81,9 @@ jobs:
           python-version: ${{ matrix.python-version }}
 
       - name: Install package
-        run: pip install .[test,cov] cmake ninja rich
+        run:
+          pip install .[test,cov] cmake ninja rich hatch-fancy-pypi-readme
+          setuptools-scm
 
       - name: Test package
         run: pytest -ra --showlocals --cov=scikit_build_core
@@ -91,9 +93,11 @@ jobs:
         with:
           name: ${{ runner.os }}-${{ matrix.python-version }}
 
+      # the min requirements are not compatible with the metadata plugin
+      # packages so we remove those first (they then won't be tested)
       - name: Min requirements
         run: |
-          pip uninstall -y cmake
+          pip uninstall -y cmake hatch-fancy-pypi-readme setuptools-scm
           pip install --constraint tests/constraints.txt .[test]
 
       - name: Setup CMake ${{ matrix.cmake-version }}

.pre-commit-config.yaml

@@ -63,21 +63,29 @@ repos:
     rev: v1.1.1
     hooks:
       - id: mypy
-        exclude: tests/packages/(simplest_c/src/simplest/__init__.py|.*/setup.py)
+        exclude: |
+          (?x)^(
+            tests/packages/simplest_c/src/simplest/__init__.py|
+            tests/packages/dynamic_metadata/src/dynamic/__init__.py|
+            tests/packages/.*/setup.py
+          )
         files: ^(src|tests)
         args: []
         additional_dependencies:
           - build
           - cattrs
           - cmake
           - exceptiongroup
-          - importlib_metadata
+          - gitpython
+          - hatch-fancy-pypi-readme
+          - importlib-metadata
           - importlib_resources
           - ninja
           - packaging
           - pyproject_metadata
           - pytest
           - rich
+          - setuptools-scm
           - tomli
           - types-setuptools
           - typing_extensions >=4; python_version<'3.11'

noxfile.py

@@ -27,7 +27,9 @@ def pylint(session: nox.Session) -> None:
     """
     # This needs to be installed into the package environment, and is slower
     # than a pre-commit check
-    session.install("-e.[dev,test]", "pylint")
+    session.install(
+        "-e.[dev,test]", "pylint", "hatch-fancy-pypi-readme", "setuptools-scm"
+    )
     session.run("pylint", "scikit_build_core", *session.posargs)
 
 
@@ -37,7 +39,7 @@ def tests(session: nox.Session) -> None:
     Run the unit and regular tests.
     """
     env = {"PIP_DISABLE_PIP_VERSION_CHECK": "1"}
-    extra = ["rich"]
+    extra = ["hatch-fancy-pypi-readme", "rich", "setuptools-scm"]
     # This will not work if system CMake is too old (<3.15)
     if shutil.which("cmake") is None and shutil.which("cmake3") is None:
         extra.append("cmake")

pyproject.toml

@@ -53,6 +53,7 @@ rich = [
 test = [
     "build[virtualenv]",
     "cattrs >=22.2.0",
+    "gitpython",
     "importlib_metadata; python_version<'3.8'",
     "pathspec >=0.10.1",
     "pybind11",
@@ -129,7 +130,7 @@ module = ["scikit_build_core.*"]
 disallow_untyped_defs = true
 
 [[tool.mypy.overrides]]
-module = ["pathspec"]
+module = ["pathspec", "setuptools_scm", "hatch_fancy_pypi_readme"]
 ignore_missing_imports = true
 
 
@@ -197,6 +198,7 @@ extend-ignore = [
     "PLR2004",
     "PLE1205",  # Format check doesn't work with our custom logger
     "E501",  # Line too long
+    "PT004",
 ]
 target-version = "py37"
 typing-modules = ["scikit_build_core._compat.typing"]

src/scikit_build_core/build/sdist.py

@@ -8,9 +8,8 @@
 import tarfile
 from pathlib import Path
 
-from pyproject_metadata import StandardMetadata
-
 from .._compat import tomllib
+from ..settings.metadata import get_standard_metadata
 from ..settings.skbuild_read_settings import SettingsReader
 from ._file_processor import each_unignored_file
 from ._init import setup_logging
@@ -69,23 +68,23 @@ def build_sdist(
     sdist_directory: str,
     config_settings: dict[str, list[str] | str] | None = None,
 ) -> str:
-    settings_reader = SettingsReader.from_file("pyproject.toml", config_settings)
+    with Path("pyproject.toml").open("rb") as f:
+        pyproject = tomllib.load(f)
+
+    settings_reader = SettingsReader(pyproject, config_settings or {})
     settings = settings_reader.settings
     setup_logging(settings.logging.level)
 
     settings_reader.validate_may_exit()
 
     sdist_dir = Path(sdist_directory)
 
-    with Path("pyproject.toml").open("rb") as f:
-        pyproject = tomllib.load(f)
-
     reproducible = settings.sdist.reproducible
     timestamp = get_reproducible_epoch() if reproducible else None
 
+    metadata = get_standard_metadata(pyproject, settings)
     # Using deepcopy here because of a bug in pyproject-metadata
     # https://github.com/FFY00/python-pyproject-metadata/pull/49
-    metadata = StandardMetadata.from_pyproject(pyproject)
     pkg_info = bytes(copy.deepcopy(metadata).as_rfc822())
 
     srcdirname = f"{metadata.name}-{metadata.version}"

src/scikit_build_core/build/wheel.py

@@ -9,14 +9,14 @@
 from pathlib import Path
 
 from packaging.version import Version
-from pyproject_metadata import StandardMetadata
 
 from .. import __version__
 from .._compat import tomllib
 from .._logging import logger, rich_print
 from ..builder.builder import Builder, archs_to_tags, get_archs
 from ..builder.wheel_tag import WheelTag
 from ..cmake import CMake, CMaker
+from ..settings.metadata import get_standard_metadata
 from ..settings.skbuild_read_settings import SettingsReader
 from ._init import setup_logging
 from ._pathutil import packages_to_file_mapping
@@ -63,16 +63,17 @@ def _build_wheel_impl(
     """
     Build a wheel or just prepare metadata (if wheel dir is None).
     """
+    pyproject_path = Path("pyproject.toml")
+    with pyproject_path.open("rb") as ft:
+        pyproject = tomllib.load(ft)
 
-    settings_reader = SettingsReader.from_file("pyproject.toml", config_settings)
+    settings_reader = SettingsReader(pyproject, config_settings or {})
     settings = settings_reader.settings
     setup_logging(settings.logging.level)
 
     settings_reader.validate_may_exit()
 
-    with Path("pyproject.toml").open("rb") as ft:
-        pyproject = tomllib.load(ft)
-    metadata = StandardMetadata.from_pyproject(pyproject)
+    metadata = get_standard_metadata(pyproject, settings, config_settings)
 
     if metadata.version is None:
         msg = "project.version is not statically specified, must be present currently"

src/scikit_build_core/metadata/__init__.py

@@ -0,0 +1,3 @@
+from __future__ import annotations
+
+__all__: list[str] = []

src/scikit_build_core/metadata/fancy_pypi_readme.py

@@ -0,0 +1,28 @@
+from __future__ import annotations
+
+from typing import Any
+
+__all__ = ["dynamic_metadata"]
+
+
+def __dir__() -> list[str]:
+    return __all__
+
+
+def dynamic_metadata(
+    pyproject_dict: dict[str, Any],
+    _config_settings: dict[str, list[str] | str] | None = None,
+) -> dict[str, str | dict[str, str | None]]:
+    from hatch_fancy_pypi_readme._builder import build_text
+    from hatch_fancy_pypi_readme._config import load_and_validate_config
+
+    config = load_and_validate_config(
+        pyproject_dict["tool"]["hatch"]["metadata"]["hooks"]["fancy-pypi-readme"]
+    )
+
+    return {
+        "readme": {
+            "content-type": config.content_type,
+            "text": build_text(config.fragments, config.substitutions),
+        }
+    }

src/scikit_build_core/metadata/setuptools_scm.py

@@ -0,0 +1,21 @@
+from __future__ import annotations
+
+__all__ = ["dynamic_metadata"]
+
+
+def __dir__() -> list[str]:
+    return __all__
+
+
+def dynamic_metadata(
+    pyproject_dict: dict[str, object],  # noqa: ARG001
+    _config_settings: dict[str, list[str] | str] | None = None,
+) -> dict[str, str | dict[str, str | None]]:
+    # this is a classic implementation, waiting for the release of
+    # vcs-versioning and an improved public interface
+    from setuptools_scm import Configuration, _get_version
+
+    config = Configuration.from_file("pyproject.toml")
+    version: str = _get_version(config)
+
+    return {"version": version}

src/scikit_build_core/settings/metadata.py

@@ -0,0 +1,50 @@
+from __future__ import annotations
+
+import importlib
+from typing import Any
+
+from pyproject_metadata import StandardMetadata
+
+from ..settings.skbuild_model import ScikitBuildSettings
+
+__all__ = ["get_standard_metadata"]
+
+
+def __dir__() -> list[str]:
+    return __all__
+
+
+def _load(
+    mod_name: str,
+    pyproject_dict: dict[str, Any],
+    config_settings: dict[str, list[str] | str] | None = None,
+) -> dict[str, Any]:
+    return importlib.import_module(mod_name).dynamic_metadata(pyproject_dict, config_settings)  # type: ignore[no-any-return]
+
+
+# If pyproject-metadata eventually supports updates, this can be simplified
+def get_standard_metadata(
+    pyproject_dict: dict[str, Any],
+    settings: ScikitBuildSettings,
+    config_settings: dict[str, list[str] | str] | None = None,
+) -> StandardMetadata:
+    # Handle any dynamic metadata
+    for field in settings.metadata:
+        if field not in pyproject_dict.get("project", {}).get("dynamic", []):
+            msg = f"{field} is not in project.dynamic"
+            raise KeyError(msg)
+
+    plugins = set(settings.metadata.values())
+    cached_plugins = {
+        key: _load(key, pyproject_dict, config_settings) for key in plugins
+    }
+
+    for field, mod_name in settings.metadata.items():
+        if field not in cached_plugins[mod_name]:
+            msg = f"{field} is not provided by plugin {mod_name}"
+            raise KeyError(msg)
+
+        pyproject_dict["project"][field] = cached_plugins[mod_name][field]
+        pyproject_dict["project"]["dynamic"].remove(field)
+
+    return StandardMetadata.from_pyproject(pyproject_dict)

src/scikit_build_core/settings/skbuild_model.py

@@ -110,6 +110,7 @@ class ScikitBuildSettings:
     sdist: SDistSettings = dataclasses.field(default_factory=SDistSettings)
     wheel: WheelSettings = dataclasses.field(default_factory=WheelSettings)
     backport: BackportSettings = dataclasses.field(default_factory=BackportSettings)
+    metadata: Dict[str, str] = dataclasses.field(default_factory=dict)
 
     #: Strictly check all config options. If False, warnings will be
     #: printed for unknown options. If True, an error will be raised.

tests/constraints.txt

@@ -3,6 +3,7 @@ cattrs==22.2.0
 pytest==7.0.0
 tomli==1.1.0
 packaging==20.9
+importlib-metadata==4.13.0
 importlib-resources==1.3.0
-pyproject-metadata==0.5.0
+pyproject-metadata==0.6.1
 pathspec==0.10.1

tests/packages/dynamic_metadata/CMakeLists.txt

@@ -0,0 +1,13 @@
+cmake_minimum_required(VERSION 3.15...3.25)
+
+project(
+  ${SKBUILD_PROJECT_NAME}
+  LANGUAGES C
+  VERSION ${SKBUILD_PROJECT_VERSION})
+
+find_package(Python COMPONENTS Interpreter Development.Module)
+set(Python_SOABI ${SKBUILD_SOABI})
+
+python_add_library(_module MODULE src/module.c WITH_SOABI)
+
+install(TARGETS _module DESTINATION ${SKBUILD_PROJECT_NAME})

tests/packages/dynamic_metadata/dual_project.toml

@@ -0,0 +1,11 @@
+[build-system]
+requires = ["scikit-build-core"]
+build-backend = "scikit_build_core.build"
+
+[project]
+name = "fancy"
+dynamic = ["version", "license"]
+
+[tool.scikit-build.metadata]
+version = "test_dual"
+license = "test_dual"

tests/packages/dynamic_metadata/faulty_dual_project.toml

@@ -0,0 +1,12 @@
+[build-system]
+requires = ["scikit-build-core"]
+build-backend = "scikit_build_core.build"
+
+[project]
+name = "fancy"
+dynamic = ["version", "readme", "license"]
+
+[tool.scikit-build.metadata]
+version = "test_dual"
+license = "test_dual"
+readme = "test_dual"

tests/packages/dynamic_metadata/faulty_project.toml

@@ -0,0 +1,10 @@
+[build-system]
+requires = ["scikit-build-core"]
+build-backend = "scikit_build_core.build"
+
+[project]
+name = "fancy"
+version = "0.0.1"
+
+[tool.scikit-build.metadata]
+readme = "scikit_build_core.metadata.fancy_pypi_readme"

tests/packages/dynamic_metadata/plugin_project.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["scikit-build-core"]
+build-backend = "scikit_build_core.build"
+
+[project]
+name = "fancy"
+dynamic = ["readme", "version"]
+
+[tool.scikit-build.metadata]
+version = "scikit_build_core.metadata.setuptools_scm"
+readme = "scikit_build_core.metadata.fancy_pypi_readme"
+
+[tool.setuptools_scm]
+
+[tool.hatch.metadata.hooks.fancy-pypi-readme]
+content-type = "text/x-rst"
+
+[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]]
+text = "Fragment #1"
+
+[[tool.hatch.metadata.hooks.fancy-pypi-readme.fragments]]
+text = "Fragment #2"

tests/packages/dynamic_metadata/pyproject.toml

@@ -0,0 +1,12 @@
+[build-system]
+requires = ["scikit-build-core"]
+build-backend = "scikit_build_core.build"
+
+[project]
+name = "dynamic"
+dynamic = ["version", "readme", "license"]
+
+[tool.scikit-build.metadata]
+version = "test_version"
+readme = "test_readme"
+license = "test_license"

tests/packages/dynamic_metadata/src/dynamic/__init__.py

@@ -0,0 +1,3 @@
+from ._module import square
+
+__all__ = ["square"]