Skip to content

wip: allow toolchains without a host-compatible variant #2891

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .python-version
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.13.3
3.12
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,10 @@ END_UNRELEASED_TEMPLATE
Set the `RULES_PYTHON_ENABLE_PIPSTAR=1` environment variable to enable it.
* (utils) Add a way to run a REPL for any `rules_python` target that returns
a `PyInfo` provider.
* (toolchains) Arbitrary python-build-standalone runtimes can be registered
and activated with custom flags. See the [Registering custom runtimes]
docs and {obj}`single_version_platform_override()` API docs for more
information.

{#v0-0-0-removed}
### Removed
Expand Down
17 changes: 17 additions & 0 deletions MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,23 @@ dev_python = use_extension(
dev_python.override(
register_all_versions = True,
)
dev_python.toolchain(python_version = "3.13")

# For testing an arbitrary runtime triggered by a custom flag.
# See //tests/toolchains:custom_platform_toolchain_test
dev_python.single_version_platform_override(
platform = "linux-x86-install-only-stripped",
python_version = "3.13.3",
sha256 = "01d08b9bc8a96698b9d64c2fc26da4ecc4fa9e708ce0a34fb88f11ab7e552cbd",
#target_compatible_with = [
# "@platforms//os:linux",
# "@platforms//cpu:x86_64",
#],
#target_settings = [
# "@@//tests/support:is_custom_runtime_linux-x86-install-only-stripped",
#],
urls = ["https://github.com/astral-sh/python-build-standalone/releases/download/20250409/cpython-3.13.3+20250409-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz"],
)

dev_pip = use_extension(
"//python/extensions:pip.bzl",
Expand Down
70 changes: 70 additions & 0 deletions docs/toolchains.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,76 @@ existing attributes:
* Adding additional Python versions via {bzl:obj}`python.single_version_override` or
{bzl:obj}`python.single_version_platform_override`.

### Registering custom runtimes

Because the python-build-standalone project has _thousands_ of prebuilt runtimes
available, rules_python only includes popular runtimes in its built in
configurations. If you want to use a runtime that isn't already known to
rules_python then {obj}`single_version_platform_override()` can be used to do
so. In short, it allows specifying an arbitrary URL and using custom flags
to control when a runtime is used.

In the example below, we register a particular python-build-standalone runtime
that is activated for Linux x86 builds when the custom flag
`--//:runtime=my-custom-runtime` is set.

```
# File: MODULE.bazel
bazel_dep(name = "bazel_skylib", version = "1.7.1.")
bazel_dep(name = "rules_python", version = "1.5.0")
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(python_version="3.13.3")
python.single_version_platform_override(
platform = "my-platform",
python_version = "3.13.3",
sha256 = "01d08b9bc8a96698b9d64c2fc26da4ecc4fa9e708ce0a34fb88f11ab7e552cbd",
target_compatible_with = [
"@platforms//os:linux",
"@platforms//cpu:x86_64",
],
target_settings = [
"@@//:runtime=my-custom-runtime",
],
urls = ["https://github.com/astral-sh/python-build-standalone/releases/download/20250409/cpython-3.13.3+20250409-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz"],
)
# File: //:BUILD.bazel
load("@bazel_skylib//rules:common_settings.bzl", "string_flag")
string_flag(
name = "custom_runtime",
build_setting_default = "",
)
config_setting(
name = "is_custom_runtime_linux-x86-install-only-stripped",
flag_values = {
":custom_runtime": "linux-x86-install-only-stripped",
},
)
```

Notes:
- While any URL and archive can be used, it's assumed their content looks how
a python-build-standalone archive looks.
- `python.toolchain()` is required if the version is unknown; if the version
is already known, it can be omitted.
- A "version aware" toolchain is registered, which means the Python version flag
must also match (e.g. `--@rules_python//python/config_settings:python_version=3.13.3`
must be set -- see `minor_mapping` and `is_default` for controls and docs
about version matching and selection).
- The labels in `target_settings` must be absolute; `@@` refers to the main repo.
- The `target_settings` are `config_setting` targets, which means you can
customize how matching occurs.

:::{seealso}
See {obj}`//python/config_settings` for flags rules_python already defines
that can be used with `target_settings`. Some particular ones of note are:
{flag}`--py_linux_libc` and {flag}`--py_freethreaded`, among others.
:::

:::{versionadded} VERSION_NEXT_FEATURE
Added support for custom platform names, `target_compatible_with`, and
`target_settings` with `single_version_platform_override`.
:::

### Using defined toolchains from WORKSPACE

It is possible to use toolchains defined in `MODULE.bazel` in `WORKSPACE`. For example
Expand Down
6 changes: 6 additions & 0 deletions python/private/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -241,11 +241,17 @@ bzl_library(
],
)

bzl_library(
name = "platform_info_bzl",
srcs = ["platform_info.bzl"],
)

bzl_library(
name = "python_bzl",
srcs = ["python.bzl"],
deps = [
":full_version_bzl",
":platform_info_bzl",
":python_register_toolchains_bzl",
":pythons_hub_bzl",
":repo_utils_bzl",
Expand Down
34 changes: 34 additions & 0 deletions python/private/platform_info.bzl
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
"""Helper to define a struct used to define platform metadata."""

def platform_info(
*,
compatible_with = [],
flag_values = {},
target_settings = [],
os_name,
arch):
"""Creates a struct of platform metadata.

This is just a helper to ensure structs are created the same and
the meaning/values are documented.

Args:
compatible_with: list[str], where the values are string labels. These
are the target_compatible_with values to use with the toolchain
flag_values: dict[str|Label, Any] of config_setting.flag_values
compatible values. DEPRECATED -- use target_settings instead
target_settings: list[str], where the values are string labels. These
are the target_settings values to use with the toolchain.
os_name: str, the os name; must match the name used in `@platfroms//os`
arch: str, the cpu name; must match the name used in `@platforms//cpu`

Returns:
A struct with attributes and values matching the args.
"""
return struct(
compatible_with = compatible_with,
flag_values = flag_values,
target_settings = target_settings,
os_name = os_name,
arch = arch,
)
Loading