Regression with @@rules_java//toolchains:bootstrap_runtime_toolchain_type in 7.10.0

[protocol7]

When updating to 7.10.0 (with Bazel 7.3.1), we are seeing a regression:

19:03:56.106  ERROR: /var/jenkins_home/workspace/tingle.7db35ddc-365e-4598-8eef-791dd4df038c/.cache/bazel/_bazel_root/46ed70dabc45918b7e6c57b0e0bab0ab/external/rules_java/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java//toolchains:platformclasspath (ef69ab5): No matching toolchains found for types @@rules_java//toolchains:bootstrap_runtime_toolchain_type.
19:03:56.106  To debug, rerun with --toolchain_resolution_debug='@@rules_java//toolchains:bootstrap_runtime_toolchain_type'
19:03:56.106  If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro.

Running with --toolchain_resolution_debug='@@rules_java//toolchains:bootstrap_runtime_toolchain_type':

bazel build //... --toolchain_resolution_debug='@@rules_java//toolchains:bootstrap_runtime_toolchain_type'
> bazel build //... --toolchain_resolution_debug=@@rules_java//toolchains:bootstrap_runtime_toolchain_type
INFO: Invocation ID: 6788bd1e-2e53-4930-b94c-bf3302a07383
INFO: Streaming build results to: https://spotify.buildbuddy.io/invocation/6788bd1e-2e53-4930-b94c-bf3302a07383
WARNING: Build option --toolchain_resolution_debug has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: ToolchainResolution: Performing resolution of @@rules_java//toolchains:bootstrap_runtime_toolchain_type for target platform @@internal_platforms_do_not_use//host:host
      ToolchainResolution:   Rejected toolchain @@local_jdk//:jdk; mismatching config settings: local_jdk_name_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_windows//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_win//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux_ppc64le//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_win_arm64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_win_arm64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_win//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux_ppc64le//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_win//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux_ppc64le//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_win_arm64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@local_jdk//:jdk; mismatching config settings: local_jdk_name_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_windows//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_win//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux_ppc64le//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_win_arm64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_win_arm64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_win//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux_ppc64le//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_win//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux_ppc64le//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_win_arm64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@local_jdk//:jdk; mismatching config settings: local_jdk_name_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_windows//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remote_jdk8_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_win//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux_ppc64le//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk11_win_arm64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_win_arm64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_win//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux_ppc64le//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk17_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_macos_aarch64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_macos//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_win//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux_ppc64le//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_linux_s390x//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution:   Rejected toolchain @@remotejdk21_win_arm64//:jdk; mismatching config settings: prefix_version_setting
      ToolchainResolution: No @@rules_java//toolchains:bootstrap_runtime_toolchain_type toolchain found for target platform @@internal_platforms_do_not_use//host:host.
ERROR: /private/var/tmp/_bazel_ngn/ec336af31efb37b2c377816813f0b83a/external/rules_java/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java//toolchains:platformclasspath (12bb0e1): No matching toolchains found for types @@rules_java//toolchains:bootstrap_runtime_toolchain_type.
To debug, rerun with --toolchain_resolution_debug='@@rules_java//toolchains:bootstrap_runtime_toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro.
...

[fmeum]

@comius

[comius]

Sorry for my ignorance.

That was caused by moving bootstrap_runtime_toolchain_type into rules_java: #210

It was necessary to make rules_java compatible with Bazel 6, which doesn't have this toolchain type defined in @bazel_tools. If we don't have a version of rules_java compatible with Bazel 6, it's quite hard to migrate repos to use newly exposed bzl files.

I see there are 2 public uses of the toolchain, rules_graalvm, rules_nixpkg (https://github.com/search?q=bootstrap_runtime_toolchain_type+%28path%3A*.BUILD+OR+path%3A*.bzl%29&type=Code&ref=advsearch&l=&l=).

Options seem to be:

• migrate users to the new location, @rules_java//toolchains:bootstrap_runtime_toolchain_type (without additional bzl symbol)
• expose toolchain type label through a .bzl files symbol, so that a change in the future doesn't break stuff (and migrate the users) - but we also don't intent to make future changes
• implement a version check, that puts the toolchain type into rules_java for Bazel 6, Bazel 8, but not Bazel 7 (this feels kind of awkward)
• backport everyting in rules_java 7.x.x to rules_java 6.x.x and keep using 2 rules_java versions

[protocol7]

For what it's worth, we do not reference @rules_java//toolchains:bootstrap_runtime_toolchain_type in our repo nor do we use rules_graalvm or rules_nixpkg. If there is anything else I can supply from our setup, let me know.

[comius]

@protocol7 thanks for the info. Do you have a way to test if #215 fixes it?

[protocol7]

@comius I did some arguably ugly local patching and that did not resolve the issue.

[comius]

@comius I did some arguably ugly local patching and that did not resolve the issue.

Can you provide a reproducer?

[protocol7]

Attaching a reproduction workspace, this is a minimisation of the actual repo in which we're seeing these issues. Note in particular .bazelrc as both of the settings there are required to reproduce. I suspect that the --repo_env workaround will stand out.

214.tar.gz

[comius]

Argh, I see what's going on.

You're missing the last line in:

load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains")
rules_java_dependencies()
rules_java_toolchains()

Without the last line, it used to work with rules_java 7.9, because bazel calls it for you from the embedded rules_java_prebuilt. However the embedded repository won't register toolchains for the new toolchain type.

[protocol7]

Thanks! That did indeed fix the issue in the repro example, however turns out I minimised it a bit too much. In our real repo, we also configure an extra toolchain. With this, we still get the same error, are we missing some further initialisation that were not previously required?
214_2.tar.gz

[honnix]

Could this because of the conflict between rules_java_builtin included from Bazel itself and rules_java. These two have different versions. More specifically, remote_java_repository from rules_java_builtin still has:

 toolchain(
     name = "bootstrap_runtime_toolchain",
     # These constraints are not required for correctness, but prevent fetches of remote JDK for
     # different architectures. As every Java compilation toolchain depends on a bootstrap runtime in
     # the same configuration, this constraint will not result in toolchain resolution failures.
     exec_compatible_with = {target_compatible_with},
     target_settings = [":version_or_prefix_version_setting"],
     toolchain_type = "@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type",
     toolchain = "{toolchain}",
 )

$ diff -u rules_java/toolchains/remote_java_repository.bzl rules_java_builtin/toolchains/remote_java_repository.bzl
--- rules_java/toolchains/remote_java_repository.bzl	2024-09-05 11:49:11
+++ rules_java_builtin/toolchains/remote_java_repository.bzl	2034-09-03 10:47:26
@@ -89,7 +89,7 @@
     # the same configuration, this constraint will not result in toolchain resolution failures.
     exec_compatible_with = {target_compatible_with},
     target_settings = [":version_or_prefix_version_setting"],
-    toolchain_type = "@rules_java//toolchains:bootstrap_runtime_toolchain_type",
+    toolchain_type = "@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type",
     toolchain = "{toolchain}",
 )
 """.format(

When using remote_java_repository from @bazel_tools, I think it is the remote_java_repository from rules_java_builtin gets called, which means we are generating old toolchain that can no longer work with rules_java 7.10.

[comius]

repro2 already fails on rules_java 7.9.1 - and I don't know precisely why. If there's a repro that works on 7.9.1 and fails on 7.10. I'll inspect it more closely.

Generally yes, rules from rules_java need to be used and not from @bazel_tools, so that consistent toolchains are used. (I don't see this is the problem with repro2)

[honnix]

Generally yes, rules from rules_java need to be used and not from @bazel_tools

Yeah this makes sense. While we can ensure in our repo we don't use rules_java_builtin through @bazel_tools, it is possible that another 3rd party rule set uses, which will likely cause inconsistency problem. Is there any plan to fix this kind of issue?

One thing to note is that, we use remote_java_repository from @bazel/tools because it also does native.register_toolchains("@" + name + "_toolchain_config_repo//:all"), which is pretty handy by hiding implementation details.

[fmeum]

The long-term way this will be fixed is by fully migrating to Bzlmod. The built-in rules_java repo is a hack and it's very hard to make changes to its structure.

[honnix]

Is it fair to ask for a patch release to get #215 (it fixes local JDK toolchain), and upgrade rules_java_builtin in bazel to the same version afterwards?

It might also be worth having something in the release notes mentioning this potential conflict when using remote_java_repository from @bazel/tools while having 7.10+ rules_java loaded.