[Bazel CI] Bazel Build Error: no such target com_google_protobuf_protobuf_java from rules_jvm_external

[satyanandak]

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4898#0198ef9c-9e4d-4566-8aa3-e3566630269a

Platform: MacOS, Ubuntu

Logs:

ERROR: �[0m/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/5f3a3e2fde2994b25bbcbe370cb970c4/external/rules_jvm_external++maven+maven/BUILD: no such target '@@rules_jvm_external++maven+maven//:com_google_protobuf_protobuf_java_util': target 'com_google_protobuf_protobuf_java_util' not declared in package '' defined by /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/5f3a3e2fde2994b25bbcbe370cb970c4/external/rules_jvm_external++maven+maven/BUILD
ERROR: �[0m/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/5f3a3e2fde2994b25bbcbe370cb970c4/external/bazel_worker_java+/src/main/java/com/google/devtools/build/lib/worker/BUILD.bazel:5:13: no such target '@@rules_jvm_external++maven+maven//:com_google_protobuf_protobuf_java_util': target 'com_google_protobuf_protobuf_java_util' not declared in package '' defined by /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/5f3a3e2fde2994b25bbcbe370cb970c4/external/rules_jvm_external++maven+maven/BUILD and referenced by '@@bazel_worker_java+//src/main/java/com/google/devtools/build/lib/worker:work_request_handlers'
ERROR: �[0mAnalysis of target '//test/rules/android_local_test/java/com/starlark_resources:single_resource_dep_without_manifest' failed; build aborted: Analysis failed
ERROR: �[0mBuild did NOT complete successfully
FAILED:�[0m 
FAILED:�[0m 
ERROR: �[0mNo test targets were found, yet testing was requested
FAILED:�[0m 

Culprit:

Steps:

git clone https://github.com/bazelbuild/rules_android
git reset b33b83d2f2ff1bc24ee578f6963ef44cee00ce1d --hard
export USE_BAZEL_VERSION=2c28cbf770866c2747237ffba77fd6e47671544a
bazel build -- //android/... //src/... -//src/java/com/example/sampleapp/... //test/... -//test/rules/... //toolchains/... //tools/... -//tools/android/...

CC Greenteam @meteorcloudy

[meteorcloudy]

@ahumesky @ted-xie Can you please take a look?

[agluszak]

Reproduced it :( https://buildkite.com/bazel/rules-kotlin-kotlin/builds/5521/steps/canvas?sid=019abc75-d465-48cb-9b71-ed64b2d5f8a1

[DoDoENT]

For us it happens whenever someone tries to update rules_java from v8.14.0 to the latest version (v9.2.0 at the moment of writing).

See the relevant slack thread

[DoDoENT]

It also happened to me by adding the bazel_dep(name = "protobuf", version = "33.1") dependency, without touching the rules_java...

[DoDoENT]

Pinning to protobuf 29.1 and rules_java 8.14. avoids having this issue.

[DoDoENT]

Apparently, it still doesn't reproduce with rules_java 8.15.2 and protobuf 31.1.

Updating either one of these (rules_java already to 8.16.0 or protobuf to 32.0) causes this issue.

[ted-xie]

I think this is some kind of bug with rules_jvm_external's maven namespacing. I can repro it and trivially fix this by either adjusting all rules_android rules_android_maven references to just "maven", or the opposite direction: transforming all of bazel-worker-api's maven references to bazel_worker_maven. I actually raised this issue to bazel-worker-api back in August this year bazelbuild/bazel-worker-api#11.

I'll let @Wyverald decide what to do with my proposal there and the PR. For rules_android, I'm okay with renaming all of our internal maven repos to just "maven", though I don't really like it. I kind of like separating tool Maven deps from app deps, and merging app and tool deps into one namespace may force our users to upgrade their deps in a non-obvious way.

[DoDoENT]

I think this is some kind of bug with rules_jvm_external's maven namespacing.

Is there a tracking issue in the rules_jvm_external repo?

For rules_android, I'm okay with renaming all of our internal maven repos to just "maven", though I don't really like it.

I agree with you here. I also prefer having separate app deps and maven deps.

[shs96c]

I'm okay with renaming all of our internal maven repos to just "maven", though I don't really like it.

Please don't do this. In rules_jvm_external I already have people complaining about the modules that do, and it's making me reconsider the idea of dependency layering --- people's builds are getting weird dependencies.

I can't see an issue for this in the rules_jvm_external repo, but if I'm mistaken, please LMK

[DoDoENT]

So, what is the correct solution for this? The mentioned change in Bazel's worker API?

[ted-xie]

@shs96c To clarify, would you prefer that all Bazel repos use their own @{module_name}_maven} repos, instead of overloading @maven? I can see pros/cons for either approach.

RE filing an issue for rules_jvm_external: I'm not 100% sure it's an issue with RJE yet, or some other quirk of either Bazel or rules_android.

[shs96c]

@ted-xie, we document this but the relevant bit is:

Modules that are expected to be included via a bazel_dep should avoid using the default name, and should always set their own (eg. rules_jvm_external uses rules_jvm_external_deps for its own dependencies)

The rule of thumb is that if your dep is going to contribute to the production code of the root repo, then you might want to consider adding to the default namespace, but only if other options (such as asking projects to add a specific maven coordinate to their dependencies themselves) have been exhausted. Otherwise, a non-root module contributing to the root module's dependencies is a no-no.