Bump protoc-bridge to 0.9.8 for Scala >= 2.12
#1688
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This required updates to `scripts/create_repository.py` to accommodate
comparisons between `_2.13` and `_3` versions of ScalaPB artifacts. The
`_3` versions of the most recent ScalaPB artifacts are only available
for Scala 3.3 and later, and `compilerplugin_3` currently still depends
on `protoc-gen_2.13`.
The `MavenCoordinates` dataclass in `scripts/create_repository.py` has
new `unversioned_artifact`, `scala_version`, and `artifact_name`
members. Updates to `MavenCoordinates.is_newer_than()` now compare
objects with matching `artifact_name` values, and take the
`scala_version` into account. These changes then precipitated:
- Replacing the `artifact_name()` call with `artifact_name` accesses.
- Replacing `ArtifactLabelMaker._remove_scala_version_suffix` with
`MavenCoordinates.unversioned_artifact`.
- Extracting `__compare_versions` from `is_newer_than` to reuse the same
code for comparing Scala versions and artifact versions. (Properly
reversed the use of `lhs` and `rhs` in the process.)
This change also removes an unnecessary `try`/`catch` block from
`scripts.ScalaPbCodeGenerator.process()` for Scala 2.12 and above.
The block is still required for the Scala 2.11 implementation of
`scripts.ScalaPbCodeGenerator.run()` and in
`scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run()`.
Finally, this change adds notes in a few places to indicate where
support remains for Scala 2.11. This supporting code can ultimately be
removed if we ever decide to drop Scala 2.11 support.
---
Tested by setting the `protobuf` dependencies to a versions I know will
break `ScalaPB` 0.11.17:
- `abseil-cpp`: 20220623.1 => 20240722.0
- `protobuf`: v21.7 => v26.1
I temporarily removed the `try`/`catch` block for `Throwable` from
`scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`.
I then ran `bazel shutdown` to stop persistent `ProtoScalaPBRule`
workers.
After that, building with Bazel 7.4.1 (since Bazel 6.5.0 can't build
`abseil-cpp` 20240722.0 without setting C++14 compiler flags) crashed as
expected, instead of hanging. Undoing the `abseil-cpp` and `protobuf`
changes and building then produced a working build.
```txt
$ USE_BAZEL_VERSION=7.4.1 bazel build //third_party/test/proto:scala
[ ...snip... ]
ERROR: third_party/test/proto/BUILD.bazel:4:14:
ProtoScalaPBRule
third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar
failed: (Exit 1):
scalapb_worker failed: error executing ProtoScalaPBRule command
(from target //third_party/test/proto:proto)
bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker
... (remaining 2 arguments skipped)
--jvm_extra_protobuf_generator_out: java.lang.VerifyError: Cannot inherit from final class
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:711)
at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:706)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:719)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
at scalapb.options.Scalapb.<clinit>(Scalapb.java:24835)
at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8)
at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:52)
at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48)
at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113)
at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$2(SocketBasedPluginFrontend.scala:31)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75)
at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118)
at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87)
at scala.concurrent.package$.blocking(package.scala:146)
at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$1(SocketBasedPluginFrontend.scala:23)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
java.lang.RuntimeException: Exit with code 1
at scala.sys.package$.error(package.scala:30)
at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44)
at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96)
at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49)
at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39)
at scripts.ScalaPBWorker.main(ScalaPBWorker.scala)
Target //third_party/test/proto:scala failed to build
ERROR: third_party/test/proto/BUILD.bazel:4:14
scala @//third_party/test/proto:proto failed:
(Exit 1): scalapb_worker failed:
error executing ProtoScalaPBRule command
(from target //third_party/test/proto:proto)
bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker
... (remaining 2 arguments skipped)
```
---
The initial motivation for this change was to try to eliminate the need
for the custom `scripts.ScalaPbCodeGenerator` implementation by
upgrading to `protoc-bridge` 0.9.8. This version contains a change to
`catch` any `Throwable` objects raised by a generator implementation in
`protocbridge.frontend.PluginFrontend.runWithInputStream()`.
- https://github.com/scalapb/protoc-bridge/blob/c574d50eaee5b800fd54493fe25c4e0eed3b9def/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122
This change originally came from:
- https://github.com/scalapb/protoc-bridge/blob/d0d56f635d13f7efaa2755ed0d2d66bdef18b588/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122
- scalapb/protoc-bridge#367
I closed scalapb/ScalaPB#1771 as a result, and reexamined the
`try`/`catch` blocks added in bazel-contrib#1630 and bazel-contrib#1637, and the
`scripts.ScalaPBCodeGenerator` implementations added in bazel-contrib#1648.
However, the last `protoc-bridge` version to support Scala 2.11 is
0.7.14. Scala 2.11 won't be able to use `protoc-bridge` 0.9.8, so we
still need to `catch` any `Throwable`s ourselves. This includes the
`catch` blocks from the Scala 2.11 `scripts.ScalaPbCodeGenerator` and
`scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`.
---
In `protoc-bridge` 0.9.7 and 0.9.8, `protocbridge.ProtocCodeGenerator`
declares a `run()` method with no implementation (and no `try`/`catch`
block).
- https://github.com/scalapb/protoc-bridge/blob/v0.9.7/bridge/src/main/scala/protocbridge/ProtocCodeGenerator.scala#L5
In `protoc-gen` 0.9.7 and 0.9.8, `protogen.CodeGenApp` implements
`protocbridge.ProtocCodeGenerator`, and `CodeGenApp.run()` wraps a call
to `CodeGenApp.process()` inside a `try`/`catch` block.
- https://github.com/scalapb/protoc-bridge/blob/v0.9.7/protoc-gen/src/main/scala/protocgen/CodeGenApp.scala#L41-L56
`scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 extends
`CodeGenApp` and implements `process()`. For this version of
`ScalaPbCodeGenerator`, available in Scala 2.12 and later, there's no
need for a `try`/`catch` wrapper on our end.
- https://github.com/scalapb/ScalaPB/blob/v0.11.17/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L11
However, the last available `compilerplugin` version for Scala 2.11 is
0.9.8. `scalapb.ScalaPbCodeGenerator` from that version implements
`protocbridge.ProtocCodeGenerator` and has a `try`/`catch` in its
`run()` method. However, the `Scalapb.registerAllExtensions(registry)`
call on line 14 lies outside this block, producing the crash described
in bazel-contrib#1648 (commit 23ae356). This is why
we need the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator`.
- https://github.com/scalapb/ScalaPB/blob/v0.9.8/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L14
`scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator`,
also extends `ProtocCodeGenerator` directly, instead of implementing
`CodeGenApp` (so it also builds with Scala 2.11). This is why
`ExtraProtobufGenerator.run()` hangs when we remove the `try`/`catch`
block and run with `protoc-bridge` 0.9.7 and `protobuf` > v25.5, even
with later Scala versions.
---
Since `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17
implements `CodeGenApp`, `scripts.ScalaPbCodeGenerator` for Scala 2.12
should be unnecessary. The only reason we need it is to maintain the
same interface as the Scala 2.11 implementation.
---
Before the `MavenCoordinates` changes, `scripts/create_repository.py`
would ScalaPB flip ScalaPB artifacts between their `_2.13` and `_3`
versions on subsequent runs (e.g., `protoc-bridge_2.13` vs.
`protoc-bridge_3`). See the comments added within
`MavenCoordinates.new()` for details.
---
Building Scala 3.3 and later versions with `compilerplugin_3` and
`protoc-bridge_3` artifacts (instead of `protoc-bridge_2.13` artifacts)
produces the following build failure:
```txt
$ RULES_SCALA_TEST_ONLY="test_scala_version 3.6.2" \
./test_thirdparty_version.sh
[ ...snip... ]
ERROR: third_party/test/proto/BUILD.bazel:4:14:
ProtoScalaPBRule
third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar
failed: (Exit 1): scalapb_worker failed: error executing command
(from target //third_party/test/proto:proto)
bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker
... (remaining 2 arguments skipped)
--scala_out: java.lang.NoClassDefFoundError:
Could not initialize class scalapb.ScalaPbCodeGenerator$
at scripts.ScalaPbCodeGenerator$.process(ScalaPbCodeGeneratorWrapper.scala:8)
at protocgen.CodeGenApp.run(CodeGenApp.scala:48)
at protocgen.CodeGenApp.run$(CodeGenApp.scala:41)
at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5)
at protocgen.CodeGenApp.run(CodeGenApp.scala:33)
at protocgen.CodeGenApp.run$(CodeGenApp.scala:32)
at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5)
at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48)
at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113)
at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$1(SocketBasedPluginFrontend.scala:31)
at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:37)
at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:60)
at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118)
at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:71)
at scala.concurrent.package$.blocking(package.scala:124)
at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1(SocketBasedPluginFrontend.scala:37)
at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:38)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
java.lang.RuntimeException: Exit with code 1
at scala.sys.package$.error(package.scala:27)
at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44)
at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96)
at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49)
at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39)
at scripts.ScalaPBWorker.main(ScalaPBWorker.scala)
ERROR: third_party/test/proto/BUILD.bazel:4:14 Building source jar
third_party/test/proto/proto_scalapb-src.jar failed: (Exit 1):
scalapb_worker failed: error executing command
(from target //third_party/test/proto:proto)
bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker
... (remaining 2 arguments skipped)
```
simuons
approved these changes
Jan 31, 2025
liucijus
approved these changes
Feb 3, 2025
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Feb 28, 2025
Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0. Part of bazel-contrib#1482. Closes bazel-contrib#1652. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `abseil-cpp`: 20220623.1 => 20250127.0 - `rules_java`: 7.12.4 => 8.9.0 - `protobuf`: 21.7 => 29.3 - `rules_proto`: 6.0.2 => 7.1.0 - `google-common-protos`: 2.52.0 => 2.53.0 This precipitated the following updates also included in this commit: - Loads `java_proto_library` from `com_google_protobuf`. - Bumps `.bazelversion` to 7.5.0. - Sets `.bazelci/presubmit.yml` to use Bazel 7.5.0 instead of 6.5.0, and `last_rc` in place of `7.x`. - Sets `common --enable_workspace --noenable_bzlmod` in `.bazelrc` and `tools/bazel\.rc\.buildkite`. - Adds `allow_empty = True` to a `glob` expression in `//test/semanticdb:lib_with_tempsrc`. - Removes Scala 2.11 test cases. Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. `WORKSPACE` builds succeed under Bazel 6.5.0 (with C++ compiler flags) for the time being, but are not guaranteed to continue working. (Bazel 6 Bzlmod builds would break anyway, because Bazel 6 doesn't provide `use_repo_rule`, required by `rules_jvm_external` 6.3, which is required by `protobuf` v29.) The `README` now documents that `scala_proto` or any rules otherwise depending on `protobuf` are no longer supported out of the box for Scala 2.11. Such users will have to ensure they register their own downgraded versions, at their own risk of future `rules_scala` 7.x incompatibility. Version bump related updates to `.bazelversion`,`WORKSPACE`, and `third_party/repositories` comprise the bulk of this change. Other small, yet important changes (other than those due to the version bumps noted above) include: - Adding a new `examples/overridden_artifacts` repository and the `overridden_artifacts_example` test case in `test/shell/test_examples.sh`. - Making `_validate_scalac_srcjar()` and `dt_patched_compiler_setup()` in `scala/private/macros/scala_repositories.bzl` more tolerant of dictionaries containing keys mapped to `None`. The new `overridden_artifacts_example` test covers this. Bazel 8 and `rules_java` 8 require `protobuf` >= v29. After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` Here's why the other changes were necessary in light of the version bumps: - `java_proto_library` from `rules_java` is now officially deprecated, hence loading it from `com_google_protobuf`. - Setting `common --enable_workspace --noenable_bzlmod` (instead of `build`) fixes `test_semanticdb_handles_removed_sourcefiles`. This test relies on `bazel query`, which is also affected by these flags, hence `common` instead of `build`. Bazel 8 defaults to `--enable_bzlmod --noenable_workspace`, causing the `WORKSPACE` run of this test to fail. - `glob` requires an explicit `allow_empty = True` parameter now that `--incompatible_disallow_empty_glob` defaults to `True` in Bazel 8. - ScalaPB 0.9.8, the last version compatible with Scala 2.11, does not support `protobuf` >= 26.0. For this reason, we must remove the Scala 2.11 test cases. We should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of `protobuf`. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details: - bazel-contrib#1631 - bazel-contrib#1648 - bazel-contrib#1687 - bazel-contrib#1688 Finally, the motivation for the non-version bump changes: - The design bug in the upcoming Bzlmod API for `overridden_artifacts` reported by @dmivankov precipitated the `examples/overriden_artifacts` repo and test. See: bazel-contrib#1482 (comment) bazel-contrib#1482 (comment) - The `_validate_scalac_srcjar()` update maintains the strict checks for mutually exclusive values, while preventing client code from having to explicitly filter out `None` entries. This pairs with the change in `dt_patched_compiler_setup()` that uses the `compiler_srcjar` dictionary. These changes helps keep the upcoming module extension a bit cleaner.
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Feb 28, 2025
Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0. Part of bazel-contrib#1482. Closes bazel-contrib#1652. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `abseil-cpp`: 20220623.1 => 20250127.0 - `rules_java`: 7.12.4 => 8.9.0 - `protobuf`: 21.7 => 29.3 - `rules_proto`: 6.0.2 => 7.1.0 - `google-common-protos`: 2.52.0 => 2.53.0 This precipitated the following updates also included in this commit: - Loads `java_proto_library` from `com_google_protobuf`. - Bumps `.bazelversion` to 7.5.0. - Sets `.bazelci/presubmit.yml` to use Bazel 7.5.0 instead of 6.5.0, and `last_rc` in place of `7.x`. - Sets `common --enable_workspace --noenable_bzlmod` in `.bazelrc` and `tools/bazel\.rc\.buildkite`. - Adds `allow_empty = True` to a `glob` expression in `//test/semanticdb:lib_with_tempsrc`. - Removes Scala 2.11 test cases. Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. `WORKSPACE` builds succeed under Bazel 6.5.0 (with C++ compiler flags) for the time being, but are not guaranteed to continue working. (Bazel 6 Bzlmod builds would break anyway, because Bazel 6 doesn't provide `use_repo_rule`, required by `rules_jvm_external` 6.3, which is required by `protobuf` v29.) The `README` now documents that `scala_proto` or any rules otherwise depending on `protobuf` are no longer supported out of the box for Scala 2.11. Such users will have to ensure they register their own downgraded versions, at their own risk of future `rules_scala` 7.x incompatibility. Version bump related updates to `.bazelversion`,`WORKSPACE`, and `third_party/repositories` comprise the bulk of this change. Other small, yet important changes (other than those due to the version bumps noted above) include: - Adding a new `examples/overridden_artifacts` repository and the `overridden_artifacts_example` test case in `test/shell/test_examples.sh`. - Making `_validate_scalac_srcjar()` and `dt_patched_compiler_setup()` in `scala/private/macros/scala_repositories.bzl` more tolerant of dictionaries containing keys mapped to `None`. The new `overridden_artifacts_example` test covers this. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (the first supporting with Bazel 6, the second dropping support). Bazel 8 and `rules_java` 8 require `protobuf` >= v29. After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` Here's why the other changes were necessary in light of the version bumps: - `java_proto_library` from `rules_java` is now officially deprecated, hence loading it from `com_google_protobuf`. - Setting `common --enable_workspace --noenable_bzlmod` (instead of `build`) fixes `test_semanticdb_handles_removed_sourcefiles`. This test relies on `bazel query`, which is also affected by these flags, hence `common` instead of `build`. Bazel 8 defaults to `--enable_bzlmod --noenable_workspace`, causing the `WORKSPACE` run of this test to fail. - `glob` requires an explicit `allow_empty = True` parameter now that `--incompatible_disallow_empty_glob` defaults to `True` in Bazel 8. - ScalaPB 0.9.8, the last version compatible with Scala 2.11, does not support `protobuf` >= 26.0. For this reason, we must remove the Scala 2.11 test cases. We should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of `protobuf`. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details: - bazel-contrib#1631 - bazel-contrib#1648 - bazel-contrib#1687 - bazel-contrib#1688 Finally, the motivation for the non-version bump changes: - The design bug in the upcoming Bzlmod API for `overridden_artifacts` reported by @dmivankov precipitated the `examples/overriden_artifacts` repo and test. See: bazel-contrib#1482 (comment) bazel-contrib#1482 (comment) - The `_validate_scalac_srcjar()` update maintains the strict checks for mutually exclusive values, while preventing client code from having to explicitly filter out `None` entries. This pairs with the change in `dt_patched_compiler_setup()` that uses the `compiler_srcjar` dictionary. These changes helps keep the upcoming module extension a bit cleaner.
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Mar 6, 2025
It turns out `protobuf` v26.6 isn't compatibile with ScalaPB 0.9.8 used by Scala 2.11: ```txt Caused by: java.lang.UnsupportedOperationException: As of 2022/09/29 (release 21.7) makeExtensionsImmutable should not be called from protobuf gencode. If you are seeing this message, your gencode is vulnerable to a denial of service attack. You should regenerate your code using protobuf 25.6 or later. Use the latest version that meets your needs. However, if you understand the risks and wish to continue with vulnerable gencode, you can set the system property `-Dcom.google.protobuf.use_unsafe_pre22_gencode` on the command line. See security vulnerability: GHSA-h4h5-3hr4-j3g2 ``` As mentioned in bazel-contrib#1710, we should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of `protobuf`. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details: - bazel-contrib#1631 - bazel-contrib#1648 - bazel-contrib#1687 - bazel-contrib#1688
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Mar 7, 2025
Contains many editorial improvements to and some extra information in the README, along with a few small improvements to the code. Broken out from bazel-contrib#1710, and part of bazel-contrib#1482 and bazel-contrib#1652. Specifically: - Adds more info on translating `@rules_scala` to `@io_bazel_rules_scala` for dependencies via repo mapping. - Merges information about the previously planned `rules_scala` 8.0.0 release into the information for 7.0.0, since it seems we may make only one major release. - Improves information about `protobuf` support for versions before v28, Scala 2.11, and the upcoming Bzlmod `compatibility_level` setting. - In `scala_config.bzl`, changes the private `_default_scala_version()` to the public `DEFAULT_SCALA_VERSION`. - Adds `allow_empty = True` to a `glob` expression in `//test/semanticdb:lib_with_tempsrc`. - Removes Scala 2.11 test cases from `test_thirdparty_version.sh` and `test_version.sh`. This change is smaller and more focused than bazel-contrib#1710, and should ultimately make that pull request smaller and/or easier to review. The motivations for the individual changes are: - The public `DEFAULT_SCALA_VERSION` constant makes this value accessible to the upcoming module extension. - `glob` requires an explicit `allow_empty = True` parameter in Bazel 8, in which `--incompatible_disallow_empty_glob` defaults to `True`. - ScalaPB 0.9.8, the last version compatible with Scala 2.11, does not support `protobuf` v25.6 or later. For this reason, we must remove the Scala 2.11 test cases, as documented in the `README.md` updates. See also bazel-contrib#1712. We should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of `protobuf`. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details: - bazel-contrib#1631 - bazel-contrib#1648 - bazel-contrib#1687 - bazel-contrib#1688
simuons
pushed a commit
that referenced
this pull request
Mar 10, 2025
* 7.0.0 README updates and minor code changes Contains many editorial improvements to and some extra information in the README, along with a few small improvements to the code. Broken out from #1710, and part of #1482 and #1652. Specifically: - Adds more info on translating `@rules_scala` to `@io_bazel_rules_scala` for dependencies via repo mapping. - Merges information about the previously planned `rules_scala` 8.0.0 release into the information for 7.0.0, since it seems we may make only one major release. - Improves information about `protobuf` support for versions before v28, Scala 2.11, and the upcoming Bzlmod `compatibility_level` setting. - In `scala_config.bzl`, changes the private `_default_scala_version()` to the public `DEFAULT_SCALA_VERSION`. - Adds `allow_empty = True` to a `glob` expression in `//test/semanticdb:lib_with_tempsrc`. - Removes Scala 2.11 test cases from `test_thirdparty_version.sh` and `test_version.sh`. This change is smaller and more focused than #1710, and should ultimately make that pull request smaller and/or easier to review. The motivations for the individual changes are: - The public `DEFAULT_SCALA_VERSION` constant makes this value accessible to the upcoming module extension. - `glob` requires an explicit `allow_empty = True` parameter in Bazel 8, in which `--incompatible_disallow_empty_glob` defaults to `True`. - ScalaPB 0.9.8, the last version compatible with Scala 2.11, does not support `protobuf` v25.6 or later. For this reason, we must remove the Scala 2.11 test cases, as documented in the `README.md` updates. See also #1712. We should consider dropping Scala 2.11 support at this point, since there's no ScalaPB release for it that supports later versions of `protobuf`. That, and we could remove some of the special case code added in the following changes, amongst other 2.11 support details: - #1631 - #1648 - #1687 - #1688 * Tweak `rules_python` and Bazel 6.5.0 `README` info Provided a better explanation for using `rules_python` 0.38.0 for now. Improved some of the language in the Bazel 6.5.0 compatibility section.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This updates to the latest
protoc-bridge, removes code that isn't necessary any longer, and documents code that must remain for now. Part of #1482.This also required the following
scripts/create_repository.pyupdates:_2.13and_3versions of ScalaPB artifacts.MavenCoordinatesdataclass inscripts/create_repository.pyhas newunversioned_artifact,scala_version, andartifact_namemembers. Updates toMavenCoordinates.is_newer_than()now compare objects with matchingartifact_namevalues, and take thescala_versioninto account.These changes then precipitated:
artifact_name()call withartifact_nameaccesses.ArtifactLabelMaker._remove_scala_version_suffixwithMavenCoordinates.unversioned_artifact.__compare_versionsfromis_newer_thanto reuse the same code for comparing Scala versions and artifact versions. (Properly reversed the use oflhsandrhsin the process.)This change also removes an unnecessary
try/catchblock fromscripts.ScalaPbCodeGenerator.process()for Scala 2.12 and above. The block is still required for the Scala 2.11 implementation ofscripts.ScalaPbCodeGenerator.run()and inscalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run().Finally, this change adds notes in a few places to indicate where support remains for Scala 2.11. This supporting code can ultimately be removed if we ever decide to drop Scala 2.11 support.
Motivation
The initial motivation for this change was to try to eliminate the need for the custom
scripts.ScalaPbCodeGeneratorimplementation by upgrading toprotoc-bridge0.9.8. This version contains a change tocatchanyThrowableobjects raised by a generator implementation inprotocbridge.frontend.PluginFrontend.runWithInputStream().However, the last
protoc-bridgeversion to support Scala 2.11 is 0.7.14. Scala 2.11 won't be able to useprotoc-bridge0.9.8, so we still need tocatchanyThrowables ourselves. This includes thecatchblocks from the Scala 2.11scripts.ScalaPbCodeGeneratorandscalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run.Since
scalapb.ScalaPbCodeGeneratorfromcompilerplugin0.11.17 implementsCodeGenApp,scripts.ScalaPbCodeGeneratorfor Scala 2.12 should be unnecessary. The only reason we need it is to maintain the same interface as the Scala 2.11 implementation.The
_3versions of the most recent ScalaPB artifacts are only available for Scala 3.3 and later, andcompilerplugin_3currently still depends onprotoc-gen_2.13. Building Scala 3.3 and later versions withcompilerplugin_3andprotoc-bridge_3artifacts (instead ofprotoc-bridge_2.13artifacts) produced a build failure.Before the
MavenCoordinateschanges,scripts/create_repository.pywould flip ScalaPB artifacts between their_2.13and_3versions on subsequent runs (e.g.,protoc-bridge_2.13vs.protoc-bridge_3). See the comments added withinMavenCoordinates.new()for details.The commit message expands greatly on the above details. I've also broken my recent one-pull-request-at-a-time rule because this change is orthogonal to all the others. (All of the remaining changes described in #1482, possibly aside from documentation updates, should land sequentially.)