Support importing across repos with bzlmod

[matts1]

🚀 feature request

Relevant Rules

py_library / py_binary

Description

If there is a python library @foo//bar:baz, how can I import it from the main repository?

Pre-bzlmod, you could write from foo.bar import baz, as it would be in the directory external/foo/bar/baz, and the external directory was in your python path. Apparently this was not documented or intended, but nevertheless, people used it, as I don't believe it was ever considered when writing rules_python.

In bzlmod, however, you have repo mapping, where the directory foo is actually stored at external/_main~~foo_ext~foo, and thus from foo.bar import baz fails. This is the direct cause of #1679, and if we solve this, we fix that bug for free.

Describe the solution you'd like

I'd personally like to make users prefix their imports to external repositories with the name of the workspace (based on the repo mapping). This would allow two python files in different repositories to import foo.bar.baz and each get their own @foo in the case of a repo name clash (this is how repo mapping is intended to be used, to allow each repo to have their own namespace of repos).

I previously achieved this in ChromeOS via a sitecustomize.py (see discussion in #1679).

Describe alternatives you've considered

I can't really think of a viable alternative. Any solution without including the repo name in the import seems like it will end up in painful corner cases where you have name conflicts.

[groodt]

Any thoughts on how this should behave if the @foo repository uses third-party dependencies from PyPI?

e.g. pydantic==1.10.17 but the root respository wishes to use pydantic==2.8.2?

Or is the expectation only limited to cross-repository dependencies that do not take additional dependencies beyond the leaf repository?

[groodt]

For the simpler cases, one option that would work for you today with bzlmod and the third-party dependency rules in rules_python, would be to use a direct url reference.

e.g foo @ git+https://github.com/foo/foo.git@1.3.1#7921be1537eac1e97bc40179a57f0349c2aee67d

That will behave as a simple sdist dependency to the resolution systems (and would work with transitive dependency resolution) and for most pure python scenarios, would work well.

For situations that involve native dependencies, or complex build backends, it becomes a lot more difficult to build from source through that mechanism.

Other simple options to consider are adding a GitHub Action that packages the foo dependency as a wheel to PyPI, or a private repository, or a github artifact and then you can also pull that down through the existing rules_python third-party dependency rules as well.

[matts1]

Any thoughts on how this should behave if the @foo repository uses third-party dependencies from PyPI?

e.g. pydantic==1.10.17 but the root respository wishes to use pydantic==2.8.2?

I believe that in that scenario, @foo's import pydantic should import 1.10.17, while the root repository's import pydantic should resolve to importing 2.8.2 (that isn't usually what you want, but in practice it should be dealt with by the module extension itself).

If @foo's module.bazel declares a dependency on 1.10.17, and the root module declares 2.8.2, it's the responsibility of rules_python to perform MVS and end up with 2.8.2. This means that in practice, they will both end up resolving to the same version.

That being said, sometimes you don't want that, which is where isolated extension usages come in (unfortunately I couldn't find good documentation on it, but it's at bazelbuild/bazel#20186)

[groodt]

it's the responsibility of rules_python to perform MVS and end up with 2.8.2. This means that in practice, they will both end up resolving to the same version.

The MVS algorithm from bzlmod applies to the versions of the bazel extensions in BCR themselves, not of any language dependencies of the code inside the root or leaf respositories.

isolated extension usages

I think this scenario is more or less solved with bzlmod, because in these scenarios, there wouldn't be python code from the root repository importing python code from a leaf repository right? The extension would provide some sort of toolchain or tool that can be used without leaking at all into the root repository. Agree?

Can you clarify a bit about your use-case?

Is your use-case an inter-bazel-extension dependency scenario? So where you want one bazel extension to be able to import python code from another bazel extension?

Or is your use-case an arbitrary source code sharing scenario outside bazel extensions?

[matts1]

The MVS algorithm from bzlmod applies to the versions of the bazel extensions in BCR themselves, not of any language dependencies of the code inside the root or leaf respositories.

That's correct, but if you have multiple module extension usages, each requesting a different set of versions of dependencies, it's up to the module extenison to decide what to do. You can give each precisely the thing verison they requested, or attempt to merge them using MVS. Generally we recommend that module extensions perform MVS because that usually works, and if it doesn't, isolated extenison usages allow you to ensure that each gets precisely what they asked for.

isolated extension usages

I think this scenario is more or less solved with bzlmod, because in these scenarios, there wouldn't be python code from the root repository importing python code from a leaf repository right? The extension would provide some sort of toolchain or tool that can be used without leaking at all into the root repository. Agree?

Generally what happens is:

1. root repo requests foo v1.2
2. other repo requests foo v1.1
3. Pip module extension receives both of these requests at the same time, performs MVS on the requests, and creates a repo @rules_python~~pip_ext~pip containing foo v1.2
4. Both of these repos map @pip to @rules_python~~pip_ext~pip, thus both getting foo v1.2

With isolated extensions, what happens is:

1. Root repo requests foo v1.2 isolated
2. Pip module extension receives the request, and creates a repo @rules_python~~~main~~pip_ext~pip containing foo v1.2
3. @pip is mapped in the main repo to @rules_python~~~main~~pip_ext~pip , thus getting foo v1.2
4. other repo requests foo v1.1
5. pip module extension receives this request, and creates a repo @rules_python~~pip_ext~pip containing foo v1.1
6. Other repo maps @pip to @rules_python~~pip_ext~pip, thus getting foo v1.1

Can you clarify a bit about your use-case?

Is your use-case an inter-bazel-extension dependency scenario? So where you want one bazel extension to be able to import python code from another bazel extension?

Or is your use-case an arbitrary source code sharing scenario outside bazel extensions?

Keep in mind that when using bzlmod, every* external repo is a module extension. We create the repository rule for chromite, then we have to use the repo rule in a module extension, and finally you make it visible to the main repo in MODULE.bazel. So although our chromite is just a simple repository rule, the path is still repo-mapped as _main~~cros_deps~chromite.

Our want is that we want the main repo to be able to import python files from other repos (but if you can get it working from the main repo, it's already repo-mapping aware, and you can get it working from any repo pretty trivially). We have a multirepo (not ideal for bazel, but it's a historical decision). As an example, we have cache.py stored at @chromite//lib:cache. So from the main repo, we'd like to be able to run import chromite.lib.cache.

*Technically, not true. You can invoke repo rules directly in the MODULE.bazel files, though it's much more limited in what you can do with it.

[groodt]

each requesting a different set of versions of dependencies, it's up to the module extenison to decide what to do

The third-party dependencies in rules_python are repositories, they're not module extensions. So import pydantic inside a python source file refers to a python dependency, not a bazel module extension dependency.

[matts1]

I think we might be getting confused because we're using different terminology. pydantic, in your example, is a repository created by the pip module extension. (@pip//pydantic is actually an alias to something along the lines of @@rules_python~~pip_ext~pydantic~1.2.3//:pydantic).

My original thought was that the solution would be to not special-case pip, but instead the pip module extension would generate a repository @@rules_python~~pip~pydantic which would then be visible via repo-mapping as @pydantic, and so import pydantic.foo would just import foo within the repo visible as @pydantic. I'm not sure how viable it is - bazel's new auto use_repo fixups (bazelbuild/bazel#17908) should help.

[groodt]

Thanks for the details. It really seems like the real issue is this:

Our want is that we want the main repo to be able to import python files from other repos

Is this supported in Java?

I would think you could possibly pull things in the way you wish if you did something like this:

MODULE.bazel

http_archive = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(name = "foo", urls = [...])

However, I'm not sure thats too much better than having your leaf repos publish source packages or pulling in the source packages through the pypi rules as direct url references.

[groodt]

My original thought was that the solution would be to not special-case pip, but instead the pip module extension would generate a repository @@rules_python~~pip~pydantic which would then be visible via repo-mapping as @pydantic, and so import pydantic.foo would just import foo within the repo visible as @pydantic. I'm not sure how viable it is

The implementation details of how we are currently importing the third-party python language ecosystem to be convenient for bazel is quite a different problem to merging multiple independent source trees through bazel.

Our pip module extension aims to take external packages from the Python ecosystem (sdist and bdist typically on PyPI but can be in private artifact storage too) and make these packages importable by python code in the root repository. These bazel foreign dependencies don't have prefixes on their python imports. So users can write python code as they would outside bazel. So "import numpy" is the same inside and outside bazel for numpy users.

Your ask is more around merging non packaged python code (ie no sdist or wheel anywhere) and having special python imports for these. That's quite a different problem space to the pip problem space. That's a bazel source deps / vendoring problem. It's not a pip/pypi python packaging problem.

It's possible to change the source deps / vendoring problem into a packaging problem if you can expose your source as a package. That's what we did by publishing "runfiles" as a wheel to PyPI. But otherwise these problems are different.

[aignas]

I'll try to add my two cents about how I understand what the ask is. I think we are conflating here quite a few problems.

1. Right now the pip.parse is assuming that python trees are isolated and we ask the user to provide a unique hub_name, which is an alternative to using the isolated module extensions. I think it is likely that we should just move to use the isolated extensions by default, just like what rules_go is forcing users to isolate their extension usage for third party dependencies if the user does not want the merging of third party dependency trees.

2. Doing MVS across multiple external dependency trees is something that we try to not do yet because we don't yet have a stable solution for "Cross compilation" of py_binary/py_image/py_library targets #260. In our case the merging is not super complex if we just chose the minimum version out of all requirement lock files, but it is also something that we haven't experimented yet. By default many dependency solvers select the highest available version of the dependencies and then doing the MVS without being more in the weeds is not possible. Because go mod is using MVS, rules_go is having an easier time to do that there, I think.

3. The imports relying on repository names previously worked but broke due to bazel moving to repo mapping model and it is quite correct in saying that you cannot rely on the repo names when doing imports. Even python proper is not doing that as the names of the imports are different from the wheel or repository names. My stance here would be to not support this ask and just follow what the bazel core developers said - the name of the repo is unstable and should not be relied upon as the repo mapping cannot work otherwise. A solution to import rules_python doesn't work under bzlmod #1679 is not pretty or generic and requires rules_python to move code around, so it is unlikely that it will ever get used.

To summarize, for now bzlmod does not really support the mixing of dependency trees and it is because in rules python we don't have a good idea on how to combine deps of one bazel_dep and another bazel_dep. Combining them requires inputting both of the requirements files and creating a new lock file. With the previous tools it would have been really difficult and with uv it is becoming more tangible, but I am not sure if it is possible. The only way to share code between two python monorepos is to ship code as wheels and consume it via requirements.

Is it something that we should block on before we release 1.0? I am not sure. Is it possible to solve it in rules_python? Possibly, but at minimum it would require users to use imports that do not rely on repository names. And we would need to understand what interface we should expose for locking requirements. The only option that I can think of today is:

• users specify their input files to pip.parse extension
• the pip.parse extension exposes a target that allows users to lock the requirements. If the usage of the extensions is not isolated, the requirements locker would get all of the inputs files as constraints.
• the lock file that gets materialized and subsequently used by rules_python is a result of all of those input files.

The main issue I see is that we cannot solve this without modifying the bzlmod workflow. I'll add a 1.0.0 milestone due to this reason.

[matts1]

Thanks for the details. It really seems like the real issue is this:

Our want is that we want the main repo to be able to import python files from other repos

That's correct, yes

I would think you could possibly pull things in the way you wish if you did something like this:

MODULE.bazel

http_archive = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(name = "foo", urls = [...])

I don't believe so. I haven't actually tried it, but since two different MODULE.bazel files could add that exact same line of code, it would have to be namespaced as _main~~foo or similar.

It's possible to change the source deps / vendoring problem into a packaging problem if you can expose your source as a package. That's what we did by publishing "runfiles" as a wheel to PyPI. But otherwise these problems are different.

Yeah, I was trying to make them the same problem by going the other way round - turning the packaging problem into a bazel problem. It was mostly done out of curiosity - I don't know if of any particular advantage to it other than consistency.

I don't think exposing the source as a package is a good long-term solution. Any solution other than just adding the py_library target as a dep and then importing it is a hack, IMO, because that's the clear winner from a UX and consistency standpoint.

I'll try to add my two cents about how I understand what the ask is. I think we are conflating here quite a few problems.

1. Right now the pip.parse is assuming that python trees are isolated and we ask the user to provide a unique hub_name, which is an alternative to using the isolated module extensions. I think it is likely that we should just move to use the isolated extensions by default, just like what rules_go is forcing users to isolate their extension usage for third party dependencies if the user does not want the merging of third party dependency trees.

2. Doing MVS across multiple external dependency trees is something that we try to not do yet because we don't yet have a stable solution for "Cross compilation" of py_binary/py_image/py_library targets #260. In our case the merging is not super complex if we just chose the minimum version out of all requirement lock files, but it is also something that we haven't experimented yet. By default many dependency solvers select the highest available version of the dependencies and then doing the MVS without being more in the weeds is not possible. Because go mod is using MVS, rules_go is having an easier time to do that there, I think.

SGTM, though like you said, this should probably be in a different bug

3. The imports relying on repository names previously worked but broke due to bazel moving to repo mapping model and it is quite correct in saying that you cannot rely on the repo names when doing imports. Even python proper is not doing that as the names of the imports are different from the wheel or repository names. My stance here would be to not support this ask and just follow what the bazel core developers said - the name of the repo is unstable and should not be relied upon as the repo mapping cannot work otherwise. A solution to import rules_python doesn't work under bzlmod #1679 is not pretty or generic and requires rules_python to move code around, so it is unlikely that it will ever get used.

That sounds like something that may be taking the bazel core developers out of context. While I haven't heard the statement, I'm guessing that the intention of their statement was that the full repository name is unstable, but that the repo-mapped name is stable (ie. _main~~foo_ext~foo is unstable, but if it was imported as "foo" in the main repo, then foo is a stable identifier that can be used to refer to that repo from the main repo). So I think it's perfectly reasonable to allow users to use import foo to import the repository visible as foo. If the repo-mapped name was unsstable, then depending on @pip//foo would be unstable in your build files, which is obviously false.

To summarize, for now bzlmod does not really support the mixing of dependency trees and it is because in rules python we don't have a good idea on how to combine deps of one bazel_dep and another bazel_dep. Combining them requires inputting both of the requirements files and creating a new lock file. With the previous tools it would have been really difficult and with uv it is becoming more tangible, but I am not sure if it is possible. The only way to share code between two python monorepos is to ship code as wheels and consume it via requirements.

I think that @groodt is right, and that we should leave them as two seperate problems here. This is not about requirements or dependencies, but about regular py_library targets. We can package code just fine. We have access to all the information we need (the REPO_MAPPING file and the paths. it's simply a question of hooking in to the import system to import it correctly. I already implemented this for ChromeOS with a sitecustomize.py, but I don't know if there's a better way to do it.

Is it something that we should block on before we release 1.0? I am not sure. Is it possible to solve it in rules_python? Possibly, but at minimum it would require users to use imports that do not rely on repository names. And we would need to understand what interface we should expose for locking requirements. The only option that I can think of today is:

• users specify their input files to pip.parse extension
• the pip.parse extension exposes a target that allows users to lock the requirements. If the usage of the extensions is not isolated, the requirements locker would get all of the inputs files as constraints.
• the lock file that gets materialized and subsequently used by rules_python is a result of all of those input files.

The main issue I see is that we cannot solve this without modifying the bzlmod workflow. I'll add a 1.0.0 milestone due to this reason.

I don't think it should be a blocker

[groodt]

I don't think exposing the source as a package is a good long-term solution. Any solution other than just adding the py_library target as a dep and then importing it is a hack, IMO, because that's the clear winner from a UX and consistency standpoint.

It's convenient for you because you seem to be primarily within a bazel ecosystem, but with a slightly awkward setup where you have a polyrepo of unpackaged python source dependencies.

However, we need to recognize that the overwhelming majority of the users of rules_python and of python in general are not in that position. The PyPA standards specify the mechanisms to distribute python packages across different projects and repositories. They can be found here packaging and distribution standards and describe sdist (source distributions) and bdist (binary distributions aka wheels).

Going "with the grain" and publishing packages that align to the standards wouldn't be a hack I think. It's actually what the vast majority of python developers work with on a daily basis. It would be more consistent with the expectations of the average python developer and wouldn't be rewriting or introducing something at python import level. Introducing that would be inconsistent to my expectations as a python developer. In a similar way that a Java engineer would turn up their nose at import rules_java.com.google.bazel.blabla;

There are no language standards for monorepos or source dependencies (that aren't sdist) like there might be for e.g. golang. Golang naturally has the benefit where the language was designed with bazel in mind and the language import structure itself maps to source repository structures.