refactor: reimplement writing namespace pkgs in Starlark #2882

aignas · 2025-05-15T14:05:25Z

With this PR I would like to facilitate the implementation of the venv
layouts because we can in theory take the srcs and the data within
the py_library and then use the expand_template to write the extra
Python files if the namespace_pkgs flag is enabled.

The old Python code has been removed and the extra generated files are
written out with bazel_skylib copy_file.

The implicit namespace_pkg init files are included to py_library
if the site-packages config flag is set to false and I think this
may help with continuing the implementation, but it currently is still
not working as expected (see comment).

Work towards #2156

rickeylev · 2025-05-16T19:42:44Z

This is basically a reimplementation of https://github.com/bazel-contrib/rules_python/blob/main/python/private/pypi/whl_installer/namespace_pkgs.py right?

Mostly LGTM.

Is it handling the "# The root of the directory should never be an implicit namespace" condition?

remove the python code, put under pipstar

I think you mean: "if pipstar is enabled, don't run the python version"

This doesn't have to be tied to pipstar. It can just be moved to be part of the pip-generated BUILD files. e.g.

py_library(
  name = "pkg",
  srcs = [":_pkgutil_inits"] + glob(**.py)
)

pkgutil_inits( # produces 0 or more pkgutil init.py files based on what it sees in srcs
  name = "_pkgutil_inits",
  srcs = glob("**")
)

aignas · 2025-05-18T03:14:29Z

Ran:

$ bazel build //docs --@rules_python//python/config_settings:venvs_site_packages=yes
INFO: Analyzed target //docs:docs (0 packages loaded, 0 targets configured).
ERROR: /home/aignas/src/github/aignas/rules_python/docs/BUILD.bazel:45:12: Sphinx building html for //docs:docs failed: (Exit 2): sphinx-build failed: error executing SphinxBuildDocs command (from target //docs:docs) bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build --show-traceback --builder html --quiet --jobs auto --fresh-env --write-all bazel-out/k8-fastbuild/bin/docs/_docs/_sources ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging

Traceback (most recent call last):
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/sphinx/cmd/build.py", line 496, in build_main
    app = Sphinx(
          ^^^^^^^
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/sphinx/application.py", line 256, in __init__
    self.setup_extension(extension)
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/sphinx/application.py", line 437, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/sphinx/registry.py", line 466, in load_extension
    metadata = setup(app)
               ^^^^^^^^^^
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/__init__.py", line 8, in setup
    from .sphinx.extension import setup as _setup
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/sphinx/extension.py", line 18, in <module>
    from autodoc2.sphinx.autodoc import AutodocObject
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/sphinx/autodoc.py", line 12, in <module>
    from autodoc2.sphinx.utils import (
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/sphinx/utils.py", line 15, in <module>
    from autodoc2.utils import WarningSubtypes
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/utils.py", line 10, in <module>
    from typing_extensions import Required
ModuleNotFoundError: No module named 'typing_extensions'

Exception occurred:
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/6/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/sphinx-build.runfiles/_main/docs/_sphinx-build.venv/lib/python3.11/site-packages/autodoc2/utils.py", line 10, in <module>
    from typing_extensions import Required
ModuleNotFoundError: No module named 'typing_extensions'
The full traceback has been saved in /tmp/sphinx-err-twbss_8e.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Target //docs:docs failed to build
Use --verbose_failures to see the command lines of failed build steps.

I checked the venv layout:

$ ll bazel-out/k8-opt-exec-ST-d57f47055a04/bin/docs/_sphinx-build.venv/lib/python3.11/site-packages
.r-xr-xr-x 8.4k aignas 18 May 12:07 _bazel_site_init.py
lrwxrwxrwx    - aignas 18 May 12:00 _yaml -> ../../../../../../+pip+dev_pip_311_pyyaml_cp311_cp311_manylinux_2_17_x86_64_3ad2a3de/site-packages/_yaml
lrwxrwxrwx    - aignas 18 May 12:00 alabaster -> ../../../../../../+pip+dev_pip_311_alabaster_py3_none_any_fc678640/site-packages/alabaster
lrwxrwxrwx    - aignas 18 May 12:00 astroid -> ../../../../../../+pip+dev_pip_311_astroid_py3_none_any_d05bfd0a/site-packages/astroid
lrwxrwxrwx    - aignas 18 May 12:00 autodoc2 -> ../../../../../../+pip+dev_pip_311_sphinx_autodoc2_py3_none_any_e867013b/site-packages/autodoc2
lrwxrwxrwx    - aignas 18 May 12:00 babel -> ../../../../../../+pip+dev_pip_311_babel_py3_none_any_4d0b5309/site-packages/babel
.r-xr-xr-x   24 aignas 12 May 19:45 bazel.pth
lrwxrwxrwx    - aignas 18 May 12:00 certifi -> ../../../../../../+pip+dev_pip_311_certifi_py3_none_any_ca78db45/site-packages/certifi
lrwxrwxrwx    - aignas 18 May 12:00 charset_normalizer -> ../../../../../../+pip+dev_pip_311_charset_normalizer_cp311_cp311_manylinux_2_17_x86_64_fd4ec41f/site-packages/charset_normalizer
lrwxrwxrwx    - aignas 18 May 12:00 docutils -> ../../../../../../+pip+dev_pip_311_docutils_py3_none_any_dafca5b9/site-packages/docutils
lrwxrwxrwx    - aignas 18 May 12:00 idna -> ../../../../../../+pip+dev_pip_311_idna_py3_none_any_946d195a/site-packages/idna
lrwxrwxrwx    - aignas 18 May 12:00 imagesize -> ../../../../../../+pip+dev_pip_311_imagesize_py2_none_any_0d8d18d0/site-packages/imagesize
lrwxrwxrwx    - aignas 18 May 12:00 jinja2 -> ../../../../../../+pip+dev_pip_311_jinja2_py3_none_any_85ece445/site-packages/jinja2
lrwxrwxrwx    - aignas 18 May 12:00 markdown_it -> ../../../../../../+pip+dev_pip_311_markdown_it_py_py3_none_any_35521684/site-packages/markdown_it
lrwxrwxrwx    - aignas 18 May 12:00 markupsafe -> ../../../../../../+pip+dev_pip_311_markupsafe_cp311_cp311_manylinux_2_17_x86_64_a123e330/site-packages/markupsafe
lrwxrwxrwx    - aignas 18 May 12:00 mdit_py_plugins -> ../../../../../../+pip+dev_pip_311_mdit_py_plugins_py3_none_any_0c673c3f/site-packages/mdit_py_plugins
lrwxrwxrwx    - aignas 18 May 12:00 mdurl -> ../../../../../../+pip+dev_pip_311_mdurl_py3_none_any_84008a41/site-packages/mdurl
lrwxrwxrwx    - aignas 18 May 12:00 myst_parser -> ../../../../../../+pip+dev_pip_311_myst_parser_py3_none_any_b9317997/site-packages/myst_parser
lrwxrwxrwx    - aignas 18 May 12:00 packaging -> ../../../../../../+pip+dev_pip_311_packaging_py3_none_any_29572ef2/site-packages/packaging
lrwxrwxrwx    - aignas 18 May 12:00 pygments -> ../../../../../../+pip+dev_pip_311_pygments_py3_none_any_9ea1544a/site-packages/pygments
lrwxrwxrwx    - aignas 18 May 12:00 readthedocs_ext -> ../../../../../../+pip+dev_pip_311_readthedocs_sphinx_ext_py2_none_any_f8c56184/site-packages/readthedocs_ext
lrwxrwxrwx    - aignas 18 May 12:00 requests -> ../../../../../../+pip+dev_pip_311_requests_py3_none_any_70761cfe/site-packages/requests
lrwxrwxrwx    - aignas 18 May 12:00 snowballstemmer -> ../../../../../../+pip+dev_pip_311_snowballstemmer_py2_none_any_c8e1716e/site-packages/snowballstemmer
lrwxrwxrwx    - aignas 18 May 12:00 sphinx -> ../../../../../../+pip+dev_pip_311_sphinx_py3_none_any_09719015/site-packages/sphinx
lrwxrwxrwx    - aignas 18 May 12:00 sphinx_reredirects -> ../../../../../../+pip+dev_pip_311_sphinx_reredirects_py3_none_any_efd50c76/site-packages/sphinx_reredirects
lrwxrwxrwx    - aignas 18 May 12:00 sphinx_rtd_theme -> ../../../../../../+pip+dev_pip_311_sphinx_rtd_theme_py2_none_any_422ccc75/site-packages/sphinx_rtd_theme
drwxr-xr-x    - aignas 18 May 12:00 sphinxcontrib
lrwxrwxrwx    - aignas 18 May 12:00 urllib3 -> ../../../../../../+pip+dev_pip_311_urllib3_py3_none_any_4e166650/site-packages/urllib3
lrwxrwxrwx    - aignas 18 May 12:00 yaml -> ../../../../../../+pip+dev_pip_311_pyyaml_cp311_cp311_manylinux_2_17_x86_64_3ad2a3de/site-packages/yaml

And I think the problem is that the typing-extensions package is just a single file inside site-packages, so it does not work because it is not included in the venv. The build still works if I run it without the flag.

As found in bazel-contrib#2882 testing, packages like `typing-extensions` which have `.py` files at the root of the `site-packages` folder don't work and it seems that the comment about `rules_python` being too eager is only half-correct. Since `namespace_pkgs` are no longer there, we can just include all of the files and if there are collisions, they will be highlighted as build errors. Now the following works: ``` bazel build //docs --@rules_python//python/config_settings:venvs_site_packages=yes ``` Stacked on bazel-contrib#2882. Work towards bazel-contrib#2156

With this PR I would like to facilitate the implementation of the venv layouts because we can in theory take the srcs and the data within the py_library and then use the expand_template to write the extra Python files if the namespace_pkgs flag is enabled. The old Python code has been removed and the extra generated files are written out with bazel_skylib write_file as a POC. This could be optimized by writing once and then doing symlinks to the original instead of writing the files, but I am not sure how this would behave, maybe fine? The implicit namespace_pkg init files are included to py_library if the site-packages config flag is set to false and I think this may help with continuing the implementation. NOTE, it seems that the pycross code that we have is using the namespace_pkg Python code, so that will be removed later in one PR. Work towards #2156

As found in bazel-contrib#2882 testing, packages like `typing-extensions` which have `.py` files at the root of the `site-packages` folder don't work and it seems that the comment about `rules_python` being too eager is only half-correct. Since `namespace_pkgs` are no longer there, we can just include all of the files and if there are collisions, they will be highlighted as build errors. Now the following works: ``` bazel build //docs --@rules_python//python/config_settings:venvs_site_packages=yes ``` Stacked on bazel-contrib#2882. Work towards bazel-contrib#2156

rickeylev · 2025-05-27T15:37:35Z

python/config_settings/BUILD.bazel

@@ -217,6 +217,15 @@ string_flag(
    visibility = ["//visibility:public"],
 )

+config_setting(
+    name = "is_venvs_site_packages",


Please use an internal name, or factor out a define_internal_flags() function (similar to what pypi).

rickeylev · 2025-05-27T15:38:47Z

python/private/pypi/namespace_pkg_tmpl.py

@@ -0,0 +1,2 @@
+# __path__ manipulation added by bazel-contrib/rules_python to support namespace pkgs.
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)


The name implies it's a template, but I don't see any template variable names?

rickeylev · 2025-05-27T15:40:13Z

python/private/pypi/namespace_pkgs.bzl

+
+load("@bazel_skylib//rules:copy_file.bzl", "copy_file")
+
+_ext = struct(


.dylib (mac) and .dll (windows) are additional extension module filename extensions

rickeylev · 2025-05-27T15:42:44Z

python/private/pypi/whl_installer/wheel_installer.py

-    Namespace packages can be created in one of three ways. They are detailed here:
-    https://packaging.python.org/guides/packaging-namespace-packages/#creating-a-namespace-package
-
-    'pkgutil-style namespace packages' (2) and 'pkg_resources-style namespace packages' (3) works in Bazel, but
-    'native namespace packages' (1) do not.
-
-    We ensure compatibility with Bazel of method 1 by converting them into method 2.


Please preserve this comment somewhere (namespace_pkgs.bzl ?). The link and why this legacy method is used were really helpful when I came across this code.

rickeylev · 2025-05-27T15:45:55Z

python/private/pypi/whl_library.bzl

@@ -389,6 +386,8 @@ def _whl_library_impl(rctx):
            metadata_name = metadata.name,
            metadata_version = metadata.version,
            requires_dist = metadata.requires_dist,
+            # TODO @aignas 2025-05-17: maybe have a build flag for this instead


Unless there's a commitment to resolve a TODO, leave them out.

(IMHO: no build flag needed)

Suggested change

# TODO @aignas 2025-05-17: maybe have a build flag for this instead

rickeylev · 2025-05-27T15:46:04Z

python/private/pypi/whl_library.bzl

@@ -457,6 +456,8 @@ def _whl_library_impl(rctx):
            name = whl_path.basename,
            dep_template = rctx.attr.dep_template or "@{}{{name}}//:{{target}}".format(rctx.attr.repo_prefix),
            entry_points = entry_points,
+            # TODO @aignas 2025-05-17: maybe have a build flag for this instead


Suggested change

# TODO @aignas 2025-05-17: maybe have a build flag for this instead

As found in #2882 testing, packages like `typing-extensions` which have `.py` files at the root of the `site-packages` folder don't work and it seems that the comment about `rules_python` being too eager is only half-correct. Since `namespace_pkgs` are no longer there, we can just include all of the files and if there are collisions, they will be highlighted as build errors. Now the following works: ``` bazel build //docs --@rules_python//python/config_settings:venvs_site_packages=yes ``` Work towards #2156

aignas requested review from rickeylev and groodt as code owners May 15, 2025 14:05

aignas mentioned this pull request May 15, 2025

Implement venv/site-packages based binaries #2156

Open

10 tasks

aignas changed the title ~~wip: reimplement namespace pkg filepath calc in starlark~~ refactor: reimplement writing namespace pkgs in Starlark May 17, 2025

aignas force-pushed the refactor/namespace-pkg-star branch 2 times, most recently from 54136c5 to 98827e5 Compare May 18, 2025 03:01

aignas force-pushed the refactor/namespace-pkg-star branch from 98827e5 to 4f1798b Compare May 18, 2025 03:20

aignas mentioned this pull request May 18, 2025

fix: symlink root-level python files to the venv #2908

Merged

aignas force-pushed the refactor/namespace-pkg-star branch from bb150a7 to 4ed19e0 Compare May 18, 2025 06:27

aignas enabled auto-merge May 18, 2025 06:28

aignas added 2 commits May 18, 2025 22:42

Merge branch 'main' into refactor/namespace-pkg-star

5a28b4b

Merge branch 'main' into refactor/namespace-pkg-star

70d13e8

rickeylev approved these changes May 27, 2025

View reviewed changes

aignas added this pull request to the merge queue May 27, 2025

Merged via the queue into main with commit dce5120 May 27, 2025
7 checks passed

aignas deleted the refactor/namespace-pkg-star branch May 28, 2025 14:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

refactor: reimplement writing namespace pkgs in Starlark #2882

refactor: reimplement writing namespace pkgs in Starlark #2882

Uh oh!

aignas commented May 15, 2025 •

edited

Loading

Uh oh!

rickeylev commented May 16, 2025

Uh oh!

aignas commented May 18, 2025

Uh oh!

rickeylev May 27, 2025

Uh oh!

rickeylev May 27, 2025

Uh oh!

rickeylev May 27, 2025

Uh oh!

rickeylev May 27, 2025

Uh oh!

rickeylev May 27, 2025

Uh oh!

rickeylev May 27, 2025

Uh oh!

Uh oh!

Uh oh!

		@@ -0,0 +1,2 @@
		# __path__ manipulation added by bazel-contrib/rules_python to support namespace pkgs.
		__path__ = __import__("pkgutil").extend_path(__path__, __name__)


		load("@bazel_skylib//rules:copy_file.bzl", "copy_file")

		_ext = struct(

Uh oh!

refactor: reimplement writing namespace pkgs in Starlark #2882

refactor: reimplement writing namespace pkgs in Starlark #2882

Uh oh!

Conversation

aignas commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rickeylev commented May 16, 2025

Uh oh!

aignas commented May 18, 2025

Uh oh!

rickeylev May 27, 2025

Choose a reason for hiding this comment

Uh oh!

rickeylev May 27, 2025

Choose a reason for hiding this comment

Uh oh!

rickeylev May 27, 2025

Choose a reason for hiding this comment

Uh oh!

rickeylev May 27, 2025

Choose a reason for hiding this comment

Uh oh!

rickeylev May 27, 2025

Choose a reason for hiding this comment

Uh oh!

rickeylev May 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aignas commented May 15, 2025 •

edited

Loading