[BUG] Parsing of __init__.py fails to reach _read_utf8_with_fallback at the end of file

[autogris]

When running Deluge 2.1.1.dev99 (deluge-torrent/deluge@7f3f7f6) with Setuptools 70.0.0, an error occurs claiming that _read_utf8_with_fallback is not defined in setuptools' init.py. By moving the function position to the head of the referred file, after the imports, the program runs normally. This error does not occur on Setuptools 69.x and earlier.

System: linux x64, kernel 6.9.0
Python: 3.11.9

Discussed in #4398

^{Originally posted by autogris May 24, 2024}
Hi. I'm trying to debug an error that emerged since upgrading setuptools to 70.0.0. When running Deluge, it gives me the following error:

Traceback (most recent call last):
  File "/usr/bin/deluge-gtk", line 33, in <module>
    sys.exit(load_entry_point('deluge==2.1.1.dev99', 'gui_scripts', 'deluge-gtk')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/__init__.py", line 63, in start
    Gtk().start()
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/__init__.py", line 44, in start
    from .gtkui import GtkUI
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/gtkui.py", line 61, in <module>
    from .pluginmanager import PluginManager
  File "/usr/lib/python3.11/site-packages/deluge/ui/gtk3/pluginmanager.py", line 12, in <module>
    import deluge.pluginmanagerbase
  File "/usr/lib/python3.11/site-packages/deluge/pluginmanagerbase.py", line 15, in <module>
    import pkg_resources
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3282, in <module>
    @_call_aside
     ^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3266, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 3295, in _initialize_master_working_set
    working_set = _declare_state('object', 'working_set', WorkingSet._build_master())
                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 589, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 926, in require
    needed = self.resolve(parse_requirements(requirements))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 787, in resolve
    dist = self._resolve_dist(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 816, in _resolve_dist
    env = Environment(self.entries)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1014, in __init__
    self.scan(search_path)
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1046, in scan
    for dist in find_distributions(item):
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2091, in find_on_path
    yield from factory(fullpath)
               ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2183, in resolve_egg_link
    return next(dist_groups, ())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2179, in <genexpr>
    resolved_paths = (
                     ^
  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 2167, in non_empty_lines
    for line in _read_utf8_with_fallback(path).splitlines():
                ^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name '_read_utf8_with_fallback' is not defined

I've tried to recompile Deluge, reinstall all python modules, to no success. The supposedly undefined function is in fact present at python3.11/site-packages/setuptools/unicode_utils.py . Is this a bug on setuptools side or Deluge? I'm on Python 3.11.9

[areebsiddiquidbz]

Also getting on M3 Pro

[abravalheri]

I believe that what is happening is a circular reference: the library calls pkg_resources to load an entry-point thar itself calls pkg_resources... (this is a hypothesis, I haven't checked).

We can avoid that by changing pkg_resources to define that method earlier in the file... However, the difficulty part here is to create a regression test based on a minimal reproducible example that does not depend on an external package.

Can anyone help to create such regression test to be added to the test suite to avoid similar problems in the future?

[abravalheri]

@autogris, @areebsiddiquidbz, how did you install deluge?
Is there any package in your environment that you have installed using the deprecated/not-to-be-used method python setup.py develop?

If that is the case, what happens if you uninstall that package and install it with the recommended method (pip install -e .)?

[abravalheri]

I tried a couple of tests with python setup.py develop and I could not replicate this problem...

diff --git i/pkg_resources/tests/test_working_set.py w/pkg_resources/tests/test_working_set.py
index 57f62b549..bc685bf0e 100644
--- i/pkg_resources/tests/test_working_set.py
+++ w/pkg_resources/tests/test_working_set.py
@@ -1,9 +1,14 @@
+import functools
 import inspect
+import os
 import re
+import shutil
+import subprocess
 import textwrap
-import functools
+import venv
 
 import pytest
+import jaraco.path
 
 import pkg_resources
 
@@ -499,3 +504,95 @@ def test_working_set_resolve(
             resolve_call()
     else:
         assert sorted(resolve_call()) == sorted(resolved_dists_or_exception)
+
+
+class TestInteropDeprecatedEgglink:
+    """The issue pypa/setuptools#4399 reports a problem when running
+    ``import pkg_resources`` and running ``WorkingSet._build_master`` in ``_call_aside``
+    and ``resolve_egg_link``.
+    """
+
+    @pytest.fixture
+    def venv_python(self, tmp_path):
+        venv.create(tmp_path / ".venv", clear=True, with_pip=False)
+        search_path = os.pathsep.join(map(str, tmp_path.glob(".venv/*/")))
+        return shutil.which("python", path=search_path)
+
+    @pytest.fixture
+    def site_packages(self, venv_python, tmp_path):
+        getsitepackages = (
+            "import os, site;"
+            "print(next(p for p in site.getsitepackages() if os.path.isdir(p)))"
+        )
+        return subprocess.check_output(
+            [venv_python, "-c", getsitepackages],
+            text=True,
+            encoding="utf-8",
+            cwd=tmp_path,
+        ).strip()
+
+    @pytest.fixture
+    def install_pkg_resources(self, venv_python, site_packages, tmp_path, request):
+        """Install this version of pkg_resources and setuptools"""
+        with open(os.path.join(site_packages, "this.pth"), "w", encoding="utf-8") as f:
+            f.write(f"{request.config.rootdir}\n")  # Optimistic UTF-8 encoding
+
+        subprocess.run(  # sanity check
+            [venv_python, "-Wignore", "-c", "import pkg_resources, setuptools"],
+            cwd=tmp_path,
+            check=True,
+        )
+
+    @pytest.fixture
+    def simple_pkg(self, tmp_path):
+        files = {
+            "pkg_root": {
+                "pkg": {
+                    "__init__.py": "def hello(): print('world')",
+                    "other.py": "import pkg_resources\ndef world(): print('hello')",
+                },
+                "setup.cfg": inspect.cleandoc(
+                    """
+                    [options.entry_points]
+                    console_scripts =
+                        pkg = pkg.other:world
+                    pkg.other =
+                        pkg = pkg.other:world
+                    """
+                ),
+            },
+        }
+        jaraco.path.build(files, prefix=tmp_path)
+        return tmp_path / "pkg_root"
+
+    @pytest.fixture
+    def egg_link(self, simple_pkg, venv_python, install_pkg_resources, tmp_path):
+        """Install ``simple_pkg`` using an egg-link, via setuptools' develop"""
+        _ = install_pkg_resources  # install setuptools
+        cmd = "__import__('setuptools').setup()"
+        subprocess.run(
+            [venv_python, "-Wignore", "-c", cmd, "develop"], cwd=simple_pkg, check=True
+        )
+        return next(tmp_path.rglob("*.egg-link"))
+
+    def test_global_working_set(
+        self, install_pkg_resources, egg_link, venv_python, tmp_path
+    ):
+        _ = install_pkg_resources
+        assert "/.venv/" in str(egg_link).replace(os.sep, "/")
+        assert egg_link.name == "pkg.egg-link"
+        assert egg_link.exists()
+
+        cmd = "print('hello'); import pkg; pkg.hello()"
+        out = subprocess.check_output(
+            [venv_python, "-Wignore", "-c", cmd],
+            text=True,
+            encoding="utf-8",
+            cwd=tmp_path,
+        )
+        print(out)
+        assert out.strip() == "hello\nworld"
+
+        exe = shutil.which("pkg", path=os.path.dirname(venv_python))
+        out = subprocess.check_output(exe, text=True, encoding="utf-8", cwd=tmp_path)
+        assert out.strip() == "hello"

========================================= test session starts ==========================================
platform linux -- Python 3.8.10, pytest-8.2.1, pluggy-1.5.0
cachedir: .tox/py/.pytest_cache
rootdir: /home/abravalheri/workspace/setuptools
configfile: pytest.ini
plugins: checkdocs-2.13.0, enabler-3.1.1, home-0.5.1, mypy-0.10.3, ruff-0.3.2, subprocess-1.5.0, timeout-2.3.1, xdist-3.6.1, typeguard-4.2.1
16 workers [41 items]
.........................................                                                        [100%]
================================================= mypy =================================================
Success: no issues found in 1 source file
========================================= 41 passed in 15.67s ==========================================
  py: OK (26.80=setup[8.93]+cmd[17.87] seconds)
  congratulations :) (27.49 seconds)

I still need help to create a minimal reproducer for this, so we can have a test for this in the test suite.

[autogris]

I don't think I ever installed anything with python setup.py develop, but I'm on Slackware, everything is installed either by official packages, or through scripts of para-official repositories (slackbuild, ponce sbo) producing packages, or pip run as root.

The script for Deluge produces a package using this installation method, which then is searched for binaries and stripped with strip --unneeded:
python3 setup.py install --root=$PKG

[autogris]

Ok, something I noticed in the traceback is that the last called .py from Deluge side is pluginmanagerbase.py. I do have plugins installed in the .egg format, which I guess are loaded before the window is draw. I will try later to uninstall all third party plugins, and report what happens. @areebsiddiquidbz, are you using third party plugins too?

Quickly checking here, I can see in the source code that the oldest plugin hasn't been updated for many years and it contains this absolute_import line, which seems suspect (https://github.com/stefantalpalaru/deluge-default-trackers/blob/master/setup.py):

from __future__ import absolute_import
from setuptools import setup

The only other third party plugin is this https://github.com/ratanakvlun/deluge-ltconfig/blob/master/setup.py

[autogris]

I've removed the plugins and also tried to recreate the eggs with python3 setup.py bdist_egg, both experiments resulted in the same original error.

[abravalheri]

I've removed the plugins and also tried to recreate the eggs with python3 setup.py bdist_egg, both experiments resulted in the same original error.

I also tried to create a minimal reproducer using bdist_egg and then write a test case using it but still everything seems to run fine.

This means that the minimal reproducible example that I am trying to write still needs more details, and I am not managing to get it right. Specifically, I did not manage to find the condition that triggers the circular references...

I would appreciate any help writing such minimal reproducer.

(my work on the reproducer so far)

diff --git c/setuptools/tests/environment.py w/setuptools/tests/environment.py
index b9de4fda6..cb1f850d8 100644
--- c/setuptools/tests/environment.py
+++ w/setuptools/tests/environment.py
@@ -1,3 +1,4 @@
+import functools
 import os
 import sys
 import subprocess
@@ -5,6 +6,7 @@ import unicodedata
 from subprocess import Popen as _Popen, PIPE as _PIPE
 
 import jaraco.envs
+from path import Path
 
 
 class VirtualEnv(jaraco.envs.VirtualEnv):
@@ -33,6 +35,16 @@ class VirtualEnv(jaraco.envs.VirtualEnv):
             kwargs["env"] = env
         return subprocess.check_output(cmd, *args, **kwargs)
 
+    @functools.cached_property
+    def site_packages(self):
+        prefix = os.path.abspath(self.dir)
+        getsitepackages = (
+            "import os, site;"
+            "opts = site.getsitepackages();"
+            f"print(next(p for p in opts if os.path.isdir(p) and {prefix!r} in p))"
+        )
+        return Path(self.run(["python", "-I", "-c", getsitepackages]).strip())
+
 
 def _which_dirs(cmd):
     result = set()
diff --git c/setuptools/tests/test_bdist_egg.py w/setuptools/tests/test_bdist_egg.py
index 12ed4d328..cf8499436 100644
--- c/setuptools/tests/test_bdist_egg.py
+++ w/setuptools/tests/test_bdist_egg.py
@@ -1,10 +1,12 @@
 """develop tests"""
 
+import inspect
 import os
 import re
 import zipfile
 
 import pytest
+import jaraco.path
 
 from setuptools.dist import Distribution
 
@@ -67,3 +69,77 @@ class Test:
         names = list(zi.filename for zi in zip.filelist)
         assert 'hi.pyc' in names
         assert 'hi.py' not in names
+
+
+class TestInteropPkgResources:
+    """The issue pypa/setuptools#4399 reports a problem when running
+    ``import pkg_resources`` and running ``WorkingSet._build_master`` in ``_call_aside``
+    and ``resolve_egg_link``.
+    """
+
+    @pytest.fixture
+    def simple_pkg(self, tmp_path):
+        files = {
+            "pkg_root": {
+                "pkg": {
+                    "__init__.py": "def hello(): print('world')",
+                    "other.py": "import pkg_resources\ndef world(): print('hello')",
+                },
+                "setup.cfg": inspect.cleandoc(
+                    """
+                    [options.entry_points]
+                    console_scripts =
+                        pkg = pkg.other:world
+                    pkg.other =
+                        pkg = pkg.other:world
+                    """
+                ),
+            },
+        }
+        jaraco.path.build(files, prefix=tmp_path)
+        return tmp_path / "pkg_root"
+
+    @pytest.fixture
+    def egg(self, simple_pkg, venv):
+        """Install ``simple_pkg`` using an egg"""
+        cmd = ["python", "-Wignore", "-c", "__import__('setuptools').setup()"]
+        venv.run([*cmd, "bdist_egg"], cwd=simple_pkg)
+        file = next(simple_pkg.glob("dist/*.egg"))
+        venv.run([*cmd, "easy_install", "--prefix", venv.dir, file], cwd=simple_pkg)
+        return venv.site_packages.glob("*.egg")[0]
+
+    PY_C = ["python", "-I", "-Wignore", "-c"]
+
+    def test_installed_egg_sanity_check(self, egg, venv):
+        assert f"{venv.site_packages}/" in str(egg).replace(os.sep, "/")
+        assert egg.exists()
+        out = venv.run([*self.PY_C, "import pkg; print(pkg.__path__)"]).strip()
+        assert f"{venv.site_packages}/" in out
+        assert ".egg" in out
+
+    @pytest.mark.parametrize(
+        "cmd, output",
+        [
+            ([*PY_C, "import pkg; pkg.hello()"], "world"),
+            ([*PY_C, "import pkg_resources;from pkg.other import *;world()"], "hello"),
+            (["pkg"], "hello"),
+            (
+                [
+                    *PY_C,
+                    inspect.cleandoc(
+                        """
+                        import pkg_resources
+                        ep = next(pkg_resources.iter_entry_points("pkg.other", "pkg"))
+                        fn = ep.load()
+                        fn()
+                        """
+                    ),
+                ],
+                "hello",
+            ),
+        ],
+    )
+    def test_global_working_set(self, egg, venv, cmd, output):
+        out = venv.run(cmd).strip()
+        print(out)
+        assert out == output

pipx run --python python3.11 tox -- -p no:cov -p no:perf -p no:mypy -x -n 4 setuptools/tests/test_bdist_egg.py
# ...
16 workers [9 items]
.........                                                                                        [100%]
========================================== 9 passed in 24.95s ==========================================
  py: OK (30.36=setup[3.15]+cmd[27.21] seconds)
  congratulations :) (31.68 seconds)

[autogris]

Sorry, I'm not really familiar with python. I could try to inspect Deluge code and see if I can identify where the thing is triggered, but I'm not sure, it seems fairly complex to my limited knowledge. I haven't seen this problem in any other python application, so maybe it is really specific to how Deluge call this library. It does not seem to be related to the .egg plugins since I've tried to run with all plugins removed and still got the same error.

[mcayanan]

Hi. I am seeing this same name '_read_utf8_with_fallback' is not defined on my end. Is there an ETA of when this would get fixed?

[abravalheri]

Hi @mcayanan , please see the discussion above that describes what is missing for us to act on this particular problem.

A way to expedite the fix would be to provide a small minimal reproducer example that do not rely on complex and existing packages so that it can help us to understand the real root cause of the problem. Are you interested in contributing with one?

[mcayanan]

Hi @abravalheri . I myself am still trying to understand root cause. I will be out of the office starting on 6/6 and will return on 6/17, so I can revisit this at that time to investigate more and provide feedback if I'm able to easily reproduce it.