Regression in installer - Directory not empty

[jaraco]

Seems like this may be failing to catch something, causing later failures in install_as_egg

I 05-14 20:55:45 executor.py:331]       Traceback (most recent call last):
I 05-14 20:55:45 executor.py:331]         File "<string>", line 2, in <module>
I 05-14 20:55:45 executor.py:331]         File "<pip-setuptools-caller>", line 35, in <module>
I 05-14 20:55:45 executor.py:331]         File "/tmp/tmpuzviy574/setup.py", line 154, in <module>
I 05-14 20:55:45 executor.py:331]           setuptools.setup(
I 05-14 20:55:45 executor.py:331]         File "/home/runner/test-env/lib/python3.10/site-packages/setuptools/__init__.py", line 114, in setup
I 05-14 20:55:45 executor.py:331]           _install_setup_requires(attrs)
I 05-14 20:55:45 executor.py:331]         File "/home/runner/test-env/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in _install_setup_requires
I 05-14 20:55:45 executor.py:331]           _fetch_build_eggs(dist)
I 05-14 20:55:45 executor.py:331]         File "/home/runner/test-env/lib/python3.10/site-packages/setuptools/__init__.py", line 92, in _fetch_build_eggs
I 05-14 20:55:45 executor.py:331]           dist.fetch_build_eggs(dist.setup_requires)
I 05-14 20:55:45 executor.py:331]         File "/home/runner/test-env/lib/python3.10/site-packages/setuptools/dist.py", line 766, in fetch_build_eggs
I 05-14 20:55:45 executor.py:331]           return _fetch_build_eggs(self, requires)
I 05-14 20:55:45 executor.py:331]         File "/home/runner/test-env/lib/python3.10/site-packages/setuptools/installer.py", line 45, in _fetch_build_eggs
I 05-14 20:55:45 executor.py:331]           resolved_dists = [_fetch_build_egg_no_warn(dist, req) for req in needed_reqs]
I 05-14 20:55:45 executor.py:331]         File "/home/runner/test-env/lib/python3.10/site-packages/setuptools/installer.py", line 45, in <listcomp>
I 05-14 20:55:45 executor.py:331]           resolved_dists = [_fetch_build_egg_no_warn(dist, req) for req in needed_reqs]
I 05-14 20:55:45 executor.py:331]         File "/home/runner/test-env/lib/python3.10/site-packages/setuptools/installer.py", line 119, in _fetch_build_egg_no_warn
I 05-14 20:55:45 executor.py:331]           wheel.install_as_egg(dist_location)
I 05-14 20:55:45 executor.py:331]         File "/home/runner/test-env/lib/python3.10/site-packages/setuptools/wheel.py", line 122, in install_as_egg
I 05-14 20:55:45 executor.py:331]           self._install_as_egg(destination_eggdir, zf)
I 05-14 20:55:45 executor.py:331]         File "/home/runner/test-env/lib/python3.10/site-packages/setuptools/wheel.py", line 130, in _install_as_egg
I 05-14 20:55:45 executor.py:331]           self._convert_metadata(zf, destination_eggdir, dist_info, egg_info)
I 05-14 20:55:45 executor.py:331]         File "/home/runner/test-env/lib/python3.10/site-packages/setuptools/wheel.py", line 175, in _convert_metadata
I 05-14 20:55:45 executor.py:331]           os.rename(dist_info, egg_info)
I 05-14 20:55:45 executor.py:331]       OSError: [Errno 39] Directory not empty: '/tmp/tmpuzviy574/.eggs/wheel-0.45.1-py3.10.egg/wheel-0.45.1.dist-info' -> '/tmp/tmpuzviy574/.eggs/wheel-0.45.1-py3.10.egg/EGG-INFO'

source: https://github.com/skypilot-org/skypilot/actions/runs/15027265800/job/42243638678 (passed on setuptools 80.4.0)

Originally posted by @cg505 in #4997 (comment)

[mnaser]

We're seeing the same thing in the OpenStack project. We noticed reverting 7379eaa fixes this issue.

I noticed that running python setup.py egg_info the first time around works but the second time causes that issue as a data point.

[jaraco]

Thanks for the report. I've yanked v80.7.0 for now while I investigate. I'll explore the example to see if I can deduce what's happening.

[cg505]

Thanks, @jaraco. Sorry for the limited info. In this test, the project is calling pip wheel on itself. I will try to create a more minimal reproduction case.

[cg505]

Confirmed that the issue is no longer present for us on 80.6.0: https://github.com/skypilot-org/skypilot/actions/runs/15027265800/job/42250193487

[cg505]

Quick repro:

git clone https://github.com/skypilot-org/skypilot
cd skypilot
rm pyproject.toml
pip wheel ./ --no-deps

This works on 80.6.0 but fails on 80.7.0. The missing pyproject.toml seems key.

[jaraco]

cg505 - that's super-helpful. Although I couldn't reproduce it with pip wheel (not sure why), I was able to reproduce it by running python setup.py egg_info; python setup.py egg_info on that project, so maybe some version of pip is doing something like that.

After some trial and error, I think I understand what's going on.

Prior to this change, if wheel (or some other setup_requires dependency) was declared but found in the existing environment, it would be used... and since setuptools vendors wheel or other environments aggressively installed it, it would not need to be installed. With the new logic, the installation would happen irrespective of the presence of that package, invoking the installer in scenarios where it wouldn't be invoked before.

So that explains why skypilot was impacted.

I'm less sure about the openstack project or other projects that might have 'setup_requires' for something other than an already satisfied requirement. The code that actuates the install, where the failure occurs, hasn't changed, which makes me think that maybe this installer code has been broken for a very long time, and users have merely worked around it by using pep 517 or ensuring the dependencies were present. If that's the case, this deprecated installer code can probably just be removed and no one will mind.

[jaraco]

I did a quick search of the issues and there aren't any solid hits on "directory not empty" or "installer.py" that seem relevant to this issue. I think it's possible that setup_requires for any not-already-satisfied requirement may have been broken for a long time.

I plan to add a workaround for the known use-case with wheel in skypilot and hope that resolves the issue for everyone else.

[mnaser]

@jaraco in our case, OpenStack projects use pbr and I noticed projects that don't use pbr were fine, but those which did are affected.

You can test against https://github.com/openstack/requirements by running the egg_info command.

[jaraco]

That's a great example. With openstack/requirements, I'm able to replicate the issue even after applying the supposed fix. That gives me another variation to chase.

[jaraco]

Okay. I found a second issue. In the discovery of already downloaded (cached) eggs, I was invoking importlib metadata wrong and not matching on any extant eggs.

7379eaa#diff-bd39b324740fc3a42f04dfbfd0046c2af101e44f597fc4b62314fa056a781072R88-R92

As a result, that causes the installer to run and install over an existing egg, and that's what triggers the novel directory not empty error. The fix was rather simple.

I'm fairly confident about it as well.