Unexpected behavior when distutils is invoked before Setuptools

[webknjaz]

TL;DR We are forced to use distutils because setuptools has broken symlink processing and this causes distutils.errors.DistutilsClassError: command class <class '__main__.SDistCommand'> must subclass Command.

It works with setuptools<48 and the changelog doesn't document any breaking behaviors for this version.

Repro:

$ git clone https://github.com/ansible/ansible.git
$ cd ansible
$ pip install -U 'setuptools>=48'
$ python setup.py sdist

(tried under Python 3.8)

Ref: ansible/ansible#70456

[jaraco]

Thanks for the report.

Setuptools 49.1 is out, should address the issue for early adopters while I triage and correct the issue.

[jaraco]

I'm able to replicate the failure:

ansible devel $ pip-run -q setuptools==48 -- setup.py sdist
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/distutils_patch.py:17: UserWarning: Setuptools is replacing distutils
  warnings.warn("Setuptools is replacing distutils")
Traceback (most recent call last):
  File "setup.py", line 338, in <module>
    main()
  File "setup.py", line 333, in main
    setup(**setup_params)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/__init__.py", line 164, in setup
    return distutils.core.setup(**attrs)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/_distutils/core.py", line 134, in setup
    ok = dist.parse_command_line()
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/_distutils/dist.py", line 484, in parse_command_line
    args = self._parse_command_opts(parser, args)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/dist.py", line 925, in _parse_command_opts
    nargs = _Distribution._parse_command_opts(self, parser, args)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/_distutils/dist.py", line 547, in _parse_command_opts
    raise DistutilsClassError(
distutils.errors.DistutilsClassError: command class <class '__main__.SDistCommand'> must subclass Command

Note the warning "Setuptools is replacing distutils".

That was meant to be a hard failure, except that PyPy somehow ends up with distutils unconditionally imported during interpreter startup.

[jaraco]

I'm able to work around the issue by ensuring that setuptools is imported prior to running the command:

$ pip-run -q setuptools==48 -- -c "import setuptools; exec(open('setup.py').read())" sdist
...
hard linking test/units/vars/test_variable_manager.py -> ansible-base-2.11.0.dev0/test/units/vars
creating dist
Creating tar archive
removing 'ansible-base-2.11.0.dev0' (and everything under it)
<string>:180: RuntimeWarning: When setup.py sdist is run from outside of the Makefile, the generated tarball may be incomplete.  Use `make snapshot` to create a tarball from an arbitrary checkout or use `cd packaging/release && make release version=[..]` for official builds.

[jaraco]

Also using PEP 517:

ansible devel $ cat > pyproject.toml
[build-system]
requires = ['setuptools', 'wheel']
build-backend = 'setuptools.build_meta'

ansible devel $ pip-run -q pep517 -- -m pep517.build -s .
running egg_info
writing lib/ansible_base.egg-info/PKG-INFO
writing dependency_links to lib/ansible_base.egg-info/dependency_links.txt
writing requirements to lib/ansible_base.egg-info/requires.txt
writing top-level names to lib/ansible_base.egg-info/top_level.txt
reading manifest file 'lib/ansible_base.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'docs/docsite/rst_warnings'
warning: no previously-included files matching '*' found under directory 'docs/docsite/_build'
warning: no previously-included files matching '*.pyc' found under directory 'docs/docsite/_extensions'
warning: no previously-included files matching '*.pyo' found under directory 'docs/docsite/_extensions'
warning: no files found matching '*.ps1' under directory 'lib/ansible/modules/windows'
warning: no files found matching '*.psm1' under directory 'test/support'
writing manifest file 'lib/ansible_base.egg-info/SOURCES.txt'
running sdist
...
Creating tar archive
removing 'ansible-base-2.11.0.dev0' (and everything under it)
setup.py:180: RuntimeWarning: When setup.py sdist is run from outside of the Makefile, the generated tarball may be incomplete.  Use `make snapshot` to create a tarball from an arbitrary checkout or use `cd packaging/release && make release version=[..]` for official builds.
  warnings.warn('When setup.py sdist is run from outside of the Makefile,'

[jaraco]

It also works with this diff on ansible:

diff --git a/setup.py b/setup.py
index 0398473d5e..7c86075495 100644
--- a/setup.py
+++ b/setup.py
@@ -9,8 +9,6 @@ import sys
 import warnings
 
 from collections import defaultdict
-from distutils.command.build_scripts import build_scripts as BuildScripts
-from distutils.command.sdist import sdist as SDist
 
 try:
     from setuptools import setup, find_packages
@@ -23,6 +21,9 @@ except ImportError:
           " install setuptools).", file=sys.stderr)
     sys.exit(1)
 
+from distutils.command.build_scripts import build_scripts as BuildScripts
+from distutils.command.sdist import sdist as SDist
+
 sys.path.insert(0, os.path.abspath('lib'))
 from ansible.release import __version__, __author__

The underlying issue is that with setuptools 48, you cannot import distutils before setuptools.

My recommendation is to (a) move the imports so distutils imports come after setuptools (quick, painless, compatible), then (b) adopt PEP 517 and use pep517 to build your sdists.

I'll update the changelog to make this change more clear.

[jaraco]

In the referenced commit, I've updated the changelog to clarify the weaknesses identified from this bug report and guide users to the best practices. Can you confirm that this guidance and applying the recommended changes in Ansible addresses the concern?