Setup_requires of dependencies misses private index

[usmcamp0811]

I work almost entirely in an offline environment where we have our own private Pypi server setup. Whenever I attempt to setup a new Python environment I run into an issue that some packages won't install correctly with pip, at least not the first time. If I manually pip install fastdiff I can then pip install pytest-runner and things work. There are other packages but this is the first one that I could nail down as being the cause of my woes.

I found that if edit /setuptools/command/easy_install.py

FROM:

...
        if not self.editable:
            self.check_site_dir()
        self.index_url = self.index_url or "https://pypi.org/simple/"
        self.shadow_path = self.all_site_dirs[:]
        for path_item in self.install_dir, normalize_path(self.script_dir):
            if path_item not in self.shadow_path:
                self.shadow_path.insert(0, path_item)
...

TO:

...
        if not self.editable:
            self.check_site_dir()
        self.index_url = self.index_url or "http://my-private.pypi/simple/"
        self.shadow_path = self.all_site_dirs[:]
        for path_item in self.install_dir, normalize_path(self.script_dir):
            if path_item not in self.shadow_path:
                self.shadow_path.insert(0, path_item)
...

I can do pip install fastdiff and the package will install correctly without the need to pip install pytest-runner.

This is a huge problem when trying to script out the build of an environment either in a Docker image or in a CI script things will just fail for no clear reason. This is due to my pyproject.toml requiring not fastdiff but rather snapshottest. Trying to track down every one of the problem packages in a build is very time consuming and having to then include a pip install X before doing poetry install or pip install -r requirments.txt is a giant pain.

So I purpose we maybe replace the hard coding of https://pypi.org/simple/ to an environment variable that maybe has a default of https://pypi.org/simple/ that way if you operate in an offline environment you can set the environment variable to your offline pypi repo and not have this headache.

[jaraco]

Part of the problem here is that easy_install is deprecated, but in some cases, setup_requires dependencies still rely on it.

However, in late versions of Setuptools (42 and later), setup_requires dependencies are loaded using pip (if available) and pass through any custom indexes.

So the question is - why is pip not being used to install the setup_requires dependencies? What version of Setuptools do you have installed?

[usmcamp0811]

@jaraco I think you're right. I just checked the with python -m easy_install --version and have version 41.2.0 installed by default in the poetry virtual environment. Doing poetry run pip install --upgrade setuptools brought be up to version 50.3.0 and I didn't have any problem. My system setuptools is at 46.4.0 so I'm not sure where the poetry environment is getting the older version. Do you have any insights into where virtual environments get their setuptools ? Just trying to think where to go next to try and correct this in a more general sense. For now upgrading setuptools is not a horrible solution but it sure would be nice if the venv got the correct version from the get go.

Thank you for the help!

EDIT: might not be fixed.. but it could be another thing causing it... not sure yet

[jaraco]

Do you have any insights into where virtual environments get their setuptools ?

I've had some experience. It depends on what tool is creating the environments, as there is virtualenv and venv, plus each has its own version history. I believe both of those bundle versions of pip, setuptools, and maybe wheel, and those bundled versions will depend on the version of virtualenv or Python installed.

I'm not familiar with poetry, so I'd inquire with that project how it creates virtualenvs... and then check the venv or virtualenv version based on that.

[jaraco]

This issue has already been fixed, so I'm closing the issue. The remaining concern is with how Setuptools can be updated, which is a question to the environment maintainer or poetry/virtualenv tools. Please reach out to those teams or to pypa/packaging-problems if there's a more meta problem that needs addressing.

Please also be aware than in #2823, I've kicked off the effort to remove the setup_requires functionality. I'd recommend looking into pyproject.toml and PEP 517/518 for builds. See jaraco/skeleton for an example.