setup_requires and install_requires don't play nicely together

Originally reported by: msabramo (Bitbucket: msabramo, GitHub: msabramo)

---

What's the latest thinking on how to deal with a package that has setup_requires=['foo'] and install_requires=['foo']?

My observation is that this causes setuptools to download a foo.egg file to the current directory, install_requires then decides it doesn't need to install the package, and then the software craps out when it tries to import it because it can't import from the egg :-(

For example, I work on a project that uses pbr. The setup.py looks like this:

import setuptools

setuptools.setup(
    setup_requires=['pbr'],
    install_requires=['pbr'],
    pbr=True)

Now starting with a clean virtualenv and no .egg files in the current directory, here's what I see (with pip 1.5.5 and setuptools 3.4.4):

$ python setup.py install

Installed /Users/marca/dev/git-repos/jenkins-job-builder/pbr-0.8.0-py2.7.egg
running install
...
Requirement already satisfied (use --upgrade to upgrade): pbr>=0.5.21,<1.0 in ./pbr-0.8.0-py2.7.egg
...

$ ls -ld *.egg
drwxr-xr-x  4 marca  staff  136 May 23 08:56 pbr-0.8.0-py2.7.egg/

OK, fine. pbr wasn't available in the virtualenv so setuptools downloaded a .egg so that it could use it to do setupy things.

Not so fine is that because the .egg file was already present because it was downloaded in the setup_requires phase, the setuptools install_requires logic decided that it didn't have to install pbr in the virtualenv (I guess at this point the .egg file is on the PYTHONPATH so it can be imported?). This is not good because code outside of setuptools doesn't know how to do anything with that .egg:

$ python -c 'import jenkins_jobs.version'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "jenkins_jobs/version.py", line 18, in <module>
    from pbr.version import VersionInfo
ImportError: No module named pbr.version

So this is not good.

Ideas:

1. Maybe setuptools puts the .egg file on the PYTHONPATH so that it can use it while creating metadata and such. Maybe when it's done, it can remove the .egg from the PYTHONPATH so that the install_requires code doesn't see the package as importable and then it would install the package in the virtualenv? A variation on the idea is to leave the PYTHONPATH but delete the .egg file (or download the egg into a temp directory that gets cleaned up as soon as it's not needed). Drawback here is that we download the package twice -- once for setup_requires and once for install_requires.
2. Have a special case in the setup_requires logic so that if install_requires of the same dependency is present, instead of downloading the .egg, it simply installs the package in a more "permanent" place on the PYTHONPATH (site-packages)?

@jaraco: What do you think?

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/209

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):

---

This seems important to projects that use pbr, so want to add a few folks who I think have an interest in pbr:

• @Mordred
• @dhellmann

Original comment by mordred (Bitbucket: mordred, GitHub: mordred):

---

You'll notice that if you do "pip install ." instead of "python setup.py install" you get the behavior you expect, which is that the depend that setuptools temporarily installs locally for the purposes for fulfilling the setup_requires does not prevent the install_requires from firing properly.

I kinda like 2 - except that it actually makes a behavioral change that could have knockon effects. Currently, other than setup_requires, in a pip installation, everything that gets listed in install_requires actually gets installed by pip and not by setuptools. This is important for situations where easy_install behaves poorly, like proxy-blocked places.

1 preseves the separation, but as you said means a double download. I wonder if easy_install could download the egg to a known location and then pass that info along to install_requires invocations so that install_requires could do the installation of the setup_requires downloaded thing. Then, if we wanted to get really clever, we could have that location be settable, so that in a pip-driven install, pip could request of setuptools that it puts downoaded setup_requires artifacts in $DIR, and then pip could consume them.

@dstufft does that sound extra crazy?

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

---

Well 2 won't actually work because setup_requires runs before install_requires has been interpreted. That's the entire point of setup_requires. Otherwise pbr could never work because setuptools would have already interpreted the install_requires before it installed and loaded pbr.

Pip can't install from a .egg and setuptools prefers them. Trying to reuse artifacts is only going to work in certain situations and it honestly sounds like more work than it's worth. Especially since in the future I plan on stealing setup_requires from setuptools and using pip to install them into a temporary directory instead of curdir.

That being said not deleting the setup_requires files is an optimization to remove the need to download it every time you run setup.py. Probably it would make the most sense to just have a .setup-cache or something in curdir which setuptools will add to the PYTHONPATH during setup.py execution but which won't show up outside of that. It also would prevent setuptools from dropping little eggs all over the curdir and make it easier to clean them up.

Original comment by tsmithe (Bitbucket: tsmithe, GitHub: tsmithe):

---

I'd just like to record my experiences here, as I reported to distutils-sig: "I've got an extension that I build with distutils. It requires numpy both to build and to run, so I have numpy in both setup_requires and install_requires. Yet setup.py builds numpy twice -- once for the build stage, and then again on installation. This seems inefficient to me -- why not just build it once?"

This seems to be a slightly different problem to the egg issue discussed by Marc, though Donald's proposed caching solution would seem to fix both.

Original comment by dhellmann (Bitbucket: dhellmann, GitHub: dhellmann):

---

I like dstufft's idea of a cache dir, but does that fix the problem that the package needs to be installed on the real path?

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):

---

@dstufft's cache idea sounds nice to me. But it sounds more like a nicety to keep the current directory looking clean (which would be nice!). I was wondering if an even smaller change to fix the problem at hand is to simply make the egg part of the PYTHONPATH only during the setup_requires phase and then remove it from the PYTHONPATH immediately afterwards (maybe using a context manager?) That way the egg doesn't get looked at during install_requires. Make sense?

Original comment by mordred (Bitbucket: mordred, GitHub: mordred):

---

That sounds actually like the 'right' thing todo - since normally that
egg would not be in the path - so the setup_requires path manip seems
like a leak, rather than a feature.

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):

---

This is extremely simplistic and thus most probably wrong when considering a library with the history and complexity of setuptools, but perhaps it will simply move things a long bit by starting with some code to discuss:

--- a/setuptools/dist.py
+++ b/setuptools/dist.py
@@ -3,6 +3,7 @@ __all__ = ['Distribution']
 import re
 import os
 import sys
+import tempfile
 import warnings
 import distutils.log
 import distutils.core
@@ -326,7 +327,7 @@ class Distribution(_Distribution):
                     links = opts['find_links'][1].split() + links
                 opts['find_links'] = ('setup', links)
             cmd = easy_install(
-                dist, args=["x"], install_dir=os.curdir, exclude_scripts=True,
+                dist, args=["x"], install_dir=tempfile.gettempdir(), exclude_scripts=True,
                 always_copy=False, build_directory=None, editable=False,
                 upgrade=False, multi_version=True, no_report=True, user=False
             )

It seems to work for this particular case that I tested:

 marca@marca-mac2.local ⮀ git-repos/jenkins-job-builder ⮀ ⭠ tox_report_coverage s? ⮀ venv ⮀
 ❯ pip uninstall -y pbr; rm -rf pbr-0.8.0-py2.7.egg; python setup.py install
Uninstalling pbr:
  Successfully uninstalled pbr
running install
Requirement already satisfied (use --upgrade to upgrade): PyYAML in ./venv/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): python-jenkins in ./venv/lib/python2.7/site-packages
Downloading/unpacking pbr>=0.5.21,<1.0
  Using download cache from /Users/marca/.pip/download-cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpbr%2Fpbr-0.8.2.tar.gz
  Running setup.py (path:/Users/marca/dev/git-repos/jenkins-job-builder/venv/build/pbr/setup.py) egg_info for package pbr
    [pbr] Processing SOURCES.txt
    warning: LocalManifestMaker: standard file '-c' not found

    [pbr] In git context, generating filelist from git
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Requirement already satisfied (use --upgrade to upgrade): pip in ./venv/lib/python2.7/site-packages (from pbr>=0.5.21,<1.0)
Installing collected packages: pbr
  Running setup.py install for pbr
    [pbr] Reusing existing SOURCES.txt
Successfully installed pbr
...

 marca@marca-mac2.local ⮀ git-repos/jenkins-job-builder ⮀ ⭠ tox_report_coverage s? ⮀ venv ⮀
 ❯ python -c 'import jenkins_jobs.version; print(jenkins_jobs.version.version_info)'
0.8.0.5

Original comment by dpursehouse (Bitbucket: dpursehouse, GitHub: dpursehouse):

---

I'm experiencing a similar problem; I'm not sure if it's exactly the same one as described in this issue.

I'm installing jenkins-job-builder in a virtualenv, from a requirements.txt file that also specifies pbr:

  pbr>=0.5.21,<1.0
  jenkins-job-builder==0.8.0

The PIP_INDEX_URL environment variable specifies a server on an internal network, and there is no access to the public internet (i.e. can't get to pypi.python.org).

During the installation, pbr is successfully installed in the virtualenv from the internal package server, but then when jjb is downloaded it tries to install pbr again, but from the external index server.

 Downloading/unpacking pbr>=0.5.21,<1.0 (from -r ../MYPROJECT/etc/requirements.txt (line 1))
  http://pypi.MYSITE.net/simple/pbr/ uses an insecure transport scheme (http). Consider using https if pypi.MYSITE.net has it available
  Running setup.py (path:/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/build/pbr/setup.py) egg_info for package pbr
    [pbr] Processing SOURCES.txt
    warning: LocalManifestMaker: standard file '-c' not found

    [pbr] In git context, generating filelist from git
    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
 Downloading/unpacking jenkins-job-builder==0.8.0 (from -r ../MYPROJECT/etc/requirements.txt (line 2))
  http://pypi.MYSITE.net/simple/jenkins-job-builder/0.8.0 uses an insecure transport scheme (http). Consider using https if pypi.MYSITE.net has it available
  http://pypi.MYSITE.net/simple/jenkins-job-builder/ uses an insecure transport scheme (http). Consider using https if pypi.MYSITE.net has it available
  Running setup.py (path:/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/build/jenkins-job-builder/setup.py) egg_info for package jenkins-job-builder
    Download error on https://pypi.python.org/simple/pbr/: [Errno 110] Connection timed out -- Some packages may not be found!
    Couldn't find index page for 'pbr' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [Errno 110] Connection timed out -- Some packages may not be found!
    No local packages or download links found for pbr
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/build/jenkins-job-builder/setup.py", line 19, in <module>
        pbr=True)
      File "/usr/lib/python2.7/distutils/core.py", line 112, in setup
        _setup_distribution = dist = klass(attrs)
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/setuptools/dist.py", line 239, in __init__
        self.fetch_build_eggs(attrs.pop('setup_requires'))
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/setuptools/dist.py", line 264, in fetch_build_eggs
        replace_conflicting=True
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/pkg_resources.py", line 580, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/pkg_resources.py", line 818, in best_match
        return self.obtain(req, installer) # try and download/install
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/pkg_resources.py", line 830, in obtain
        return installer(requirement)
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/setuptools/dist.py", line 314, in fetch_build_egg
        return cmd.easy_install(req)
      File "/home/hudsonslave/root/workspace/MYJOB/MYPROJECT/.virtualenv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 587, in easy_install
        raise DistutilsError(msg)
    distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pbr')
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.python.org/simple/pbr/: [Errno 110] Connection timed out -- Some packages may not be found!

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):

---

Anyone try out my diff?

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Not yet. It's been a busy week, and I'm currently concerned about the failing tests in the tip with the new zipimport. Once that's settled, released, and stable, I'll revisit this issue. Feel free to ping periodically if you don't see any activity.

Original comment by rbuchholz (Bitbucket: rbuchholz, GitHub: rbuchholz):

---

@jaraco, ping?

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

@msabramo Why does your patch work? I don't understand why installing the package to a temporary directory doesn't trigger the install_requires directive to be satisfied for pbr. Is it because '.' is on the sys.path, and thus setuptools will consider any eggs on sys.path as installed?

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):

---

I think it's what you said, but I'm not positive as my brain has swapped this out. :-)

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

It seems unwise to me to change setuptools to use a global temp directory as a cache simply to address this issue. One concern is the possibility that an untrusted agent could write to that directory in advance, fulfilling the requirement with untrusted code.

Another option that's been proposed is to have the eggs created in a specially-named directory, such as "./.egg-cache". That also might address the issue, and would avoid the additional security risk (see #80).

There are additional issues too that I would like to see solved, such as the ability for setup_requires requirements to be more strict than system-installed packages (#141).

So the proposed patch doesn't address the issue directly and probably has unwanted consequences. I'd prefer a solution that addresses the issue directly by carefully keeping setup_requires and install_requires packages distinct, but I'd also accept a PR that bypasses the issue by fixing #80.