New pre-release of numpy breaks an old repository with pinned versions

[betatim]

Since a few days ago our CI started failing for one of the 'frozen in time" repos we use to check if we can still build old images.

The problem is that a new version of numpy was pre-released a few days ago. Since then one of our "check we don't break old repos in the wild" tests is now installing the latest version of numpy, which requires Python 3 but the repo wants Python 2.

The weird thing is that the repository specifies a version of numpy that it would like to install, a Python version and a pinned matplotlib version as well.

This is the end of the trace back we see:

      File "/srv/conda/envs/kernel/lib/python2.7/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-VBFt7K/numpy-1.17.0rc1/setup.py", line 31, in <module>

    RuntimeError: Python version >= 3.5 required.
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-sgij5M/matplotlib/

This is while it is trying to install matplotlib-1.5.1. It is odd that this is trying to install numpy 1.17 from source instead of 1.10.4 which is what is specified in requirements.txt

Figuring out why this happens is high priority as it will block all PRs.

[davidrpugh]

Don't use pip regularly; according to the docs the order of installation process is as follows.

1. Identify the base requirements. The user supplied arguments are processed here.
2. Resolve dependencies. What will be installed is determined here.
3. Build wheels. All the dependencies that can be are built into wheels.
4. Install the packages (and uninstall anything being upgraded/replaced).

From the Travis logs it seems like Matplotlib is downloaded and installed first, which seems odd because pip should install packages in topological order meaning that NumPy (and any other Matplotlib deps) should be installed first.

[minrk]

It's related to setup_requires since there's no wheel for matplotlib 1.5.1 which has numpy in setup_requires, which is handled by setuptools (not pip). Since there's no wheel for matplotlib, it's building from source, pulling in numpy during build. At the time it does this, it doesn't know that numpy is topologically before matplotlib (it can't know when building from source, since the install_requires metadata hasn't been determined yet). Since setup_requires and easy_install are deprecated functionality, they've never been fixed to support things like python_requires metadata that would forbid the new prerelease which has dropped py2.

The basic answer is: pinning matplotlib 1.5.1 doesn't work for Python 2.7 unless numpy is installed first, and for packages installed from source at least, requirements.txt must itself be in topological order.

Related: pypa/setuptools#917 which proposes PEP 518 as the replacement for setup_requires, but no real plan to fix the deprecated setup_requires to use pip (since it could break other things).

[davidrpugh]

That all makes sense to me but not sure where it leaves us. Should we open a PR to update the repo's requirements.txt file to put numpy ahead of matplotlib? Not sure what else can be done...other than dropping the repo from the test suite.

[betatim]

So we've been building numpy from source for matplotlib all along but now the version of numpy that gets pulled in for that doesn't work with Python 2 any more which is why we see the failure now?

We can't edit the repository because it is a Zenodo record that is immutable.

What we could do is take the repo and create a new Zenodo entry where the order in the requirements.txt is switched.

[minrk]

So we've been building numpy from source for matplotlib all along but now the version of numpy that gets pulled in for that doesn't work with Python 2 any more which is why we see the failure now?

Yes, that's exactly right. This is why truly reproducible builds are hard!