Upgrade setuptools to a more recent version

[russwinch]

I'm starting to get incompatibility warnings about the version of setuptools supplied in this buildpack:

ERROR: google-auth 1.7.1 has requirement setuptools>=40.3.0, but you'll have setuptools 39.0.1 which is incompatible.

If I manually pip install the google-auth package, setuptools is automatically updated but this doesn't seem the case when the app is built.

The version provided in the build pack was released 18 Mar 2018. Latest is 46.1.3

Would it be possible to get an update to setuptools in the next release?

[rvanlaar]

Also needed for cairocffi: https://github.com/Kozea/cairocffi/blob/master/setup.cfg#L43
Or enable the options to upgrade setuptools ourselves during the installation.

Made a PR for this: #973

[russwinch]

it would be good to get a response to this, or the PR from @rvanlaar with a potential fix, it's been a number of weeks with no acknowledgement

[edmorley]

@russwinch Hi! Sorry for the delayed reply, we've just had a changeover of Python language owner, and it's going to take a while for me to get through the discovery/switching role onboarding process. I very much want to upgrade setuptools and other out of date dependencies in this buildpack, though they are high risk so will need some investigation/time to reduce chance of breaking other customers.

[rvanlaar]

About breaking other builds, yes it might break their builds, however it currently also breaks builds and prevents users from installing certain packages.

[trevorphillipscoding]

Just starting my heroku app and installing it gives me same errors, including the cairocffi one.

ERROR: google-auth 1.18.0 has requirement setuptools>=40.3.0, but you'll have setuptools 39.0.1 which is incompatible. ERROR: tinycss2 1.0.2 has requirement setuptools>=39.2.0, but you'll have setuptools 39.0.1 which is incompatible. ERROR: cssselect2 0.3.0 has requirement setuptools>=39.2.0, but you'll have setuptools 39.0.1 which is incompatible. ERROR: cairocffi file-.cairocffi-VERSION has requirement setuptools>=39.2.0, but you'll have setuptools 39.0.1 which is incompatible. ERROR: weasyprint 51 has requirement cairocffi>=0.9.0, but you'll have cairocffi file-.cairocffi-VERSION which is incompatible. ERROR: weasyprint 51 has requirement setuptools>=39.2.0, but you'll have setuptools 39.0.1 which is incompatible.

[jualopver]

I had the same issue with cairocffi and weasyprint.

[edmorley]

@russwinch @rvanlaar @trevorphillips @jualopver Hi :-)

Now that the pip/setuptools/wheel installation steps have been fixed/refactored in #1007, we're ready to upgrade setuptools to a newer release.

I was hoping to add a testcase at the same time, to prove that previously broken packages install fine. However I'm unable to reproduce?

What version of Python are you using? Are you using Pip or pipenv? What version of the Python buildpack? And can you reproduce on a clean app?

For example I tried:

src $ mkdir testcase
src $ cd testcase/
testcase $ git init
...
testcase $ h create edmorley-testcase
...
testcase $ echo 'cairocffi' > requirements.txt
testcase $ git add -A; git commit -m 'bump'; git push heroku main
[main (root-commit) 42c7829] bump
 1 file changed, 1 insertion(+)
 create mode 100644 requirements.txt
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 239 bytes | 239.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Python app detected
remote: -----> Installing python-3.6.11
remote: -----> Installing pip
remote: -----> Installing SQLite3
remote: -----> Installing requirements with pip
remote:        Collecting cairocffi
remote:          Downloading cairocffi-1.1.0.tar.gz (68 kB)
remote:        Collecting cffi>=1.1.0
remote:          Downloading cffi-1.14.1-cp36-cp36m-manylinux1_x86_64.whl (400 kB)
remote:        Collecting setuptools>=39.2.0
remote:          Downloading setuptools-49.2.0-py3-none-any.whl (789 kB)
remote:        Collecting pycparser
remote:          Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
remote:        Building wheels for collected packages: cairocffi
remote:          Building wheel for cairocffi (setup.py): started
remote:          Building wheel for cairocffi (setup.py): finished with status 'done'
remote:          Created wheel for cairocffi: filename=cairocffi-file_.cairocffi_VERSION-py3-none-any.whl size=88788 sha256=8b80f7f5d88aa4a7ac674167ca26986287611c5039de879cf38254fc495432ef
remote:          Stored in directory: /tmp/pip-ephem-wheel-cache-ovzow6w8/wheels/bd/b2/c1/d88d84a95f5dbbaea6361c304a3c76500f88a886ee148de78f
remote:        Successfully built cairocffi
remote:        Installing collected packages: pycparser, cffi, setuptools, cairocffi
remote:          Attempting uninstall: setuptools
remote:            Found existing installation: setuptools 39.0.1
remote:            Uninstalling setuptools-39.0.1:
remote:              Successfully uninstalled setuptools-39.0.1
remote:        Successfully installed cairocffi-file-.cairocffi-VERSION cffi-1.14.1 pycparser-2.20 setuptools-49.2.0
remote: -----> Discovering process types
remote:        Procfile declares types -> (none)
remote:
remote: -----> Compressing...
remote:        Done: 45.1M
remote: -----> Launching...
remote:        Released v3
remote:        https://edmorley-testcase.herokuapp.com/ deployed to Heroku
remote:
remote: Verifying deploy... done.
To https://git.heroku.com/edmorley-testcase.git
 * [new branch]      main -> main

[russwinch]

Hi @edmorley thanks for having a look at this.

I have replicated your results with a clean app

remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Python app detected
remote: -----> No change in requirements detected, installing from cache
remote: -----> Installing pip 20.0.2, setuptools 39.0.1 and wheel 0.34.2
remote: ERROR: google-auth 1.7.1 has requirement setuptools>=40.3.0, but you'll have setuptools 39.0.1 which is incompatible.
remote: -----> Installing SQLite3
remote: -----> Installing requirements with pip
remote:        Collecting setuptools>=40.3.0
remote:          Downloading setuptools-49.2.0-py3-none-any.whl (789 kB)
remote:        Installing collected packages: setuptools
remote:          Attempting uninstall: setuptools
remote:            Found existing installation: setuptools 39.0.1
remote:            Uninstalling setuptools-39.0.1:
remote:              Successfully uninstalled setuptools-39.0.1
remote:        Successfully installed setuptools-49.2.0
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote:
remote: -----> Compressing...
remote:        Done: 46M
remote: -----> Launching...
remote:        Released v4
remote:        https://es-setuptools39-0-1-test.herokuapp.com/ deployed to Heroku

Curiously, this automatic upgrade does not happen when deploying the app I originally saw the error in:

ERROR: google-auth 1.20.0 has requirement setuptools>=40.3.0, but you'll have setuptools 39.0.1 which is incompatible.
       Installing collected packages: SQLAlchemy, MarkupSafe, Mako, python-editor, six, python-dateutil, alembic, aniso8601, atomicwrites, attrs, blinker, cachetools, certifi, chardet, Click, text-unidecode, Faker, factory-boy, idna, urllib3, requests, pytz, protobuf, googleapis-common-protos, pyasn1, rsa, pyasn1-modules, google-auth, google-api-core, google-resumable-media, google-cloud-core, google-cloud-storage, uritemplate, httplib2, google-auth-httplib2, google-api-python-client, msgpack, cachecontrol, google-cloud-firestore, firebase-admin, itsdangerous, Werkzeug, Jinja2, Flask, WTForms, Flask-Admin, Flask-Cors, PyJWT, Flask-JWT-Extended, flask-log-request-id, Flask-Login, Flask-SQLAlchemy, Flask-Migrate, pyrsistent, jsonschema, flask-restplus, Flask-Script, flask-talisman, greenlet, gevent, gspread, gunicorn, more-itertools, numpy, oauth2client, oauthlib, pandas, Pillow, pluggy, psycopg2-binary, py, pycryptodome, pytest, coverage, pytest-cov, sentry-sdk, shortuuid, stripe, voluptuous
       Successfully installed Click-7.0 Faker-2.0.4 Flask-1.0.2 Flask-Admin-1.5.3 Flask-Cors-3.0.8 Flask-JWT-Extended-3.17.0 Flask-Login-0.5.0 Flask-Migrate-2.3.1 Flask-SQLAlchemy-2.4.1 Flask-Script-2.0.6 Jinja2-2.10.1 Mako-1.1.0 MarkupSafe-1.1.1 Pillow-7.1.1 PyJWT-1.7.1 SQLAlchemy-1.3.18 WTForms-2.2.1 Werkzeug-0.16.0 alembic-1.0.7 aniso8601-8.0.0 atomicwrites-1.3.0 attrs-19.3.0 blinker-1.4 cachecontrol-0.12.6 cachetools-3.1.1 certifi-2019.11.28 chardet-3.0.4 coverage-5.2.1 factory-boy-2.12.0 firebase-admin-4.3.0 flask-log-request-id-0.10.1 flask-restplus-0.13.0 flask-talisman-0.7.0 gevent-20.4.0 google-api-core-1.22.0 google-api-python-client-1.10.0 google-auth-1.20.0 google-auth-httplib2-0.0.4 google-cloud-core-1.0.3 google-cloud-firestore-1.8.1 google-cloud-storage-1.23.0 google-resumable-media-0.5.0 googleapis-common-protos-1.52.0 greenlet-0.4.16 gspread-3.1.0 gunicorn-20.0.4 httplib2-0.18.0 idna-2.8 itsdangerous-1.1.0 jsonschema-3.2.0 more-itertools-8.0.0 msgpack-1.0.0 numpy-1.17.4 oauth2client-4.1.3 oauthlib-3.1.0 pandas-0.25.3 pluggy-0.13.1 protobuf-3.12.4 psycopg2-binary-2.8.4 py-1.8.0 pyasn1-0.4.8 pyasn1-modules-0.2.7 pycryptodome-3.9.4 pyrsistent-0.15.6 pytest-4.3.0 pytest-cov-2.8.1 python-dateutil-2.8.1 python-editor-1.0.4 pytz-2019.3 requests-2.22.0 rsa-4.0 sentry-sdk-0.14.3 shortuuid-0.5.0 six-1.13.0 stripe-2.20.3 text-unidecode-1.3 uritemplate-3.0.1 urllib3-1.25.7 voluptuous-0.11.5
-----> Discovering process types
       Procfile declares types -> release, web
-----> Compressing...
       Done: 123.6M
-----> Launching...
 !     Release command declared: this new release will not be available until the command succeeds.
       Released v10

Presumably the issue could be to do with our combination of requirements? But I have upgraded every package that depends on setuptools and have not been able to trigger the setuptools upgrade seen in the clean app.

This this app is deployed via the github integration, vs heroku command line in the clean app, if this helps.

What version of Python are you using? 3.8.3
Are you using Pip or pipenv? Pip
What version of the Python buildpack? not specified, presumably the latest heroku/python

[edmorley]

@russwinch Thank you for that package list - using that I could reproduce.

Running pip uninstall setuptools in a one-off followed by pip check showed:

zope-interface 5.1.0 requires setuptools, which is not installed.
zope-event 4.4 requires setuptools, which is not installed.
protobuf 3.12.4 requires setuptools, which is not installed.
jsonschema 3.2.0 requires setuptools, which is not installed.
gunicorn 20.0.4 requires setuptools, which is not installed.
google-auth 1.20.0 requires setuptools, which is not installed.
google-api-core 1.22.0 requires setuptools, which is not installed.
gevent 20.6.2 requires setuptools, which is not installed.

Which led to the reduced testcase of:

# requirements.txt
protobuf
google-auth

$ git push heroku main
...
remote: -----> Python app detected
remote: -----> Requirements file has been changed, clearing cached dependencies
remote: -----> Installing python-3.6.11
remote: -----> Installing pip 20.0.2, setuptools 39.0.1 and wheel 0.34.2
remote: -----> Installing SQLite3
remote: -----> Installing requirements with pip
remote:        Collecting protobuf
remote:          Downloading protobuf-3.12.4-cp36-cp36m-manylinux1_x86_64.whl (1.3 MB)
remote:        Collecting google-auth
remote:          Downloading google_auth-1.20.0-py2.py3-none-any.whl (91 kB)
remote:        Collecting six>=1.9
remote:          Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
remote:        Collecting rsa<5,>=3.1.4; python_version >= "3.5"
remote:          Downloading rsa-4.6-py3-none-any.whl (47 kB)
remote:        Collecting pyasn1-modules>=0.2.1
remote:          Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
remote:        Collecting cachetools<5.0,>=2.0.0
remote:          Downloading cachetools-4.1.1-py3-none-any.whl (10 kB)
remote:        Collecting pyasn1>=0.1.3
remote:          Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
remote:        ERROR: google-auth 1.20.0 has requirement setuptools>=40.3.0, but you'll have setuptools 39.0.1 which is incompatible.
remote:        Installing collected packages: six, protobuf, pyasn1, rsa, pyasn1-modules, cachetools, google-auth
remote:        Successfully installed cachetools-4.1.1 google-auth-1.20.0 protobuf-3.12.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 rsa-4.6 six-1.15.0

I believe the reason pip doesn't upgrade setuptools in this case is pypa/pip/issues/988.

ie: For transitive dependencies the first version specified "wins" - and since protobuf depends on setuptools (with no version; so is already met), the later google-auth dependency on setuptools>=40.3.0 is ignored.

Thankfully a new pip resolver which fixes this is already in progress:
https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html
https://discuss.python.org/t/announcement-pip-20-2-release/4863

Anyway for now I'll update setuptools which will prevent this being an issue regardless of the resolver :-)