The setuptools and wheel versions are not fully pinned

[edmorley]

As part of Python setup, the buildpack has to manage the installation of pip, setuptools and wheel, so that they are available for the later pip install of user-provided dependencies.

Changes in these packages can have a significant impact on the build (particularly since the buildpack has to manually modify site-packages as part of BUILD_DIR -> /app path rewriting, so is sadly sensitive to implementation details), so it's important that we pin to tested versions so that builds don't suddenly break after a new upstream release.

Equally as important as preventing new releases from being installed unintentionally, is ensuring that:

• these packages in existing apps are upgraded if the buildpack pinned versions change (so they benefit from fixes, and to reduce the version matrix when reasoning about buildpack changes)
• any user-provided custom versions of these packages are cleaned up if they are later removed from requirements.txt (to prevent their presence in the build cache causing hard to debug situations)

For pip, the buildpack currently does the right thing:

• if pip is not present, a pinned version is installed (currently 9.0.2 for pipenv users, 19.1.1 for Python 3.4, and 20.0.2 otherwise)
• if pip is present, the version is checked against the expected pinned version and upgraded/downgraded if necessary

However for setuptools, the behaviour is inconsistent:

• for clean installs (new app, manually cleared build cache, Python/stack version upgrade), a pinned setuptools version is installed (currently 39.0.1)
• for non-clean installs:
  • if the pip version was changed since the last build, then setuptools is upgraded/downgraded to the pinned version (if necessary)
  • otherwise, the setuptools version (or even the presence of setupools at all) is not checked/managed

And for wheel, the situation is worse:

• for clean installs, the wheel version is not pinned so the latest is installed
• for non-clean installs, so long as any version of wheel is installed, it's never upgraded/changed

The above leads to both unpredictability of installed versions, as well as confusing UX when performing other actions (eg failures after upgrading to a new Python patch version, that had nothing to do with the Python upgrade itself, but just happened to trigger installing newer setuptools/wheel etc).

We should pin all three to specific versions, and ensure that these versions are checked/managed during each build, and not just after a cache-clearing event.

See:

heroku-buildpack-python/bin/steps/python

Lines 150 to 161 in 156b07c

	# If a new Python has been installed or Pip isn't up to date:
	if [ "$FRESH_PYTHON" ] || [[ ! $(pip --version) == *$PIP_UPDATE* ]]; then
	puts-step "Installing pip"
	# Remove old installations.
	rm -fr /app/.heroku/python/lib/python*/site-packages/pip-*
	rm -fr /app/.heroku/python/lib/python*/site-packages/setuptools-*
	/app/.heroku/python/bin/python "$GETPIP_PY" pip=="$PIP_UPDATE" &> /dev/null
	/app/.heroku/python/bin/pip install "$ROOT_DIR/vendor/setuptools-39.0.1-py2.py3-none-any.whl" &> /dev/null
	fi