Do not exclude legitimately installed packages from pip freeze

[piotr-dobrogost]

Silently excluding legitimately installed packages like setuptools, distribute and pip is not ok. For instance, as requirements file can contain options for pip, someone might want to ensure that the proper version of pip (able to understand these options) would be available. Also, @qwcode gave rationale for not excluding setuptools in issue #1606:

with pip 1.5 able to install from wheel without setuptools, I can see a real case where it's meaningful for setuptools to be listed, i.e. when one of the packages needs setuptools due to an import of pkg_resources, but the install is just using pip (and no setuptools) to install from wheel.

Reasons for having setuptools as a requirement were given in answers to my recent question Is there any sense to put setuptools as a requirement? on distutils-sig mailing list.

There should be an option so that users could decide themselves what packages if any they want to exclude. In any case there should be information about excluded packages presented in the output of every command affected.

By legitimately installed I mean all outside of stdlib as excluding stdlib packages with metadata is another subject – see issue #1570.

[dstufft]

Putting pip inside of a requirements.txt file is almost never what you want. For instance, you stated:

For instance, as requirements file can contain options for pip, someone might want to ensure that the proper version of pip (able to understand these options) would be available.

However the problem is, if you have a requirements file like that, then an incompatible version of pip will bomb out trying to read that requirements.txt file before it even has a chance to upgrade pip. Even if everything goes without a hitch and you successfully upgrade/downgrade pip to a different version via the requirements.txt file it won't actually take any effect until the next invocation of pip (because the current version will have already been imported). If we want to handle the case where you modify the version of pip in a single invocation then we're going to need additional code to handle that.

In the past there was the same sort of problem with setuptools. You couldn't import pip without setuptools installed and you couldn't upgrade setuptools in the same process and have it take effect in the installation of the rest of the packages in that installation run. Now a days this is no longer the case, however I would still argue that it should be excluded by default because rarely does someone actually want to assert a particular version of setuptools (and in fact it can be harmful, especially on platforms like Heroku).

I do not believe we should be putting pip into the output of pip freeze by default, and I feel pretty strongly about that. setuptools/distribute being included by default I still don't think is generally applicable and can be harmful, but I could probably be convinced otherwise. One option is to include setuptools/distribute by default, but have it commented out so that it's obvious that it's there and what version was installed, but that it doesn't affect anything without end user work. Another option is to make the list of what to ignore configurable. If the user hasn't specified any ignores it uses our default list. If they configure any at all they are responsible for configuring them all.

[pfmoore]

I don't have a strong opinion about setuptools, but I agree with @dstufft that including pip is wrong. And I do not want to see distribute included by default.

People can manually edit requirements files however they like - as far as I can see this is purely about using freeze and not reviewing or amending its output in any way. I think I'd need to see a specific, clear use case where the current behaviour causes an error and where manual editing of the requirements file is not practical, before I'd think this was worth it.

[qwcode]

One option is to include setuptools/distribute by default,
but have it commented out so that it's obvious that it's there and what version was installed
Another option is to make the list of what to ignore configurable.
If the user hasn't specified any ignores it uses our default list.

I'd be ok with either or both of these.

[Ivoz]

I think part of the issue is that pip and setuptools are pretty much "build environment" requirements, not regular "install" requirements; but requirements.txt' use case was only ever consciously designed for the latter.

If you separate these two concepts, you come up with a modicum solution:

pip install -r build_requirements.txt
pip install -r requirements.txt

Where build_requirements.txt had something like:

setuptools==<version>
pip==<version>

[taion]

As a reference point for how other package managers handle this, NPM explicitly splits out:

• Dependencies: https://docs.npmjs.com/files/package.json#dependencies
• Development dependencies: https://docs.npmjs.com/files/package.json#devdependencies
• A "shrinkwrap" that locks down all dependency and sub-dependency versions: https://docs.npmjs.com/cli/shrinkwrap

I think in general something like pip freeze that locks down the full set of dependencies and sub-dependencies is not the right way to handle development dependencies anyway.