Support force resolving dependency versions a la yarn's "resolutions" support.

[chris-codaio]

Is your feature request related to a problem? Please describe.

GitHub is complaining that a transitive dependency has a vulnerability and I see no way in pipenv to force a resolution in the same way I currently can in the javascript ecosystem with yarn (https://classic.yarnpkg.com/en/docs/selective-version-resolutions/).

I'm using ScoutSuite which in turn relies on oci which relies on a version of cryptography that GitHub considers vulnerable. It looks like oci has no immediate plans to upgrade this bad dependency (oracle/oci-python-sdk#299). Since I don't use Oracle cloud, I'm happy if the oci dependency from ScoutSuite doesn't work. I just want to force update this dependency to resolve to version >= 3.2 where the issue is resolved.

Describe the solution you'd like

A resolutions section of the pipfile equivalent to yarn's resolutions section which behaves in the same way during install/resolution.

Describe alternatives you've considered

The only viable alternative I see is to fork one or both of the repos above and make the necessary changes myself - way too much effort for what should be a one-line change in my local pipfile.

Additional context

N/A

[frostming]

You can achieve this by updating the dependency version:

$ pipenv update

Or alternatively, you can pin the dependency version in your Pipfile:

[packages]
cryptography = ">=3.2"

Tell me if it solves your problem.

[chris-codaio]

No, this doesn't work as the ScoutSuite dependency relies on oci which has a dependency specifically to version 2.8 of the cryptography lib: https://github.com/oracle/oci-python-sdk/blob/master/requirements.txt#L5.

Attempting to override that locally using the suggestion above results in:

Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
[ResolutionFailure]:   File "/usr/local/Cellar/pipenv/2020.11.4/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 741, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
[ResolutionFailure]:   File "/usr/local/Cellar/pipenv/2020.11.4/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 702, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/usr/local/Cellar/pipenv/2020.11.4/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 684, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/usr/local/Cellar/pipenv/2020.11.4/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 1378, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/usr/local/Cellar/pipenv/2020.11.4/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 1093, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/usr/local/Cellar/pipenv/2020.11.4/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 818, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Could not find a version that matches cryptography==2.8,>=1.1.0,>=3.2 (from -r /var/folders/vd/r7ftxx9j2mx3gsyc80tsnskw0000gn/T/pipenv6f3i2losrequirements/pipenv-6wl5z1qj-constraints.txt (line 7))
Tried: 0.1, 0.2, 0.2.1, 0.2.2, 0.3, 0.4, 0.5, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6, 0.6.1, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.9, 0.9.1, 0.9.2, 0.9.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.1.1, 1.1.2, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.4, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.6, 1.7, 1.7.1, 1.7.2, 1.8, 1.8.1, 1.8.2, 1.9, 2.0, 2.0.1, 2.0.2, 2.0.3, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.2, 2.2, 2.2.1, 2.2.1, 2.2.2, 2.2.2, 2.3, 2.3, 2.3.1, 2.3.1, 2.4, 2.4, 2.4.1, 2.4.1, 2.4.2, 2.4.2, 2.5, 2.5, 2.6, 2.6.1, 2.6.1, 2.7, 2.7, 2.8, 2.8, 2.9, 2.9, 2.9.1, 2.9.1, 2.9.2, 2.9.2, 3.0, 3.0, 3.1, 3.1, 3.1.1, 3.1.1, 3.2, 3.2, 3.2.1, 3.2.1
There are incompatible versions in the resolved dependencies:
  cryptography>=3.2 (from -r /var/folders/vd/r7ftxx9j2mx3gsyc80tsnskw0000gn/T/pipenv6f3i2losrequirements/pipenv-6wl5z1qj-constraints.txt (line 7))
  cryptography==2.8 (from oci==2.23.5->scoutsuite==5.10.1->-r /var/folders/vd/r7ftxx9j2mx3gsyc80tsnskw0000gn/T/pipenv6f3i2losrequirements/pipenv-6wl5z1qj-constraints.txt (line 10))
  cryptography>=1.1.0 (from adal==1.2.4->scoutsuite==5.10.1->-r /var/folders/vd/r7ftxx9j2mx3gsyc80tsnskw0000gn/T/pipenv6f3i2losrequirements/pipenv-6wl5z1qj-constraints.txt (line 10))

[frostming]

So if you have a package that has exactly pinned dependencies, it is impossible to have cryptography resolved to 3.2. It means oci only works under cryptography==2.8, any fancy commands or options can't help in this situation. You may have to ignore that security warning unless oci updates the dependency constraints.

[chris-codaio]

I agree that this is how it works today, but I'm asking for a feature to override that pinned dependency locally.

[frostming]

No, it is not a flaw. You can't pin cryptography to 3.2 when 2.8 is pinned by oci and you can't override the dependency version without changing the code that can only work with 2.8, otherwise your app might get broken

[chris-codaio]

I'm quite happy to take the risk in this case. That said, here are the reasons you might want to do this (from Yarn's site):

1. You may be depending on a package that is not updated frequently, which depends on another package that got an important upgrade. In this case, if the version range specified by your direct dependency does not cover the new sub-dependency version, you are stuck waiting for the author.

2. A sub-dependency of your project got an important security update and you don’t want to wait for your direct-dependency to issue a minimum version update.

3. You are relying on an unmaintained but working package and one of its dependencies got upgraded. You know the upgrade would not break things and you also don’t want to fork the package you are relying on, just to update a minor dependency.

4. Your dependency defines a broad version range and your sub-dependency just got a problematic update so you want to pin it to an earlier version.

[frostming]

Well, I am sold. In case the risk is known and acceptable.

I prefer to add an important flag to the entry in Pipfile, like what it does in CSS, meaning it should take precedence even incompatibility is found.

[uranusjr]

I wrote a while ago about my thoughts on this for the pip equivalent to this pypa/pip#8076. TL;DR It is too much a footgun to allow users to override dependency information directly, and it would be better for all sides (the users, installer authors, maintainers of the package to be overriden) if the procedure can be done more transparently.

I proposed an alternative approach to the issue: pypa/pip#8076 (comment). This should not be too difficult to integrate this into pipenv (add a new section in Pipfile; perform the vendor+patch procedure automatically during sync), but someone needs to carry out the required design and implementation work. I don’t personally have a use for the feature at all and lack the interest on it.

[chris-codaio]

I disagree that this is too much of a footgun. It seems to work quite well in the javascript ecosystem - unclear why the same pattern wouldn't work well here in python as well.

Here's a link to the RFC from javascript land (specifically the yarn tool): https://github.com/yarnpkg/rfcs/blob/master/implemented/0000-selective-versions-resolutions.md

[uranusjr]

You are free to disagree; this is what the PEEP procedure is for, to persuade pipenv maintainers. It is not nearly enough to say “JavaScript has this” since by the same measure we would have list.sort() work on heterogeneous lists and convert everything to str 🙂 It is not the way to design to decide on a solution first and reason it retrospectively; you need to describe the problem to solve, consider alternatives and their respective pros and cons, and only land on a solution if it offers the best trade-offs to the initial problem.

[mstrofbass]

One of the jokes I make about the Python ecosystem is how all the library maintainers think they can do a better job solving problems that have been long solved by similar libraries in other languages and inevitably release a far inferior product.